Compare commits
25 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 admin
|
53d962d95b | ||
|
admin
|
836b47f1ec | ||
|
admin
|
bb187f3503 | ||
|
admin
|
9db8aacdad | ||
|
admin
|
78278502ac | ||
|
admin
|
2228b9f6b4 | ||
|
admin
|
ee743477b4 | ||
|
admin
|
ad6397308c | ||
|
admin
|
d750a90d0e | ||
|
admin
|
a942ae90d6 | ||
|
admin
|
20a77d367b | ||
|
admin
|
a57a8363df | ||
|
admin
|
bffe2d9dea | ||
|
admin
|
242584af4e | ||
|
admin
|
43e5a5496d | ||
|
admin
|
c701b2530b | ||
|
admin
|
54d1ee8210 | ||
|
admin
|
1d7071e8fd | ||
|
admin
|
9ec5f76a88 | ||
|
admin
|
f84f94d987 | ||
|
admin
|
2954b0be3a | ||
|
admin
|
5c487313fa | ||
|
admin
|
bcaa9576b2 | ||
|
admin
|
301f4e9e92 | ||
|
admin
|
b0ae2c574a |
@@ -138,7 +138,14 @@ something, not only sessions that touch `documentation/` — which is why it is
|
||||
and the eleven names head `OPEN-ITEMS.md`, and `register_shape_gate.py` refuses anything else.
|
||||
- **Root `STATUS.md`** — at the end of every session in which something shipped, broke or was
|
||||
decided. It is a **view** of `OPEN-ITEMS.md`; nothing may exist only there. One screen, written for
|
||||
the operator in plain language, and deliberately **not** `CONTEXT.md`.
|
||||
the operator in plain language (STE English), and deliberately **not** `CONTEXT.md`. How to update it:
|
||||
1. **Replace** the „Updated" paragraph (at most 3 sentences, the newest change only) — never prepend; the old one
|
||||
moves to `documentation/backlog/STATUS-HISTORY.md` under a dated heading.
|
||||
2. A morning list goes under a dated sub-heading in „Waiting on you — do".
|
||||
3. Remove what you finished: a done „do" item, a fixed „broken" line, a decided row (its ruling is in its row first).
|
||||
4. A decide row: the question in one answerable sentence, at most 2 options, CC's pick first with the reason, the
|
||||
other option's cost, what happens if the operator does nothing, and who is blocked.
|
||||
5. `scripts/status_view_gate.py` checks it on every push (warn-first until 2026-10-27, then it refuses; R-1174).
|
||||
- **The golden, weekly** (operator ruling 2026-10-15 11:26, `09` §3 decision 250; decision 7 stands): **a controller
|
||||
release needs no golden** — the floor carries it, MinAgent declared, after the three test boxes (`09` §3 decision 250
|
||||
item 2). The golden is baked **every Sunday night** when the floor is above it (`runbooks/nightly-rotation.md`) and
|
||||
|
||||
+126
-1
@@ -1,4 +1,129 @@
|
||||
# REPORT — evening/night brief 2026-10-19 (one stop less per night, no false tunnel error, R-893 measured, hub and agent fixes ready, a scanner on every push, ep0's first weekly clean-up)
|
||||
# REPORT — the night briefs of 2026-10-19 and 2026-10-20 (part 2 run by day): one stop less per night, the safe undo for a failed off-site restore, hub 0.162.0 / agent 0.161.0 / installer 1.35.0 delivered, downloads out of the backups, a scanner on every push
|
||||
|
||||
# PART 2 — the night brief part 2 (run by day, 2026-10-20): your four answers, the safe undo, downloads out of the backups, Immich, the update test
|
||||
|
||||
**Safest readings taken alone (written first, as the brief asks):**
|
||||
1. **R-1083 is not in agent 0.161.0.** An agent key would make 9202's controller book the agent's whole-guest backups,
|
||||
one of which writes to ep0 — a change on ep0 the brief forbids; 9202 is also outside the `felhom` pool.
|
||||
2. **All five downloaders' folders are `excluded`, not only the two named** (qBittorrent, Transmission; also NZBGet,
|
||||
SABnzbd, MeTube): the brief's gate rule — a download folder is never optional or mandatory — covers all five.
|
||||
3. **No update-test edge was run:** the bench (guest 9401) and the drill catalog are outside the brief's machines and
|
||||
repos. The list done and the list left are written; the ask is on the morning list.
|
||||
4. The brief was written for a night run and ran by day; its „stop by 05:00" was read as a time box, not a clock.
|
||||
|
||||
## Part 2 — in plain words
|
||||
|
||||
- **Morning list:** vouch agent 0.161.0 · decide D21 (Immich) · MeTube's videos out of the second-drive copy — keep? ·
|
||||
the bench and the drill catalog for the update test — yes?
|
||||
- **Hub 0.162.0 is live.** Its own deploy was the real restart test: Tester-2 (down since 4 October) got no repeat mail
|
||||
— read in the hub and, separately, in your mailbox.
|
||||
- **Agent 0.161.0 runs on demo-hp, Tester 1 and demo-felhom.** On demo-hp the agent said „busy" during a backup and
|
||||
„free" before and after. **Not vouched — that is yours.**
|
||||
- **Installer 1.35.0 is published.** The served file equals the tag's file.
|
||||
- **A failed off-site restore now puts the app back exactly as it was** (your D20 = A, controller 0.336.0, the floor).
|
||||
Proven on the scratch box: after a forced failure the database matched its fingerprints from before, the app ran,
|
||||
nothing was held; a normal restore then worked.
|
||||
- **Downloads are out of the backups** (torrent, usenet, MeTube): the controller's own check copies 0 of them now.
|
||||
- **Immich's off-site copy:** 1.33 GB, of which 102 MB must survive. Two options; pick B (leave out the AI model files).
|
||||
- **ep0's first weekly clean-up:** done this morning (Part 10 of part 1 below).
|
||||
|
||||
## Part 2 — each part
|
||||
|
||||
| Part | Done? | Why / how |
|
||||
|---|---|---|
|
||||
| 1a hub deploy | **done** | `manifests/hub.yaml` → 0.162.0 (`e1d349f1`), only Deployment/hub synced (the only OutOfSync); image `sha256:69285e64…` = the build; Synced/Healthy; reports from all three boxes; floor read 0.335.0; Day-0 agent list has no `-step1`. R-1001 + R-1011 closed. |
|
||||
| 1b agent release | **done** | R-1083 left out (above). `release-agent.sh 0.161.0` from a clean clone; signed `agent_update` per box: demo-hp 06:38, Tester 1 06:54, demo-felhom 06:55 UTC, each COMPLETED, 76/76, committed. `host_busy` live; tunnel `absent` = the fixture row (9202's real Docker 29 answer); R-1010 by its tests. R-1070 + R-1010 closed. |
|
||||
| 1b R-921 live | **not done** | The no-stop case happens only on the night schedule (a press backs up the local tier only). Row R-1173 with a dated check after demo-hp's next off-site night. |
|
||||
| 1c installer | **done** | tag `installer-v1.35.0`, both webpage pins (`b0ae2c57`), webpage Deployment synced alone; served sha256 `f7a196fd…` = tag. R-1167 closed. |
|
||||
| 2 safe undo | **done** | below. R-893 + R-1169 closed. |
|
||||
| 3 downloads | **done, one read left** | catalog `c50adfc`, gate `download-binds`; classifier proof. demo-felhom (qBittorrent) runs its next second-drive copy 2026-10-21 03:30 — dated check on R-1019 (NARROWED). |
|
||||
| 4 Immich | **done (plan only)** | below. R-49 → READY, D21. |
|
||||
| 5 update test | **not done** | fixture side complete for every installed app; the edges need the bench / drill catalog (above). R-462 NARROWED. |
|
||||
| 6 ep0 clean-up | **done** | part 1, Part 10. |
|
||||
| 7 controller release | **done** | 0.336.0 below. |
|
||||
|
||||
## Part 2 — the safe undo, the 9202 proof (`07` §6.3; `audits/night2-2026-10-20/P2/`)
|
||||
|
||||
- Code `93fb251`: the fit check before the stop, the copy right after it (`backups/restore-undo/<app>/`), the put-back on
|
||||
every failure branch, the hold only when the put-back fails, the copy removed on success. 11 tests; nine mutations red.
|
||||
- **9202, rc.1** (a helper with the fences in full; Rallly + postgres 18; the same forced failure as on 19 October):
|
||||
|
||||
| | Live before | After the forced failure |
|
||||
|---|---|---|
|
||||
| public tables | 36 | 36 |
|
||||
| markers | A, B, C, D | A, B, C, D |
|
||||
| `r893_only_old` (backup only) | absent | **absent** (on 19 October: present) |
|
||||
| schema / data / row-count md5 | `05828c77` / `e5c1a72f` / `d8bcbc4b` | **equal** |
|
||||
| app | running | **running**, no hold |
|
||||
| the copy | — | 69 MB in 0.9 s, removed after |
|
||||
|
||||
The sentence: „…visszaállítása sikertelen. Nem változott semmi: az alkalmazás adatai visszakerültek a visszaállítás
|
||||
előtti állapotba, és az alkalmazás újra fut". Then a normal restore of the same snapshot: the snapshot's content
|
||||
(A–C, the backup-only table present), the app running, the copy removed. 9202 back on 0.331.0 as found.
|
||||
- Not measured live (each pinned by a red-proved test): the version-change branch, the `_files` sentence, the refusals
|
||||
before the stop, a failing put-back, the operator event (9202 has no hub).
|
||||
|
||||
## Part 2 — download binds (`15` §7; `audits/night2-2026-10-20/P3/`)
|
||||
|
||||
| App | Bind (userdata) | Class now | Copied by a tier |
|
||||
|---|---|---|---|
|
||||
| qBittorrent | media/torrents | excluded (was optional) | none (was the second drive) |
|
||||
| Transmission | media/transmission, media/transmission/watch | excluded, excluded | none |
|
||||
| NZBGet | media/nzbget | excluded (was optional) | none |
|
||||
| SABnzbd | media/usenet | excluded (was optional) | none |
|
||||
| MeTube | media/metube | excluded (was optional) | none |
|
||||
| Sonarr, Radarr | media (rw) | excluded | none |
|
||||
| Jellyfin, Plex, Emby | media (:ro) | excluded | none |
|
||||
| Prowlarr | no userdata bind | — | — |
|
||||
|
||||
## Part 2 — Immich (`audits/night2-2026-10-20/P4/immich-offsite-plan.md`)
|
||||
|
||||
Model cache 824 MB (62 %) · raw database folder 283 MB (21 %) · Immich's own backups 115 MB (9 %) · logical dump 54 MB
|
||||
· originals 49 MB · thumbnails 2 MB. **A** (leave out all three re-fetchable parts, −92 %) needs a never-run restore
|
||||
from the dump alone on the vectorchord image; **B** (only the model cache, −62 %) needs one restore with an empty cache.
|
||||
**Pick B now**, A's other parts after their proof.
|
||||
|
||||
## Part 2 — update-test edges
|
||||
|
||||
None run (above). Installed apps: 16; with a fixture: 15; without: bentopdf (keeps no data). Upstream within a major,
|
||||
waiting for a tested step: PrivateBin 2.0.6→2.1.0, Home Assistant 2026.9.4→2026.10.2, RomM 5.3.1→5.4.0, Plex
|
||||
1.41.4→1.43.4, Kimai 2.67.0→2.70.0 (`audits/night2-2026-10-20/P5/r462-list.txt`).
|
||||
|
||||
## Part 2 — versions, commits, rows
|
||||
|
||||
- Hub 0.162.0 deployed (`e1d349f1`). Agent 0.161.0 released (`5152123`, tag `v0.161.0`; notes `4e1621c`). Installer
|
||||
1.35.0 published (`b0ae2c57`). Controller 0.336.0 the floor (`93fb251`; notes `aa9e09e`, `7ca59e2`), image
|
||||
`sha256:93cd67d8…8d73`. Catalog `c50adfc`. Evidence `301f4e9e`, `bcaa9576`.
|
||||
- i18n: no page template changed; seven Go-side keys (HU + EN), one rewritten (R-1169); no fixture changed.
|
||||
- **System poster:** no listed fact changed — none of the three.
|
||||
- **Register 160 before, 156 after. Closed 7:** R-1001, R-1011, R-1167, R-1070, R-1010, R-893, R-1169. **Opened 3:**
|
||||
R-1171 (demo-hp's local copy never restore-tested), R-1172 (the put-back logs at ERROR; an empty folder stays),
|
||||
R-1173 (R-921's no-stop not yet seen live). Re-stated: R-1019 (NARROWED, dated 2026-10-21), R-1083, R-462 (NARROWED),
|
||||
R-49 (READY, D21). **Both briefs together: 161 before, 156 after; closed 11, opened 6.**
|
||||
|
||||
## Observations — part 2
|
||||
|
||||
1. demo-hp's local whole-guest copy is never restore-tested: one copy kept, a 24 h settle, replaced at 24 h. FILED: R-1171
|
||||
2. A successful put-back is logged at ERROR, and an empty `backups/restore-undo/` folder stays after a restore. FILED: R-1172
|
||||
3. R-921's no-stop branch has no live observation yet (it needs a night schedule). FILED: R-1173
|
||||
4. R-462's „only three apps are tested" was out of date: 58 apps have fixtures on both venues. NOT-A-FINDING: the row is re-stated with today's list.
|
||||
5. The hub deploy at 06:03 UTC removed the old pod's log, which held the 07:10 Budapest off-site audit lines of today (R-895). NOT-A-FINDING: R-895's dated check is 2026-10-21; that run happens in the new pod.
|
||||
6. In step 4 of the 9202 drill the data-only md5 differed from a read taken before snapshot 2, while the schema and every row count were equal. NOT-A-FINDING: the two reads were at different moments (the read, then the snapshot's own dump); the claim under test — before and after the forced failure — matched on all three fingerprints.
|
||||
7. The helper's brief expected 37 tables after the normal restore; snapshot 2 has 36. NOT-A-FINDING: last night's 37 was the mixed state, not the snapshot (my arithmetic, corrected).
|
||||
8. The first background wait loop for demo-hp's agent update printed a time and stopped before the box had updated. NOT-A-FINDING: the version was read directly and a foreground loop used instead.
|
||||
9. A Python edit of the controller CHANGELOG failed on a quote character, and the commit carried only REPORT.md. NOT-A-FINDING: fixed by the next commit (`7ca59e2`).
|
||||
|
||||
## Part 2 — teardown
|
||||
|
||||
- Machine: 9202 back on 0.331.0 as found (Rallly removed through the product, the SFTP target, images and files removed
|
||||
by name, settings and the off-box folder put back); demo-hp 9201's probe script removed.
|
||||
- Host: demo-hp `/tmp` copies removed; nothing on `pve/data`; no whole-guest backup of 9202.
|
||||
- Hub: per-box floors set and cleared; the global floor raised; nothing else.
|
||||
- ep0: read only.
|
||||
|
||||
---
|
||||
|
||||
# PART 1 — the evening/night brief of 2026-10-19
|
||||
|
||||
## In plain words
|
||||
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
# REPORT — the status page back to one screen, and a gate that keeps it there (2026-10-20, day)
|
||||
|
||||
## In plain words
|
||||
|
||||
The status page was 107 lines and 31,695 bytes; now it is 50 lines and about 5,500 characters outside the decide table,
|
||||
which holds the 8 decisions that are still open. The „do" list went from about 25 items to 7 „still needed? (yes/no)"
|
||||
questions, 2 morning-list items and 5 other lines (plus one „Later" line of row ids); CC checked the rest in the hub
|
||||
and on DooPlex, and 7 were already done. **You answer:** the „Still needed? (yes/no)" list below, one word per line.
|
||||
|
||||
## Still needed? (yes/no) — copied from STATUS.md
|
||||
|
||||
1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). Since 2026-10-16.
|
||||
2. The N100 sheet, about 30 minutes: Plex, the admins in Prowlarr, Sonarr and Radarr, your download sources
|
||||
(`documentation/audits/n100-lived-in-2026-10-15/SHEET.md`). Since 2026-10-15.
|
||||
3. Look at these pages on demo-hp when you have time: Calibre-Web („Adatok és mentés"), the file manager, AdventureLog,
|
||||
Plex, the app page, the Plex and Sonarr guides („Útmutató"). Since 2026-10-16.
|
||||
4. Remove the test server „plex-k6" from your Plex account (plex.tv → Settings → Authorized Devices). Since 2026-10-15.
|
||||
5. Ask Google to index `https://felhom.eu/sajat-felho-tarhely` in Search Console (R-1035). Since 2026-10-14.
|
||||
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable.
|
||||
Since 2026-10-14.
|
||||
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
|
||||
|
||||
## What CC checked, and how (method: the hub's GET pages with CC's scoped token, from the running pod; read-only local checks on DooPlex)
|
||||
|
||||
| „Do" item | Verdict | Evidence |
|
||||
|---|---|---|
|
||||
| Vouch agent 0.161.0 | **done** — the Day-0 manifest names 0.161.0 with the asked sha256 | `documentation/audits/status-view-2026-10-20/hub-reads.txt` |
|
||||
| Delete the test customer Tester3 | **done** — not in the customer overview | same |
|
||||
| Discard registrations 40 and 41 | **done** — the Hosts page has no „Unclaimed appliances" section (only `registered`/`bound` rows render) | same |
|
||||
| Delete Tester4's old host `Tester4-31fa0e` | **done** — not on the Hosts page | same |
|
||||
| Say „tear it down" for Tester 1's media tests | **done** — R-1022 closed 2026-10-17 | `CLOSED-ITEMS.md` |
|
||||
| Hub master key into the password manager, then shred the file | **done** — `new.key` is gone; its folder is there (control) | `…/local-checks.txt` |
|
||||
| Hub 0.153.0 „no token can …" | not a task — a statement; moved to history | — |
|
||||
| Pull the failed disk Z6A07P2G | **not done** — still in DooPlex (lsblk; its twin is the control) | `…/local-checks.txt` |
|
||||
| The Facebook `first-post/` leftovers | **not done** — 15 files still there | same |
|
||||
| The `deny`-list edit in homelab-manifests | **not done** — still uncommitted | same |
|
||||
| Old „broken" line: a Longhorn volume at 91 % | **gone** — Alertmanager lists only the two R-934 disk alarms | same |
|
||||
| Old „broken" line: CI jobs get lost (R-887) | **gone** — R-887 closed 2026-10-19 (381 jobs, 0 lost) | `CLOSED-ITEMS.md` |
|
||||
|
||||
Every decided row (D1, D2, D10–D20) left the decide table only after its ruling was found in its closed row in
|
||||
`CLOSED-ITEMS.md`. The whole old page is in `STATUS-HISTORY.md` under „Moved from STATUS on 2026-10-20" (headings
|
||||
moved down two levels, otherwise unchanged).
|
||||
|
||||
## Register
|
||||
|
||||
- **Four numbers:** rows before **156**, after **153**, opened **1** (R-1174), closed **4**.
|
||||
- **Closed:** R-1121 (the rotation file is gone; control `~/.bashrc`), R-1118 (its last part, R-1134, closed
|
||||
2026-10-17; `sec doctor` all OK), R-231 (scripts in git and installed from it, the first run green; its same-disk
|
||||
facts live on in R-232) — evidence in `CLOSED-ITEMS.md`, `audits/status-view-2026-10-20/local-checks.txt`,
|
||||
`audits/answers-2026-10-12/s20-backup-scripts.txt`.
|
||||
- **Merged:** R-993 into R-991 (one decision, D5; the proposal and the „no host floor at install" fact copied into R-991).
|
||||
- **Narrowed:** R-132 (the hub password was rotated 2026-10-15; left: the break-glass hash field in the public repo);
|
||||
R-1049 → WATCHING (the fix is live since hub 0.151.0; it no longer waits on the operator).
|
||||
- **Owner cell fixed:** R-919 `CC` → `operator (upload + the look), then CC (close)` — its next action is the operator's.
|
||||
- **Operator-owned rows (43), still waiting on the operator?** yes for all, except: R-933 — **CC first** (the size
|
||||
measurement), then the operator's go; R-388 — **partly** (slice 1 is built; the scope decision is open);
|
||||
R-920 — parked by the operator's own choice (c). On the page: decide D3–D9, D21; do (still needed, the rest);
|
||||
the „Later" line for the remaining 23.
|
||||
- **Rows over 4.5 KB (not rewritten, §N.7):** R-336 7.6 KB, R-516 7.0, R-462 6.8, R-433 6.7, R-264 5.8, R-919 5.0,
|
||||
R-440 4.6, R-925 9.0.
|
||||
|
||||
## The gate — `scripts/status_view_gate.py` (R-1174)
|
||||
|
||||
- **Checks:** size outside the decide table (≤ 50 lines, ≤ 6,000 characters); one „Updated" paragraph (≤ 3 sentences,
|
||||
no „Earlier" chain); no Hungarian letter outside a quote; every cited R-n open; no decided/closed/done row in the
|
||||
decide table; the P1 row under „What is broken" or „decide"; every open row whose Owner names the operator under
|
||||
„decide" or „do". Exit 0 clean, 1 convicted, 2 inconclusive (no sections, missing section, no or empty register).
|
||||
- **Switch:** warn-first until **2026-10-27** (`SWITCH_DATE`), then a refusal; the date is also a due check for
|
||||
R-1174, so the next push on or after that date names it.
|
||||
- **Tests** (`scripts/test_status_view_gate.py`, 39 cases, all PASS): the genuine fixture before and after the switch;
|
||||
the clean run's „what it looked at" line; per check, decoys that warn before and refuse from the switch — size (50
|
||||
more lines; few lines and too many characters), updated (a prepended paragraph, an Earlier chain, four sentences, no
|
||||
paragraph), english (a Hungarian do-line), open-ids (a closed row cited), decide (a DECIDED row; a „done" pick), p1
|
||||
(the P1 line deleted; moved to What works), operator (the Later line deleted; a row only in „broken"); genuine
|
||||
controls that must pass (a long decide table, version numbers in a sentence, Hungarian inside a quote); four
|
||||
inconclusive cases; and the real `STATUS.md` against the real register.
|
||||
- **Red-proof:** each check removed in turn → exactly its own decoys FAIL (size 4, updated 8, english 2, open-ids 2,
|
||||
decide 4, p1 4, operator 4), then put back → 39/39. The switch moved to 2099 → the 14 „refuses" cases FAIL; moved
|
||||
one day earlier → the 14 „warns on 10-26" cases FAIL. Put back → 39/39.
|
||||
- **Run against the old page** (as it was at `2954b0be`): 75 convictions across all seven checks (81 lines, 25,472
|
||||
characters outside decide; 13 decided rows; 24 operator rows missing; the P1 only in „do").
|
||||
- Wired: `repo_gates.py` (fast set, docstring entry 19), `test_gate_decoys.py` (COVERS + runs the suite), found by
|
||||
`script_tests_gate.py`; `decoy_coverage_gate.py` reports it COVERED.
|
||||
|
||||
## The CLAUDE.md edit (repo `felhom.eu/CLAUDE.md`, end-of-session checklist, line 139–141)
|
||||
|
||||
- **Before:** „… One screen, written for the operator in plain language, and deliberately **not** `CONTEXT.md`."
|
||||
- **After:** the same, plus „(STE English) … How to update it:" and five numbered lines — replace the „Updated"
|
||||
paragraph (never prepend; the old one to `STATUS-HISTORY.md`); a morning list under a dated sub-heading in „do";
|
||||
remove what you finished; the decide-row rule; the gate checks it (R-1174).
|
||||
- **Why allowed:** a format rule, not a loosened safety rule (`PROMPT-TEMPLATE.md` §9 rule 9, decision 150).
|
||||
|
||||
## Commits and the running session
|
||||
|
||||
`2954b0be` register pass · `9ec5f76a` the page · `1d7071e8` the gate · `54d1ee82` the instruction · this report.
|
||||
**The rehearsal session shares this clone.** Its commit `f84f94d9` (audit evidence only) landed between my parts 1
|
||||
and 2; it touched neither `STATUS.md` nor the register, so there was nothing to merge. **Its `git push` carried my
|
||||
parts 2–4 to `origin` with its own** (the shared branch); its hook ran with `--scope=docs`. My own `--fast` run of the
|
||||
same content was green (rc 0), and `scripts/push_scope.py` classifies the full range correctly as CODE. Its decoy
|
||||
suite briefly planted changes in `website/gyik.html` and `scripts/felhom-host-install.sh` while I worked (R-928's
|
||||
shape); I staged only my own paths, and both files were clean again afterwards. `unproven.py --summary`: unchanged
|
||||
(NOT WALKED 35 of 55).
|
||||
|
||||
**Gate runs.** Every push: `repo_gates.py --fast`, rc 0 (the last one on the final tree, this report included). **Not
|
||||
followed: „the full run before the last push".** The full run adds exactly one gate, `iso-bootstrap`, which starts a
|
||||
container from an ISO image; on DooPlex that is a Docker run on production (workspace rule, and a standing feedback
|
||||
memory), and it checks only `scripts/iso/`, which this session did not touch.
|
||||
|
||||
## Observations
|
||||
|
||||
- „My commits were pushed by the other session's push, under a docs-scope hook" — **NOT-A-FINDING:** CI re-runs the
|
||||
whole fast set on every push whatever the hook's scope, and CI was green for every push of this session (by `head_sha` on the `jobs` endpoint, attempt 1 each): `2954b0be` job 2131, `54d1ee82` job 2132 (it carried parts 2–4), `c701b253` job 2133 — all `success`; the
|
||||
shared-clone behaviour is already recorded (R-928, the shared-worktree rule).
|
||||
- Tester4 shows DOWN with no host on the hub — **NOT-A-FINDING:** the rehearsal session reinstalls Tester4 today.
|
||||
- „A Longhorn volume at 91 %" had no row — **NOT-A-FINDING:** the alarm is no longer active (Alertmanager, with the two
|
||||
R-934 alarms as the control).
|
||||
|
||||
## Teardown
|
||||
|
||||
Provisioned nothing: no machine, no host, no hub write (GET only, CC's scoped token).
|
||||
@@ -736,6 +736,19 @@ stayed (37 tables, the live app had 36); the definition did not move (same versi
|
||||
answered 409. The restore page's „the database is back as it was" is false for it. Design note with the two options and
|
||||
the pick (A: a copy of the app's live volumes, put back on failure — one copy's disk for the restore's length, a fit
|
||||
check before the stop): `audits/night-2026-10-19/P3/design-R-893.md`; the decision is on `STATUS.md`.
|
||||
**`[DESIGN]` 2026-10-20 — „put back exactly as it was" (R-893 option A, operator decision D20 = A, `09` §3 decision 192
|
||||
second half; controller 0.336.0 is the floor).** Before the stop, every named volume the snapshot will replace is measured
|
||||
and its copy is added to the restore's own free-space need (R-357's exact rule); too little room or an unmeasurable volume
|
||||
refuses with nothing stopped. Right after the stop the volumes are tarred into `<namespace>/backups/restore-undo/<app>/`
|
||||
(outside every capture set). On ANY failure after a volume or the definition moved, the copies are poured back, a volume
|
||||
the snapshot added is removed, the live definition is written back and the app is STARTED; the household reads that the
|
||||
restore failed and nothing changed (with placed files: its database and volumes are back, the copied files stay); the
|
||||
operator gets `backup_run_failures` leg `restore-put-back`. The hold (`restore_mixed`) remains only for a put-back that
|
||||
fails — then the copy is kept for support — and the double failure (replay AND rollback) holds as before. On success the
|
||||
copy is removed. **`[FACT]` 2026-10-20 (9202, controller 0.336.0-rc.1, `audits/night2-2026-10-20/P2/live/`):** the same
|
||||
forced failure as on 2026-10-19 → 36 tables, the backup-only table absent, schema/data/row-count md5 equal to before,
|
||||
the app running, no hold, the copy (69 MB, 0.9 s) gone; a normal restore of the same snapshot then worked. The R-893
|
||||
known limit above is closed for named volumes; placed FILES are still not taken back (the plain-files mix, as ruled).
|
||||
|
||||
**[DESIGN] 2026-08-22 — the failure ladder of a database restore: replay → rollback → hold.**
|
||||
Recorded here rather than only in a closed register row, because a decision that survives only inside
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
# C1 reviewer 2 — delete and retention (fresh, read-only; clones at controller 7ca59e2)
|
||||
|
||||
| # | Sev guess | harm | mechanism |
|
||||
|---|---|---|---|
|
||||
| D1 | P1 (mech high, reach medium) | Shares second-drive copy: one drive group's reconcile deletes ANOTHER drive's share mirrors (on the night a drive is decommissioned/away; or every night when two groups target one drive, then a full re-copy) | `internal/backup/tier2_shares.go:135` groups by source; `:201-202` own destBase/legRels; `:230` `tier2ReconcileRoots(destBase, sharesTier2DestRoots(destBase), legRels)`; `:266-267` ReadDir lists every drive key on disk; `tier2.go:299-329` removes any child not in this group's legs; `shares_payload.go:233-251` away/decommissioned drives drop out |
|
||||
| D2 | P1 optional legs / P2 mandatory; design question with a defect corner | A missing or emptied app folder on the drive deletes its second-drive copy within one night; optional legs (audiobooks, comics, ROMs, Immich external library) are in no other tier | `tier2_capture.go:55-61` missing leg skipped (optional silently) → `tier2.go:455` reconcile deletes its mirror; run recorded success `:465`; emptied → `rsync -a --delete` `tier2.go:751` via `:443`; catalog optional legs `audiobookshelf/.felhom.yml:24-26`, komga, romm, immich `:32-34`; 07 §8.2 „data legs are NOT guarded" covers a leg leaving the class set, not a declared-but-missing leg |
|
||||
| D3 | P2 high (mech) | A host clock ahead during one whole-guest backup silences that tier until real time passes it; the hub reads the same clock and says OK | agent `internal/localapi/server.go:1178-1190` negative age never due; `internal/backup/runner.go:607-629` newest ctime unbounded; hub `monitor/deadline_tiers.go:155`, `deadline.go:243`; R-973 cap counts future entries `backup_cap.go:61`; restic: `offbox_window.go:133-136` pinned guard refuses every window → quota fills; NAS tier no guard `:209-216`; `offbox_abandon.go:248,287` set-aside rm on box clock |
|
||||
| D4 | P2 | A failed LOCAL unit restore leaves the app running on mixed data, no undo copy (same as R1 of reviewer 1 — independent agreement) | `restore_unit.go:430-489`; `writeSafetyDump` only in `offbox_reconstitute.go`; 07 §6.3 ladder not scoped to off-site |
|
||||
| D5 | P2 order defect / depth design | A dump that fails its own content check has ALREADY replaced last night's good dump; Tier-2 mirrors it an hour later | `appbackup/dbdump.go:388-397` size-only check, `:399` rename, `:419` `ValidateDump` after; `backup.go:688-690`; volume tars promoted with no content check `backup.go:902` |
|
||||
| D6 | P3 med | „Megőrzött adatok" can list a RUNNING app's live folder as deletable leftover if its app.yaml is unreadable | `stacks/manager.go:672`; `kept.go:220-235` skips `!Deployed || AppConfig==nil`; `ListKept :330-358`; `DeleteKept :385-402` RemoveAll; `deploy.go:1180-1190` write+rename without fsync |
|
||||
| D7 | P3 low-med | A unit restore can point the app back at an OLD drive; F5 then deletes the current drive's unit | `restore_unit.go:90-111, 507`; `cmd/controller/main.go:3252`; `deploy.go:820`; `backup.go:1517-1574` `pruneStalePrimaryDirs` even when this run's capture failed (`:726-733`) |
|
||||
| D8 | P3/P4 design | local whole-guest keep=1 prunes the only good archive before the new one is proven | agent `runner.go:229`, `:139-161`, restore-test settle 24 h `:462` |
|
||||
|
||||
**Checked SAFE (reviewer):** hub off-site deletions by exact identity (`offsitekeys.go:366-515`); pinned off-site retention guard refuses future/too-new snapshots and big plans (`offbox_window.go:132-161`); dump and tar writes crash-safe (tmp+fsync+rename); R-403 hollow-unit guard; drive migration deletes source only after verify (`stacks/migrate.go:614-667`); app removal refusals and protected folders (`delete.go:136-153, 241-301, 839-846`); mirror/undo clean-ups allow-listed/prefix-guarded; agent prune never on PBS unless opted in; restore-test skips unsettled/future/foreign-key archives; `ForgetRetiredDrives` mount units only; SMB share removal deletes no folder. Not reviewed: agent `vzpartial` janitor, tenantsync/ep0 prune config, hub RESET beyond `PurgeRepos`.
|
||||
@@ -0,0 +1,14 @@
|
||||
# C1 reviewer 3 — keys that gate recovery (fresh, read-only; controller 7ca59e2, hub 5c48731, agent 4e1621c)
|
||||
|
||||
| # | Sev guess | harm | mechanism |
|
||||
|---|---|---|---|
|
||||
| K1 | P2 (P1 with K2 / a RESET) high | **A reinstall through „delete the old host first" (D19/R-1154) cuts the new box off from the household's escrow: the recovery code is never asked, a NEW key is minted, and the old off-site history is set aside by itself on night one; typing the old code answers „wrong code"** | hub `api/passphrase_scope.go:58` 409 forces the delete; `store.go:3204-3210` delete demotes escrow under the OLD host_id; `handler.go:1261` new host_id; `GetEscrowStatusForCustomer` `store.go:3500-3527` joins on `hosts` → nil (superseded join on deleted host also gone) → `identity_blob_present`/`superseded_present` false; controller `offbox.go:542-555` mints new pw, `:1788` no recovery offer, `:905-906` firstNight → `resetOrphanedRepo`; `handleHostEscrowRetainedGet` 404 for a deleted host (`handler.go:1691`); contrast purge `store.go:4185-4190` DOES include `host_deletions` |
|
||||
| K2 | P2 high | Abandoning ONE set-aside copy deletes EVERY retained recovery package of the customer — other set-aside copies become ciphertext nobody can open | hub `handler.go:626-627` → `PurgeSupersededEscrowForCustomer` `store.go:4185-4190` deletes all rows for the customer (incl. deleted hosts); the comment at `:4176-4179` states the invariant it breaks |
|
||||
| K3 | P3 med-high | A ceremony while off-site is not configured seals a package with NO repo password and supersedes the one that had it | controller `escrow_handlers.go:217` re-stage only when configured; agent `main.go:2975-2981`; hub `store.go:3392`; `escrow_confirm.go:222-258` stale card local only |
|
||||
| K4 | P3 med | The global „create your recovery code" bar links to the ceremony the backups page hides while a recovery offer is outstanding | `backups_remote.html:144-160` vs `escrow_banner.go:23-33` + hu.json:1682; no `recoveryOffer()` refusal in the wizard/start handlers; settle gate `main.go:876`, `offsiteapply.go:84` |
|
||||
| K5 | P3 med-high | After a reinstall recovered on the reuse path, the NEW PBS key K is never escrowed and nothing compares fingerprints | `felhom-pbs-apply:99` autogen; `escrow_confirm.go:182-205` auto-confirm on hash; no compare of `host_escrow.key_fingerprint` vs live key (agent `internal/backup/runner.go:407-429` only) |
|
||||
| K6 | P3/P4 design | Whole-guest off-site (PBS) copies run before any escrow exists; host lost before the ceremony → unreadable archives | controller gates Tier 3 (`offbox.go:781`); agent `internal/backup`, `internal/pbsdr` have no escrow gate |
|
||||
| K7 | P4 low | escrow PUT split in two statements; a failure leaves a row whose hash claims a key its blob lacks | hub `handler.go:1388, 1412-1420`; `store.go:3402-3408` |
|
||||
| K8 | P4 | operator off-site re-issue resets the provider password before saving it | hub `offsite.go:217-245` |
|
||||
|
||||
**Checked SAFE (reviewer):** mint guard while the hub holds a package (`offbox.go:542-555, 659-720`); orphaned repo moved aside, never re-inited/deleted (`offbox.go:886-943, 332-397`); supersession retains both blobs (`store.go:3343-3416`); host delete demotes not destroys (`store.go:3164-3229`); ceremony two-word confirm; `escrow.Create` self-verifies; recovery refuses to replace a different local key; `ClearOffboxTarget`; PBS wrapper has no delete verb; hub abandon delete limited to one path after a delay (`offsitekeys/service.go:464-496`).
|
||||
@@ -0,0 +1,24 @@
|
||||
# C1 reviewer 1 — restore and rollback (fresh, read-only; clones at controller 7ca59e2 / v0.336.0)
|
||||
|
||||
Raw candidates as returned by the reviewer, condensed; each is UNVERIFIED until C2 marks it. File:line are in the controller unless named.
|
||||
|
||||
| # | Sev guess | harm | mechanism |
|
||||
|---|---|---|---|
|
||||
| R1 | P2 high | A local / second-drive unit restore that fails halfway restarts the app on half-replaced data, no undo copy; the next night's capture then overwrites the only restore point with the broken state | `internal/backup/restore_unit.go:429-431` failed stop only warns; `restore.go:217-218` `docker volume rm -f` error ignored before untar (10-min limit `:237-241`); `restore_unit.go:466-486` DB replay failure → app started anyway; one restore point per app, promoted in place (`atomicPromoteTar`) |
|
||||
| R2 | P2 high | A power cut / controller restart during a restore starts the app on a mixed state; the undo copy is never used at boot | `offbox_reconstitute.go:942` marker → `appstop_marker.go:307` StartStack with no restore check; `offbox_restore_undo.go:137` undo dir only dropped, never read at start; unit restore and .fab import set no marker → `bootrecon/bootrecon.go:210-213,269` starts the app; updates have a journal (`stacks/update.go:1420-1489`), restores do not |
|
||||
| R3 | P2 high | A cut-off shared-folder restore can be „placed" and reported as restored while files are missing; scratch deleted | `shares_restore.go:138` readiness = `rErr == nil && len(entries) > 0` (R-358 class, not fixed for shares); 30-min ctx `web/offbox_handlers.go:716`; place reports N files and deletes scratch `:301`; free-space gate fails open and fixed 2 GiB `:107` |
|
||||
| R4 | P2 high | A `.fab` export silently drops a whole drive folder that contains a symlink; import never sets owners | `appexport/fabplan.go:292` `tar.FileInfoHeader(info,"")` then `os.Open`+`io.Copy` on a symlink; `export.go:714` warn-and-continue; `assertBundleDataComplete` refuses only when no data at all; `restore.go:1157-1176` extracts only dirs/regular files, no chown |
|
||||
| R5 | P2 med-high | Second-drive whole restore swaps files, then refuses the DB half, leaves the app STOPPED and says „the app is untouched" | `tier2_whole.go` stops + merges first, then `RestoreFromRecoveryUnitAtWith` whose refusals (`restore_unit.go:366-371, 386-389, 413-416, 420-423`) assume nothing happened; en.json:2667 „the restore did not start. The app is untouched" |
|
||||
| R6 | P2 med | `.fab` import into a STOPPED app merges into existing volumes; the default export (stop box ticked) has no SQL dump | `appexport/restore.go:361-374` volumes removed only if running; `docker cp` merge `:721`; `templates/app_data.html:221` stop default; `export.go:262` stop before `dumpDatabase` `:366`; `appbackup/dbdump.go:142` discovers only running containers |
|
||||
| R7 | P2/P3 high | `.fab` import has no concurrency guard, no undo, no record | `restore.go:181-186`; `web/handler_export.go:312-353` no `restoreOpBlocked()`; `RemoveStackVolumes` before anything else, no copy |
|
||||
| R8 | P2 med | `.fab` import drops the family gate, sign-up lock and version pin → a family-only app opens to anyone | `cmd/controller/main.go:3517-3530` bare `AppConfig`; `stacks/family_gate.go:173-190` removes the gate file when `FamilyGate == nil`; R-773 fix only on the unit path (`stacks/deploy.go:840-870`) |
|
||||
| R9 | P3 med | `.fab` import trusts the bundle's HDD_PATH; empty → relative path; older userdata overwrites newer | `restore.go:589-631` |
|
||||
| R10 | P3 high | `.fab` DB import: plain psql without ON_ERROR_STOP; postgis not classified | `restore.go:1052-1053`, `:928-937` |
|
||||
| R11 | P3 med | volume replace never checks the old volume is gone | `restore.go:218` |
|
||||
| R12 | P3 high | interrupted second-drive whole restore leaves `<dst>.felhom-restoring` (O_EXCL) → every retry fails „file exists" | `tier2_whole.go:86` |
|
||||
| R13 | P3 design | full off-site download capped at 30 min per press; household not told to retry | `web/offbox_handlers.go:455` |
|
||||
| R14 | P4 med | place/full restore re-read the LATEST snapshot instead of the downloaded one | `offbox_restore.go:675-679`, `offbox_reconstitute.go:657-661` |
|
||||
|
||||
**Inventory drift (July → now):** three more customer restore entries exist: the Tier-2 unit restore (R-102), the Tier-2 whole restore (decision 26, `tier2_whole.go`), the kept-data Load (`kept_load.go`).
|
||||
|
||||
**Checked SAFE (reviewer):** `.fab` path confinement (`appexport/restore.go:1104-1107,1152-1155`; `validate.go:28-66`; `web/handler_export.go:385-394`); off-site placement map refuses `..`/outside/`backups/` (`offbox_restore.go:603-640`); R-358 marker after restic success; full off-site restore fit/safety-dump/undo checks before stop, fail closed (`offbox_reconstitute.go:719-818, 909-927`); Tier-1 refusals before change; main loader `ON_ERROR_STOP=1 --single-transaction` (`appbackup/dbdump.go:775-784`); Tier-2 file restore `rsync --ignore-existing`, never `--delete`; whole restore newer-wins + Lchown; **NFC/NFD: no normalisation anywhere — every lane compares bytes, so the missing-only compare sees TWO names (consistent on one filesystem)**; agent RestoreLXC force only when asked. Not reviewed: `kept_load.go`, hub operator restore paths.
|
||||
@@ -0,0 +1,40 @@
|
||||
# C1 reviewer 4 — time and concurrency (fresh, read-only; controller 7ca59e2, agent 4e1621c, felhom.eu 5c48731)
|
||||
|
||||
## (A) Schedulers and what each does on Sunday 2026-10-25 (03:00 CEST → 02:00 CET; 02:00–02:59 happens twice)
|
||||
|
||||
| owner | job | schedule | zone | file:line | 2026-10-25 | after 3 days off |
|
||||
|---|---|---|---|---|---|---|
|
||||
| controller | db-dump | daily W (default 02:30) | Budapest (`time.Date`, day+1) | main.go:1262, scheduler.go:499-518 | once, at the SECOND 02:30 (CET = 01:30Z) — Go normalises the ambiguous time to the later offset | not replayed by the scheduler; the catch-up runs it once 15 min after start (nightchain.go:277-345) |
|
||||
| controller | tier2-backup | W+60m | Budapest | main.go:1352 | once, 03:30 CET | once, by the catch-up |
|
||||
| controller | offbox-backup + app-update leg | W+105m | Budapest | main.go:1519 | once, 04:15 CET | catch-up runs the off-site leg only |
|
||||
| controller | offsite-abandon-sweep 05:10, offsite-proof 05:30, offsite-integrity 06:00 | fixed | Budapest | main.go:1561, 1613, 1591 | once | **no catch-up** (finding T6) |
|
||||
| controller | metrics-prune 04:00, fill-watch 03:30, asset-sync 05:00, selfupdate-auto 04:30 (off by default) | fixed | Budapest | main.go:1621,1671,1768,1741 | once | floor update runs on any hub reply |
|
||||
| controller | whole-guest quiesce loop | poll 5 min; gate [W+2h, W+6h) by wall minute; valve when age > cadence+24h | Budapest wall | quiesce.go:150, 191-205, 790-818 | gate 04:30–08:30 CET (4 h real) | valve fires ~5 min after start, any time of day |
|
||||
| controller | ~20 `Every` tickers | monotonic | n/a | main.go:898-2146 | unaffected | restart at boot |
|
||||
| agent | whole-guest due | age ≥ cadence | UTC | localapi/server.go:1182-1192 | unaffected | due at once |
|
||||
| agent | restore-test | 30 min after start, then 6 h | UTC | backup/schedule.go:148-177 | unaffected | after 30 min |
|
||||
| agent | OS leg (+ kernel reboot if hub „Tonight") | after any non-manual primary backup +90 s, 20 h min gap | no wall clock | server.go:982-986, main.go:1272-1280, leg.go:655-664, kernel.go:101-114 | follows the backup | follows the valve backup = daytime (T8) |
|
||||
| agent | fstrim | hourly; due Wed 10:00, only 10:00–20:59 | host local | fstrim/fstrim.go:39-50,110 | n/a | next eligible hour |
|
||||
| hub | scheduleDaily: db-snapshot 02:00 (keep 2), prune 04:30, deadline-check 05:00, offsite-key-audit 07:10 | `next.Add(24*time.Hour)` | Budapest | hub cmd/hub/main.go:1049-1072 | **every job runs TWICE** (T9) | n/a |
|
||||
| host timers | mgmt-watchdog, crash-guard-check | monotonic | n/a | felhom-agent/configs/*.timer | unaffected | after boot |
|
||||
| DooPlex timers | hub-db-backup 02:30; dooplex-offsite 00:20; hub-db-restore-test Sun 04:30; offsite-restore-test Sun 05:30; ep0-copy-gc 08:00 | OnCalendar local | DooPlex | felhom.eu/scripts/*/*.timer | Oct 25 is a Sunday: both weekly restore tests run on the DST day; 02:30 is in the repeated hour (systemd behaviour not verified) | Persistent=true |
|
||||
|
||||
## (B) Candidates
|
||||
|
||||
| # | Sev guess | harm | mechanism |
|
||||
|---|---|---|---|
|
||||
| T1 | P2 med-high | The nightly second-drive mirror (`rsync --delete`) can run DURING a restore from that same mirror and delete the files not yet copied back | `tier2.go:479-520` RunAllTier2 checks held/version/disconnected only; `:443` mirror, `:744-751` `--delete`; restore `tier2_whole.go:301` takes only `acquireRunning`; Tier-2 leg takes only `LegLock` (main.go:1348-1352); `update_guard.go:595-601` isHeld ignores restores |
|
||||
| T2 | P2 if true; low-med | The Debian 13 controller image may lack tzdata → every schedule on UTC silently | `Dockerfile:77-100` debian:13.7-slim, no tzdata, no `time/tzdata` import; fallbacks scheduler.go:19-23 (logs), quiesce.go:749-751 (silent), funcmap.go:62-66 |
|
||||
| T3 | P3 med | Whole-guest start creeps ~5 min/night until a night is skipped silently | agent `server.go:1184` `age >= Cadence` no slack; age from archive start; controller poll 5 min `quiesce.go:150`; gate refusal `:807-818` |
|
||||
| T4 | P3 med-high | The whole-guest quiesce stops apps in the middle of a restore or an update | `quiesce.go:213-275` no `IsRunning()`/restore check; `RunningAppStacks` (stacks/manager.go:620-636); `TriggerNow` (:419-445) same; `DumpAppVolumesSafe` (backup.go:1001-1045) can restart an app quiesce stopped |
|
||||
| T5 | P3 high | The nightly off-site copy is skipped SILENTLY when anything else holds the backup flag, and the catch-up never retries | `offbox.go:981-996` INFO „skipped — another backup is running" return nil; `withLeg` MarkEnded unconditionally (main.go:1246); proof 05:30 / integrity 06:00 take the same flag |
|
||||
| T6 | P3 high | Off-site check and proof never run on a box off at 05:30–06:00; the code comment says the opposite | main.go:1575-1577 comment vs 1591-1619; `catchUpLegs` nightchain.go:17-18; no hub alarm on integrity/proof age |
|
||||
| T7 | P3 high | Hub raises a false „no DB dump" error every morning for W after ~04:45 or evening W | hub `monitor/deadline.go:416-419` midnight Budapest; `:556-565` → `expected_dbdump_missed`; hub has no idea of W |
|
||||
| T8 | P3 design | The OS leg and kernel reboot follow ANY scheduled backup, including the daytime valve after power-on | agent server.go:982-986; main.go:1279 labels „night"; kernel.go:101-114; hub osupdates/kernel.go:46-48,181-185 |
|
||||
| T9 | P4 high | Hub daily jobs run TWICE on 25 October (db-snapshot keep-2 prunes a day early; duplicate deadline alarms); once an hour late in spring | hub cmd/hub/main.go:1058-1062 `next.Add(24*time.Hour)`; the controller fixed this (scheduler.go:511-516) |
|
||||
| T10 | P4 high | Spring forward 2027-03-28: db-dump and tier2 fire at the same instant; tier2 may mirror yesterday's unit | scheduler normalisation; `LegLock` main.go:1237 plain mutex |
|
||||
| T11 | P4 design | A clock ahead stops whole-guest backups silently (same as D3) | server.go:1182-1192; quiesce.go:816-817; deadline.go:244; backup_cap.go:61 |
|
||||
| T12 | P4 high | self-update busy check before the image pull, swap after it with no re-check | updater.go:532-545, :764 |
|
||||
| T13 | P4 med | double-clicking a restore shows it as failed while it runs | handlers.go:1853, :1874 `BeginRestoreOp` after the check; second POST's `EndRestoreOp(false)` |
|
||||
|
||||
**Checked SAFE (reviewer):** controller DST day+1 arithmetic and late-fire skip (scheduler.go:342-348, 511-516); night ledger persisted, seeded, one catch-up for many nights; quiesce marker persisted and recovered; in-memory backup flag (no stale lock); flock-based OS-leg locks; restic stale locks self-heal; two restores blocked by `m.running`; Remove refused during restore/quiesce; catch-up and whole-guest wait for each other; agent due/cap UTC durations; off-site prune guard refuses future snapshots; host power-off re-checks; no cron/OnCalendar in the installer; hub staleness uses the hub's own receive clock.
|
||||
@@ -0,0 +1,22 @@
|
||||
# C2 register search, 2026-10-20T10:11:13Z, OPEN-ITEMS.md + CLOSED-ITEMS.md at c701b253; grep -ciE
|
||||
1 open / 2 closed : R-893
|
||||
0 open / 0 closed : tier2_shares|_shares.*tier|shares.*second.drive|Megoszt.*2\. ment
|
||||
0 open / 2 closed : host_deletions|retained.*delet|deleted host|escrow.*reinstall
|
||||
2 open / 7 closed : scheduleDaily|DST|daylight|oraallit|time.?zone
|
||||
1 open / 8 closed : restore-undo|undo copy|safety dump|unit restore.*fail
|
||||
1 open / 0 closed : ValidateDump|dump.*replac.*valid
|
||||
0 open / 0 closed : SharesScratchReady|shares.*scratch|shares restore
|
||||
0 open / 4 closed : tier2_whole|whole restore|Teljes vissza
|
||||
0 open / 3 closed : optional leg|missing on disk|--delete
|
||||
0 open / 0 closed : PurgeSuperseded|abandon.*purge
|
||||
0 open / 1 closed : future.dated|clock ahead|clock.*future|negative age
|
||||
0 open / 0 closed : another backup is running
|
||||
1 open / 2 closed : expected_dbdump_missed
|
||||
2 open / 1 closed : symlink
|
||||
0 open / 1 closed : family.gate.*fab|fab.*family|\.fab
|
||||
0 open / 0 closed : felhom-restoring
|
||||
0 open / 0 closed : quiesce.*restore|restore.*quiesce
|
||||
1 open / 4 closed : offsite-proof|integrity.*catch|proof.*catch
|
||||
0 open / 0 closed : double.click|dupla
|
||||
0 open / 1 closed : disconnected.*target|away.*target
|
||||
Every non-zero hit was read: none describes the candidate (positive control R-893 found in both).
|
||||
@@ -0,0 +1,51 @@
|
||||
# C2 — the main session's verdict on every candidate (2026-10-20)
|
||||
|
||||
Method: each candidate's `file:line` read in LIVE source (controller `7ca59e2` = v0.336.0, agent `4e1621c`, felhom.eu
|
||||
`5c48731`+); then (a) the register searched, open AND closed, with ASCII fragments — positive control `R-893` found in
|
||||
both; (b) the design test against `07-backup-architecture.md` and `CONTEXT.md`. Verdict words:
|
||||
|
||||
- **CONFIRMED** — the mechanism read true in live source AND a proof ran (unit test against production code, or live).
|
||||
- **CONFIRMED-CODE** — the mechanism read true in live source; no proof ran yet.
|
||||
- **PLAUSIBLE** — the reviewer's reading, spot-checked only.
|
||||
- **NOT-A-FINDING** — with the reason.
|
||||
|
||||
Register: none of the candidates below is in OPEN-ITEMS or CLOSED-ITEMS (searches in `C2-register-search.txt`).
|
||||
|
||||
| id | verdict | design or defect | note |
|
||||
|---|---|---|---|
|
||||
| **D1** shares second-drive copy pruned by another drive's group | **CONFIRMED** (unit test, production code, fake mirror seam — `C3-D1-shares-tier2/`) | defect | **Worse than reported:** with shares on two drives that pick the same target, EVERY night the second group's reconcile deletes the first group's copy (`removed stale dest dir hdd_2/b_share`) — one drive's shares never have a second copy while the run says ok. And the night a drive goes away, its copy is deleted (`removed stale dest dir hdd_3/csaladi_fotok`). |
|
||||
| D1b an AWAY drive is still chosen as the second-drive target | CONFIRMED-CODE (+ the same unit test: `hdd_3/iskola => hdd_1/...` after hdd_1 was marked disconnected) | defect | `tier2.go:151-180` loops `GetSchedulableStoragePaths()` (`settings.go:1593-1603` excludes only Decommissioned); no Disconnected check. On a real box the away drive's empty mount point is on the guest's root disk |
|
||||
| **K1** reinstall after „delete the old host" loses the route to the old escrow | **CONFIRMED** at the hub store (unit test against production code, `C3-K1-escrow-after-host-delete/`); the controller half (new key minted, history set aside on night one) CONFIRMED-CODE | defect (03 §8a / 07 §7.0 promise the route; D19+R-1154 removed it unrecorded) | hub `store.go:3164-3229` demotes into `host_escrow_superseded` under the OLD host_id and deletes the `hosts` row; `GetEscrowStatusForCustomer` `store.go:3499-3527` joins on `hosts` → nil for the new box; `customer_reset.go:48,155` and `store.go:4189` DO include `host_deletions`. Live proof needs Tester4 off-site ON (B7) |
|
||||
| K2 abandoning one set-aside copy purges every retained package | CONFIRMED-CODE (`store.go:4185-4190`) → R-1182 | defect (breaks `store.go:4176-4179`'s own invariant) | `store.go:4185-4190` customer-wide delete read true |
|
||||
| **R1/D4** a failed LOCAL unit restore: no copy of the live state, app started anyway | **CONFIRMED LIVE** on 9202 (`C3-R1-local-restore-9202/`): live row destroyed, DB crash-loops, no undo copy, raw English Docker text to the household | **design question** for the operator: decision 154 (R-638 A) chose „start anyway"; D20 (R-893 A) gave „put back exactly" to the OFF-SITE path only | `restore_unit.go:428-489` |
|
||||
| R2 a crash mid-restore starts the app on a mixed state; the undo copy is never read at start | PLAUSIBLE | defect | `appstop_marker.go:307`, `offbox_restore_undo.go:137` |
|
||||
| R3 a cut-off shares restore can be placed and reported restored | CONFIRMED-CODE | defect (R-358 class, fixed for apps, not shares) | `shares_restore.go:131-140` readiness = non-empty dir |
|
||||
| R4 `.fab` export drops a drive folder holding a symlink | PLAUSIBLE | defect | `appexport/fabplan.go:292`, `export.go:714` |
|
||||
| R5 second-drive whole restore: files swapped, then the unit half refuses → app left STOPPED, text says „untouched" | **CONFIRMED** (unit test, production code: `C3-R5-whole-restore/` — 1 file restored + 1 replaced, app stopped, no start, message „az alkalmazás és az adatai érintetlenek") | defect | `tier2_whole.go:301-340`: stop + merge, then `RestoreFromRecoveryUnitAtWith` whose early refusals return before its own stop/start; `hu.json` `err.backup.unit_versions_mixed` „Az alkalmazás érintetlen" |
|
||||
| R6 `.fab` import into a stopped app merges volumes | PLAUSIBLE | defect | `appexport/restore.go:361-374` |
|
||||
| R7 `.fab` import has no guard / undo / record | PLAUSIBLE | defect | |
|
||||
| R8 `.fab` import drops the family gate | **CONFIRMED LIVE** on 9202 (`C3-R1187e-fab-family-gate/`: stranger 401 before, 200 after the import) → **R-1191 (P1)** | defect | `cmd/controller/main.go:3517-3530`, `stacks/family_gate.go:173-190` |
|
||||
| R9–R11, R13, R14 | PLAUSIBLE | defect / design (R13) | low |
|
||||
| R12 interrupted whole restore leaves `*.felhom-restoring`, every retry fails | **CONFIRMED** (unit test: `C3-R12-leftover-temp/`) → R-1188 (c) | defect | `tier2_whole.go:86-87` O_EXCL; removed only on an in-process error |
|
||||
| **D2** a missing / emptied app folder deletes its second-drive copy within one night | **CONFIRMED** for a MISSING folder (unit test, production code + real copy seam: `C3-D2-missing-leg/` — mirror of `photos` deleted, run „ok", the warning says only „nem kerültek a másodlagos mentésbe"); the EMPTIED case CONFIRMED-CODE (`rsyncMirror` = `rsync -a --delete`, `tier2.go:744-751`; the test seam does not delete) | defect-shaped corner of a choice: 07 §8.2 „data legs are NOT guarded" is about a leg LEAVING the class set; this is a declared leg that is MISSING, which the code itself logs as a gap (`tier2_capture.go:55-61`) and then reconciles away (`tier2.go:454`) | P1 for optional legs (no other tier holds them) |
|
||||
| D3 / T11 a host clock ahead silences the whole-guest tier; the hub reads the same clock | PLAUSIBLE (B6 #8 on the rehearsal VM is its live test) | defect | agent `localapi/server.go:1178-1190` |
|
||||
| D5 a dump that fails its own check has already replaced the good one | **CONFIRMED** (unit test with a docker stub: `C3-D5-dump-replaced/`) → R-1185 | defect for the ORDER | `appbackup/dbdump.go:388-422`: size check, rename (`:399`), then `ValidateDump` |
|
||||
| D6 kept-data page can list a running app's folder | PLAUSIBLE | defect | needs an unreadable app.yaml |
|
||||
| D7 unit restore can point the app at an old drive | PLAUSIBLE (low) | defect | |
|
||||
| D8 local whole-guest keep=1 | design (07 §6.1 R-684 chose it) | design | **NOT-A-FINDING: the operator chose keep=1 for demo-hp; recorded in 07** |
|
||||
| T1 the nightly mirror (`rsync --delete`) can run during a restore from that mirror | **CONFIRMED** (unit test, production rsync mirror: `C3-T1-mirror-during-restore/`) → R-1180 | defect | `RunAllTier2` (`tier2.go:479-520`) takes no backup flag; the whole restore holds only `acquireRunning` (`tier2_whole.go:301`) |
|
||||
| T2 controller image without tzdata → UTC schedules | **NOT-A-FINDING:** measured on 9202 (controller 0.331.0, Debian 13 base): `/usr/share/zoneinfo/Europe/Budapest` present, scheduler logs „scheduled for … CEST" | — | |
|
||||
| T3 whole-guest start creeps ~5 min/night | PLAUSIBLE | defect | needs the agent journal over weeks |
|
||||
| T4 whole-guest quiesce stops apps mid-restore | PLAUSIBLE | defect | `quiesce.go:213-275` |
|
||||
| T5 nightly off-site copy skipped SILENTLY when the flag is held; catch-up never retries | CONFIRMED-CODE → R-1186 | defect (the code chose nil for the nightly path; the catch-up gap is not chosen) | `offbox.go:981-996`, `main.go:1246` |
|
||||
| T6 proof / integrity never run on a box off at 05:30–06:00 | PLAUSIBLE | defect (comment `main.go:1575-1577` says the opposite) | |
|
||||
| T7 hub false „no DB dump" for late or evening W | PLAUSIBLE | defect | hub `monitor/deadline.go:416-419, 556-565` |
|
||||
| T8 the OS leg follows a daytime valve backup | design question | design | 07 §6.1 / 11 §8.1 set no time bound |
|
||||
| **T9** hub daily jobs run TWICE on 2026-10-25 | **CONFIRMED** (replay of the hub's exact arithmetic, `C3-T9-hub-dst/replay.txt`) | defect | deadline-check fires 04:00 CET AND 05:00 CET; db-snapshot twice (keep-2 → the previous day's copy pruned a day early). **In five days, on the live hub.** |
|
||||
| T10 spring forward collision | PLAUSIBLE | defect (P4) | |
|
||||
| T12, T13 | PLAUSIBLE | defect (P4) | |
|
||||
| K3–K8 | PLAUSIBLE | K6 design | |
|
||||
|
||||
**NFC/NFD (the brief's question for A.1.4):** reviewer 1 found no normalisation anywhere — rsync, restic and the Go
|
||||
merge compare names byte for byte, so the missing-only compare sees **two names** for the NFC/NFD pair. The household
|
||||
data carries the pair; B8 measures it live.
|
||||
@@ -0,0 +1,26 @@
|
||||
# go test at controller main 7ca59e2 (v0.336.0), scratch clone, fake mirror seam (no rsync/docker); 2026-10-20T10:04:31Z
|
||||
=== RUN TestRehearsalD1_DeadDriveShareCopyIsPruned
|
||||
zz_rehearsal_d1_test.go:46: drive keys: map[hdd_1:hdd_1 hdd_2:hdd_2 hdd_3:hdd_3]
|
||||
zz_rehearsal_d1_test.go:47: POSITIVE CONTROL — copies after run 1 (both drives present): [hdd_1: hdd_3/iskola hdd_2: hdd_1/csaladi_fotok]
|
||||
zz_rehearsal_d1_test.go:48: mirror calls run 1: [hdd_1/csaladi_fotok=>hdd_2/backups/secondary/_shares/hdd_1/csaladi_fotok data/shares-payload=>hdd_2/backups/secondary/_shares/_payload hdd_3/iskola=>hdd_1/backups/secondary/_shares/hdd_3/iskola data/shares-payload=>hdd_1/backups/secondary/_shares/_payload]
|
||||
zz_rehearsal_d1_test.go:56: FAULT — copies after run 2 (hdd_1 away): [hdd_1: hdd_3/iskola hdd_2: hdd_1/csaladi_fotok]
|
||||
zz_rehearsal_d1_test.go:57: mirror calls run 2: [hdd_3/iskola=>hdd_1/backups/secondary/_shares/hdd_3/iskola data/shares-payload=>hdd_1/backups/secondary/_shares/_payload]
|
||||
--- PASS: TestRehearsalD1_DeadDriveShareCopyIsPruned (0.02s)
|
||||
=== RUN TestRehearsalD1_TwoGroupsOneTarget
|
||||
zz_rehearsal_d1_test.go:73: copies after two runs: [hdd_1: hdd_3/c_share hdd_2: hdd_1/a_share]
|
||||
zz_rehearsal_d1_test.go:75: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_1 → hdd_2/backups/secondary/_shares (18 B)
|
||||
zz_rehearsal_d1_test.go:75: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_3 → hdd_1/backups/secondary/_shares (36 B)
|
||||
zz_rehearsal_d1_test.go:75: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_1 → hdd_2/backups/secondary/_shares (18 B)
|
||||
zz_rehearsal_d1_test.go:75: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_3 → hdd_1/backups/secondary/_shares (36 B)
|
||||
--- PASS: TestRehearsalD1_TwoGroupsOneTarget (0.01s)
|
||||
=== RUN TestRehearsalD1_SameTargetThenDriveDies
|
||||
zz_rehearsal_d1_test.go:89: POSITIVE CONTROL run 1: [hdd_1: hdd_3/csaladi_fotok]
|
||||
zz_rehearsal_d1_test.go:91: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_2 → hdd_1/backups/secondary/_shares (18 B)
|
||||
zz_rehearsal_d1_test.go:91: LOG [INFO] [backup] Tier 2 reconcile: removed stale dest dir hdd_2/b_share
|
||||
zz_rehearsal_d1_test.go:91: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_3 → hdd_1/backups/secondary/_shares (42 B)
|
||||
zz_rehearsal_d1_test.go:100: FAULT run 2 (hdd_3 away): [hdd_1: hdd_2/b_share]
|
||||
zz_rehearsal_d1_test.go:102: LOG [INFO] [backup] Tier 2 reconcile: removed stale dest dir hdd_3/csaladi_fotok
|
||||
zz_rehearsal_d1_test.go:102: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_2 → hdd_1/backups/secondary/_shares (18 B)
|
||||
--- PASS: TestRehearsalD1_SameTargetThenDriveDies (0.01s)
|
||||
PASS
|
||||
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.049s
|
||||
+104
@@ -0,0 +1,104 @@
|
||||
package backup
|
||||
|
||||
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate D1). Never committed.
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"sort"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func listShareCopies(env *sharesEnv) []string {
|
||||
var out []string
|
||||
for name, d := range env.drives {
|
||||
base := filepath.Join(NamespaceRoot(d, true), "backups", "secondary", SharesPseudoStack)
|
||||
keys, _ := os.ReadDir(base)
|
||||
for _, k := range keys {
|
||||
if !k.IsDir() || k.Name() == filepath.Base(sharesPayloadDestRel) {
|
||||
continue
|
||||
}
|
||||
subs, _ := os.ReadDir(filepath.Join(base, k.Name()))
|
||||
for _, s := range subs {
|
||||
out = append(out, name+": "+k.Name()+"/"+s.Name())
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Strings(out)
|
||||
return out
|
||||
}
|
||||
|
||||
func keyOwner(env *sharesEnv) map[string]string {
|
||||
m := map[string]string{}
|
||||
for name, d := range env.drives {
|
||||
m[sharesDriveKey(d)] = name
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// A drive whose shares were mirrored goes away; another share group's run lands on the same target.
|
||||
func TestRehearsalD1_DeadDriveShareCopyIsPruned(t *testing.T) {
|
||||
env := newSharesEnv(t, "hdd_1", "hdd_2", "hdd_3")
|
||||
env.addShare(t, "hdd_1", "csaladi_fotok", true)
|
||||
env.addShare(t, "hdd_3", "iskola", true)
|
||||
if err := env.m.RunSharesTier2(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Logf("drive keys: %v", keyOwner(env))
|
||||
t.Logf("POSITIVE CONTROL — copies after run 1 (both drives present): %v", listShareCopies(env))
|
||||
t.Logf("mirror calls run 1: %v", env.mirrored)
|
||||
if err := env.sett.SetDisconnected(env.drives["hdd_1"], true, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
env.mirrored = nil
|
||||
if err := env.m.RunSharesTier2(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Logf("FAULT — copies after run 2 (hdd_1 away): %v", listShareCopies(env))
|
||||
t.Logf("mirror calls run 2: %v", env.mirrored)
|
||||
}
|
||||
|
||||
// Shares on two drives that both pick the SAME target: does group A's reconcile remove group B's copy?
|
||||
func TestRehearsalD1_TwoGroupsOneTarget(t *testing.T) {
|
||||
env := newSharesEnv(t, "hdd_1", "hdd_2", "hdd_3")
|
||||
env.addShare(t, "hdd_1", "a_share", true)
|
||||
env.addShare(t, "hdd_3", "c_share", true)
|
||||
var removed []string
|
||||
env.m.logger = newCaptureLogger(&removed)
|
||||
if err := env.m.RunSharesTier2(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := env.m.RunSharesTier2(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Logf("copies after two runs: %v", listShareCopies(env))
|
||||
for _, l := range removed {
|
||||
t.Logf("LOG %s", l)
|
||||
}
|
||||
}
|
||||
|
||||
// Shares on hdd_2 and hdd_3 (both pick hdd_1 as target). Then hdd_3 dies.
|
||||
func TestRehearsalD1_SameTargetThenDriveDies(t *testing.T) {
|
||||
env := newSharesEnv(t, "hdd_1", "hdd_2", "hdd_3")
|
||||
env.addShare(t, "hdd_2", "b_share", true)
|
||||
env.addShare(t, "hdd_3", "csaladi_fotok", true)
|
||||
var logs []string
|
||||
env.m.logger = newCaptureLogger(&logs)
|
||||
if err := env.m.RunSharesTier2(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Logf("POSITIVE CONTROL run 1: %v", listShareCopies(env))
|
||||
for _, l := range logs {
|
||||
t.Logf(" LOG %s", l)
|
||||
}
|
||||
logs = nil
|
||||
if err := env.sett.SetDisconnected(env.drives["hdd_3"], true, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := env.m.RunSharesTier2(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Logf("FAULT run 2 (hdd_3 away): %v", listShareCopies(env))
|
||||
for _, l := range logs {
|
||||
t.Logf(" LOG %s", l)
|
||||
}
|
||||
}
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
package backup
|
||||
|
||||
import (
|
||||
"log"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type capW struct{ dst *[]string }
|
||||
|
||||
func (c capW) Write(p []byte) (int, error) {
|
||||
s := string(p)
|
||||
if strings.Contains(s, "reconcile") || strings.Contains(s, "tier-2 copied") {
|
||||
*c.dst = append(*c.dst, strings.TrimSpace(s))
|
||||
}
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func newCaptureLogger(dst *[]string) *log.Logger { return log.New(capW{dst}, "", 0) }
|
||||
@@ -0,0 +1,8 @@
|
||||
# go test at controller 7ca59e2 (v0.336.0), scratch clone; the real copyTree mirror seam (rsync --delete stands in by copyTree for the emptied case — see note); 2026-10-20T10:19:17Z
|
||||
=== RUN TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror
|
||||
zz_rehearsal_d2_test.go:22: POSITIVE CONTROL after run 1: mirror photo=true doc=true
|
||||
zz_rehearsal_d2_test.go:33: run 2 err=<nil>; status="ok" warning="Figyelmeztetés: a(z) app alkalmazás egyes adatmappái nem kerültek a másodlagos mentésbe: appdata/app/photos."
|
||||
zz_rehearsal_d2_test.go:34: FAULT after run 2: mirror photo=false doc=true (the second-drive copy is the household's only local copy of these files)
|
||||
--- PASS: TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror (0.02s)
|
||||
PASS
|
||||
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.030s
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
package backup
|
||||
|
||||
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate D2). Never committed.
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror(t *testing.T) {
|
||||
m, src, target, prov := newTier2V2(t, "app")
|
||||
prov.has["app"] = true
|
||||
prov.binds["app"] = []ClassifiedBind{mHDD("appdata/app/photos"), mHDD("appdata/app/docs")}
|
||||
mustWrite(t, filepath.Join(src, "appdata", "app", "photos", "family.jpg"), "PHOTO")
|
||||
mustWrite(t, filepath.Join(src, "appdata", "app", "docs", "contract.pdf"), "PDF")
|
||||
if err := m.RunTier2("app"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
destBase := filepath.Join(target, "backups", "secondary", "app")
|
||||
photo := filepath.Join(destBase, "hdd", "appdata", "app", "photos", "family.jpg")
|
||||
doc := filepath.Join(destBase, "hdd", "appdata", "app", "docs", "contract.pdf")
|
||||
t.Logf("POSITIVE CONTROL after run 1: mirror photo=%v doc=%v", exists(photo), exists(doc))
|
||||
|
||||
// The household (or an app bug, or fsck → lost+found) loses the live photos folder; the docs folder is emptied.
|
||||
if err := os.RemoveAll(filepath.Join(src, "appdata", "app", "photos")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := os.Remove(filepath.Join(src, "appdata", "app", "docs", "contract.pdf")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
err := m.RunTier2("app")
|
||||
cd := m.settings.GetCrossDriveConfig("app")
|
||||
t.Logf("run 2 err=%v; status=%q warning=%q", err, cd.LastStatus, cd.LastWarning)
|
||||
t.Logf("FAULT after run 2: mirror photo=%v doc=%v (the second-drive copy is the household's only local copy of these files)", exists(photo), exists(doc))
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
# go test at controller 7ca59e2 (v0.336.0), scratch clone; a docker STUB on PATH (no real docker reached); 2026-10-20T11:07:38Z
|
||||
=== RUN TestRehearsalD5_InvalidDumpReplacesTheGoodOne
|
||||
zz_rehearsal_d5_test.go:26: POSITIVE CONTROL: last night's dump, 152 B, ValidateDump={Valid:true TableCount:1 Error: FileSize:152 ModTime:2026-10-20 13:07:39.022566077 +0200 CEST UserTableFound:false UserRows:0 LooksEmpty:false}
|
||||
2026/10/20 13:07:39 [WARN] [backup] ValidateDump FAIL: /tmp/TestRehearsalD5_InvalidDumpReplacesTheGoodOne2769942191/002/unit/app-postgres.sql — dump file too small (< 100 bytes)
|
||||
zz_rehearsal_d5_test.go:31: tonight's dump: err=<nil> validation={Valid:false TableCount:0 Error:dump file too small (< 100 bytes) FileSize:99 ModTime:2026-10-20 13:07:39.026566071 +0200 CEST UserTableFound:false UserRows:0 LooksEmpty:false}
|
||||
zz_rehearsal_d5_test.go:32: FAULT: the file on disk now = 99 B, still holds CREATE TABLE photos: false
|
||||
--- PASS: TestRehearsalD5_InvalidDumpReplacesTheGoodOne (0.00s)
|
||||
PASS
|
||||
ok gitea.dooplex.hu/admin/felhom-controller/internal/appbackup 0.008s
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
package appbackup
|
||||
|
||||
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate D5 / R-1185). Never committed. A docker STUB on PATH
|
||||
// (no real docker is reached) prints a header-only dump, as a database that was re-initialised empty would.
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"log"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestRehearsalD5_InvalidDumpReplacesTheGoodOne(t *testing.T) {
|
||||
bin := t.TempDir()
|
||||
stub := "#!/bin/sh\ncase \"$1\" in\n inspect) echo true ;;\n exec) printf -- '-- PostgreSQL database dump\\nSET statement_timeout = 0;\\n\\n--\\n-- PostgreSQL database dump complete\\n--\\n' ;;\nesac\n"
|
||||
if err := os.WriteFile(filepath.Join(bin, "docker"), []byte(stub), 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("PATH", bin)
|
||||
dumpDir := filepath.Join(t.TempDir(), "unit")
|
||||
final := filepath.Join(dumpDir, "app-postgres.sql")
|
||||
good := "-- PostgreSQL database dump\nCREATE TABLE public.photos (id int);\nCOPY public.photos (id) FROM stdin;\n1\n2\n\\.\n\n--\n-- PostgreSQL database dump complete\n--\n"
|
||||
os.MkdirAll(dumpDir, 0o755)
|
||||
os.WriteFile(final, []byte(good), 0o644)
|
||||
t.Logf("POSITIVE CONTROL: last night's dump, %d B, ValidateDump=%+v", len(good), ValidateDump(final, DBTypePostgres))
|
||||
|
||||
db := DiscoveredDB{ContainerName: "app-db", ContainerID: "0123456789abcdef", DBType: DBTypePostgres, DBUser: "u", DBName: "d", StackName: "app"}
|
||||
res := DumpOneTo(context.Background(), db, final, log.New(io.Discard, "", 0), false)
|
||||
b, _ := os.ReadFile(final)
|
||||
t.Logf("tonight's dump: err=%v validation=%+v", res.Error, res.Validation)
|
||||
t.Logf("FAULT: the file on disk now = %d B, still holds CREATE TABLE photos: %v", len(b), contains(string(b), "CREATE TABLE public.photos"))
|
||||
}
|
||||
|
||||
func contains(s, sub string) bool { return len(s) >= len(sub) && (func() bool { for i := 0; i+len(sub) <= len(s); i++ { if s[i:i+len(sub)] == sub { return true } }; return false })() }
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
# go test, hub store at felhom.eu 5c48731 (scratch clone), in-memory SQLite; 2026-10-20T10:12:12Z
|
||||
=== RUN TestRehearsalK1_NewBoxAfterHostDeleteSeesNoEscrow
|
||||
zz_rehearsal_k1_test.go:20: POSITIVE CONTROL (old host live): status=&{IdentityBlobPresent:true ResticPwSHA256:SHA_OLD CreatedAt:2026-10-18T00:00:00Z SupersededPresent:false SupersededAt:} err=<nil>
|
||||
zz_rehearsal_k1_test.go:27: retained rows under the DELETED host_id: 1 (identity blob present: true)
|
||||
zz_rehearsal_k1_test.go:34: FAULT (new box, what its ACK carries): status=<nil> err=<nil>
|
||||
zz_rehearsal_k1_test.go:36: => GetEscrowStatusForCustomer returns nil: identity_blob_present=false AND superseded_present=false for the new box
|
||||
zz_rehearsal_k1_test.go:39: retained rows reachable under the NEW host_id: 0
|
||||
--- PASS: TestRehearsalK1_NewBoxAfterHostDeleteSeesNoEscrow (0.04s)
|
||||
PASS
|
||||
ok gitea.dooplex.hu/admin/felhom-hub/internal/store 0.048s
|
||||
+40
@@ -0,0 +1,40 @@
|
||||
package store
|
||||
|
||||
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate K1). Never committed.
|
||||
import "testing"
|
||||
|
||||
func TestRehearsalK1_NewBoxAfterHostDeleteSeesNoEscrow(t *testing.T) {
|
||||
s := newTestStore(t)
|
||||
const cust = "Tester4"
|
||||
// The household's first box, with an escrowed package (K blob + the identity blob that opens the off-site repo).
|
||||
if err := s.UpsertHost(&Host{HostID: "Tester4-old", CustomerID: cust, APIKey: "k1"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if _, _, err := s.SaveHostEscrow("Tester4-old", []byte("k-blob"), "fp", "zk", "2026-10-18T00:00:00Z", "SHA_OLD"); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := s.SaveHostDRBundle("Tester4-old", []byte("identity-opening-the-old-repo")); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st, err := s.GetEscrowStatusForCustomer(cust)
|
||||
t.Logf("POSITIVE CONTROL (old host live): status=%+v err=%v", st, err)
|
||||
|
||||
// The operator deletes the old host (the installer's 409 tells them to), escrow acknowledged → demoted.
|
||||
if err := s.DeleteHost("Tester4-old", true); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ret, _ := s.ListSupersededEscrow("Tester4-old")
|
||||
t.Logf("retained rows under the DELETED host_id: %d (identity blob present: %v)", len(ret), len(ret) == 1 && len(ret[0].IdentityBlob) > 0)
|
||||
|
||||
// The reinstalled box enrolls under a NEW host_id for the same customer.
|
||||
if err := s.UpsertHost(&Host{HostID: "Tester4-new", CustomerID: cust, APIKey: "k2"}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
st, err = s.GetEscrowStatusForCustomer(cust)
|
||||
t.Logf("FAULT (new box, what its ACK carries): status=%+v err=%v", st, err)
|
||||
if st == nil {
|
||||
t.Logf("=> GetEscrowStatusForCustomer returns nil: identity_blob_present=false AND superseded_present=false for the new box")
|
||||
}
|
||||
ret2, _ := s.ListSupersededEscrow("Tester4-new")
|
||||
t.Logf("retained rows reachable under the NEW host_id: %d", len(ret2))
|
||||
}
|
||||
+306
@@ -0,0 +1,306 @@
|
||||
Tue Oct 20 10:09:14 UTC 2026
|
||||
filebrowser Up About an hour (healthy)
|
||||
felhom-controller Up About an hour (healthy)
|
||||
paperless-webserver Up 3 hours (healthy)
|
||||
paperless-postgres Up 3 hours (healthy)
|
||||
paperless-redis Up 3 hours (healthy)
|
||||
immich-server Up 3 hours (healthy)
|
||||
immich-postgres Up 3 hours (healthy)
|
||||
immich-machine-learning Up 3 hours (healthy)
|
||||
immich-redis Up 3 hours (healthy)
|
||||
traefik Up 4 days
|
||||
0a4fb1a71c72cf704d1afd44e1ab24a8a68e11b8d82ef1dcb71a75d61a9dc1a3
|
||||
0ce36c2e6db54e085f71a1436c4c89a9807a930c373198f451fa1665f90f0cab
|
||||
0e2ca5e8738b95134285a666b1a7622bfcabc4728fb9269cc972dfaccbd59d45
|
||||
0f4002ab75695fb312775a8203cc5213b256f1b01d255210cb5cdd0e50cd9070
|
||||
1a8fe48409be3defe01713e2f621091630b965e1805cbb5872d1738043efd35c
|
||||
1afe3420067df65e0d3e53baaec3debc39d2d42d333c7a3c266e4f6ff98e04f7
|
||||
1b41940af5c4fa82af59320d6c81ba622d97f2e43d8ec51c50dcf65c2fdcb1e8
|
||||
1f5120663962c7b03539bb993ab736be4800096e3d1f7147eb922d44b308ae6a
|
||||
01feef94eb7e38a3679921b66cfe206e08adfd9b360ed4a60414cde606de1643
|
||||
2a71e17dfa1e930ffe0c213cd1615c8d1adcbbf8d61615a38064d331c265c02e
|
||||
2aaab84bf843817d9e6f32b673277fd1e0c2c8c2d038b8126df44927b1b41006
|
||||
2ac49fc9cc2839f55c8abae76039950d15a5c301ea5b3918fa3e5d8715a15f42
|
||||
2bbdb463b29493d60d352bafefccbe57856cf80bd2cc193b61f9bcc1310fb45e
|
||||
2bc95b104191ffd31bdfdbe1e038d284d0624c04735094494c2c77d957e8e905
|
||||
2dcb4bd2c3c618d71f582d33d5b1243e6ab075d3b00abc238d4430668a2b6f55
|
||||
2e4c79db25c47436b44245376b1f6ff1991208ba081fddce1938c3897fc63c07
|
||||
2ef8994a563ae88024ba29db4e349d26e6e3889e0683d30d1fb258208c9a10b4
|
||||
2f08e882c5d23373e2df5486dce28d4728330f9f0131a2735ebc938bfb4d7b3a
|
||||
02f42b55ee403ceef63a472c3624da0b48fab9e6aa6dc0c7800ec68b2f2041ef
|
||||
3a3d7bc33a1ba21ae4a39c8cb345254cf66cd1dc60afae64e2299513b094b93b
|
||||
3ac9820f84d0a50f233c63a62ee1b63b92acac026d68c60a8227f370b0be00b6
|
||||
3d9a93769342e83a3c512d517b99cd64f8e794eb305c009e25c73a6f6627ad29
|
||||
3d9c1f23abbcef771a4c38a75d75865a12d76a4359fe83c1828652dba3f91710
|
||||
3f015f020f49d4967d2ac71b94060e6eaec86bc8852cf9a0b5f3dd8133dd528c
|
||||
3fa030697dbaffb6809991bb69a335dc4e9accfafd4ed5ee953ed676673912e6
|
||||
4a9ed2aed1fe75a0b02f688e29a20acc2bf7e081ba7518a7d75451be30fdc4e1
|
||||
4bbaae5ae8cc8cc3330a73bb872b16e1420805197fa9bad903f7db9a1f358041
|
||||
4c54a4cda3c5280d5b13facf3089f8a0710c4570876aaeb679025aabf92df99f
|
||||
4e052b4ef5602a68eddcea049bbc4e79b306012cf8b7f0c9cfc3e6c39af64fce
|
||||
4ea780981a54c3e87aa72eb80436e02f3c6b3aaeab96ecf869602f5f5b0b546e
|
||||
4f2ebb779645de5cf9d24e93c6fbd9d95d358b390dec94eed6f0223016785e42
|
||||
4f2266bb0b8a9b902eaa439ae9225963d6fe4359fe30569c2d8afa926b582585
|
||||
004ae946f5997471f4554ab7158a3b982b72391b59c59be40ea79b501eb1dab9
|
||||
5b84763ca63a28ae79c1393d700c534e7dd6e610f890f540981e49972e3854e9
|
||||
5c7dd57f445d87a39eea1e7b299e1a0acbbc946d6bfc92ba165c4af8933b322a
|
||||
5d47a8950cfd56492e0f8b4ed6bfbf470a2e22811206ac0da0b79e520890b41e
|
||||
5def44013bf255d4cd97c3829ac7ae895c607333a3ee586a4090bda128d379fd
|
||||
5ec44fe3223976727132521b3055187e88783716a1f5fd07d1625205acff9888
|
||||
6a84846fd8e476d206eafb8f5a529ecbdf6b1021daab890429077e32ef4c2b6b
|
||||
6aef4d0808bef53a15fc6d19614b641a5e36c941f2729572cfef8cfc8e36bbfd
|
||||
6b5a55f08cc63514e70a93a3eb5d294e9b109e990521bb0ddcea4818a8df1456
|
||||
6b8acdd640ae067c2ee5235ddea8ad4c0e1eec2aaef1277b426923f56303e07d
|
||||
6bcaa587af8bf282f83605713213c20275b1c8178c28de97b154af25bb6a674f
|
||||
6bdf1f89b902e5a7134a87e0269229531ec6c8391256ea1bf466d2d6b66dd8c9
|
||||
6bf7012e27730a047bff7d69ceadef9c06433783d0b9582faefa9c9d0f966c86
|
||||
6ce965b12082d21ed8ade00884de6477faea9f97cf7a85e4b5ab3302d3517fce
|
||||
6ddb2a450b19b940661c677b20936e92e8de54b8bc76f8dbacc56398d7d13a38
|
||||
6e21c6398b13ebac045a11b136fbb2d289e6abe99401eed9108b5464754a4c87
|
||||
6e0521e4314af444375b56e348a5e3d34aceb88d2d5e2f48b6986c27250e395e
|
||||
6f19aa563c4e3e2000ed997157bc7f5bc8a45ee1d77c2cdcd1fd239117a62705
|
||||
7b5530f7f440ff6477f2ee5885b37ff9cfbf933d7908225998875b233b4d2a2a
|
||||
7bfe063b0155297f5bca22c7ed02ae7ef9ce1b968670607415f7720e1e8649f0
|
||||
7ef24d6bf9fb190db2572505b489e45731a326c5c9c4ae6e36ffd7874560bbab
|
||||
7fc41ddc47424ea11d4940f7062cdae69b92af645e6759f8759f47b689a19cb7
|
||||
8b44a3824395c930d0db784e5650ba7b58663090c3bd72a44f9dc716008a9642
|
||||
8bdcbc26d645ae3bc60742debe32650510bcfb7c69ad4416688af5b1815e5645
|
||||
8c2a7cf3f809218d20576368d4d419a63a07a89c3b12cc389fb2f3424fb45b73
|
||||
8c4d45c90f336c0ffd5d619765d8d2a45e802d619548f187bb3f2672b3534969
|
||||
8cae3c834dd9284d1d0069de042c2d50e49a138a78d93a59eb752272e4a5e4f4
|
||||
8d2ab3e77a72ebd708630e22f2eefc0a66f73cd94ed2abbfc4f666c504a3f172
|
||||
008dfd2a2e98c6b6673ee471d72b9e150cfe02859eb3a3bbd31a3c582b5a3a46
|
||||
9a2e6958ba1ea4443ade7619d326c84c50dbc444810b47904ff625969725a091
|
||||
9a86729a66d00d8be95f93062381b9bd91b16f94b654636681553fec6c7c7985
|
||||
9abdd18e6a7b07800067237c93d48448d49d9ad2388a91d35653323c2bed4389
|
||||
9c59fdcefb7ed258e615f8cb688334a69aebdb8e38fd1075a60d6fd98167065a
|
||||
9c033896f0b00023816a49cea882a342c9b62ca4528640969a0d22c21f507505
|
||||
9db05859a329f9544f5a48cbd987f02fd00e48f160dcf6ee8331d8accfb423b4
|
||||
9e9fa98cb2344c5eda9bae7417557f2bc2503de1eed8219fdc2d2d378e1883fe
|
||||
9e06437718bd8f9a27240eff7b425406aae50f061ea47a6991b9830ee2af9f59
|
||||
9ee6f261789b97ea0887a6a905bf5bba61a7070cc91ff64cda0091923d9cdf25
|
||||
09ea67268ce8c0442357ba316f6431cd96143da6bb54008d3c342be8b6ddb169
|
||||
13b744ec13f00bd3e5e2ecaa37bf5385af38cf0ff94fd05e660cfed3339a0c05
|
||||
13d76d014354ffb4e5aabfe00eeed5206b98c502dcfd515545fdc7210316bf61
|
||||
18e51d43a77ac282e108fd8a8e55207282cdabd2219af1e47c2c214117179f51
|
||||
20ee459190ea3442ab7cf0c288182033fb84b75a15bdfc14540e80ab60dec542
|
||||
22ab14c474cdb6cf9f873dcb6c8ce85dfc588abb46e2bd0925cdeff384437c46
|
||||
24a30fa240cec28f9a8b6d02a47ad1fda39674f094ae095b0d6ec3cbf099ed44
|
||||
24da0da42dc2eb9b6d35dcb7cd513b1e5c7d2ee38b6d59d9a1d51f0d01f6f920
|
||||
27a56be2953f040882ecd07fd8b91937f8cb17c25d1ceed56656ed2075518f86
|
||||
31f3b426743b1cff7952948995915586480486ed7fd06eef3379cf54b8fb3d9a
|
||||
38ba44b0ff38f8df925512a477b8b5b4cba216a8ec0bd51b7463a86a88c42fea
|
||||
43d6928bc4bb7a1e097122424e35f22393a0ccf2e03863d7dc3cce44163c8ae8
|
||||
49d0cebfc3199cef395ae27a6ffa479340c1c2f69da9248a777d7bff9b6b9ca9
|
||||
49dbe15539d7170822958bb0699d140c4d493215236b32679340588710d5452b
|
||||
054cfa9b87316799ecd24ecfb480a38aa04b5f85a478a3db95c41706e5b63523
|
||||
57e9dc8d409d939d877a513023c95146400dd10078919ab9402d8144d5f30ca0
|
||||
62e86d5d6aded4a8ba536696c0df0f0af89f2cd7b4b893a7383e3abc00eeadc5
|
||||
63ddb5c60e6cf39dea412903cf1d6c0164a1146bcf4ab98ee94c9b1f9e45c6bb
|
||||
63e2655474549ef03aaf7d9d741502287d2f01170c5ca008d7b09b8e00b1be66
|
||||
67eff4c5c60d493dc2a9f7841f7f138b610dc389c32793a27a62ddc13d417a1c
|
||||
68d18a8d3519b6ea6928bbca40f9db7500b700f66e6eab236e7c32a214de2cc7
|
||||
68f83c9613471fe0e046a1f9252b71e86aa50d36315c675e92dc44a7cdd0267e
|
||||
71cb6156444948894aec343c9a2c0c7674aca84d292d090c7f786d13146709ed
|
||||
72ed18f5962e48dd5fb54e7d91d2fd0479017c27b2c60fc58a5d6b6e55242757
|
||||
74c6b39b7b926dd56db16f175026c2bf9e80a912916f273c122227672f181d03
|
||||
075b39bc7ae8c56a6d99923e4938ddbbb4174df6f0e653abf9c99b420b401a31
|
||||
78c01e39d1997eb54344f02f14dc02c7230cfb14ba58abc1a11feab2e9af85be
|
||||
78e7acb9228c1df4ca4c706d459b97c8e79e8399fe06476ce72bd919a3b2b77f
|
||||
83a8287d1e8700aa7e207b2c617a68557ce6f83f373f2c3c6d1443ae50032826
|
||||
83bc388f3ae110e72ba51a8f0b68f7bbdec3685907c1ff08095b63a1759502af
|
||||
85d6f7309436e59cce9e373b0aa1626fa5e12d8d81163773f5f2c45da9d56543
|
||||
88c6573974e81f1fff96999bf54b80c66bc6f1e9af6746b3cd6801616198e0fd
|
||||
92e2e73e3aef6e5330d2ce8aed3c58e7e3632a6f191f2b03c997ccabd245bb65
|
||||
93c2b9d0a1f54c2e6355d206d93bf2309f81076628aae9d868406c288b0f6bf4
|
||||
98a80f5f7903ef45f92f74a91f34dc01d527f8a2fe98b4fbe76711f5259416a5
|
||||
99f7aaac2646c66dd0eeeebc157b8563b56299c27ab4bf64e13920d089ca7156
|
||||
104e16e4796c52fe212c04b06417161e0bab20fc01632bfcf71421b73b979fbc
|
||||
122fe66bacc9f04b3f3e4f6d0a3b15abdef8321eed36637a1284e3bd145463b8
|
||||
158c590779b033e48b5efcf895fd0bad4c6f277303d72b963cd0802ecac1f440
|
||||
218a186ab13a38d514cf9f523e5798452ae4e5ec76ecf26fe1a46ae22b47a508
|
||||
283e9d53742a13c471e5f59f53a57c179d461f0017ad5f8db27881cdd994cd04
|
||||
305b75817cf5c06105b19dc8c56ff0d796dc6e267ae5bcfc6018aa3d12037744
|
||||
325ec2aa2a6c307bb82d4dcb600fa3ff127aa6d1f35dc683ab61124ef5f0f5b3
|
||||
338f7535ae9ef91754e76604238ca7652273ffce9e6f0a1300de0b1edb61da77
|
||||
343d41a3b38b5a7eac10750eb0213b81780edd3075e9a1fdf3a2c479f4d2474c
|
||||
346eaddf98e91d6d6e7b8c526074c27847e4a18be01b41027bec93abc2b7b2fb
|
||||
355ba69bab94a95e00e87c6c4808587881b9675699083e087917ec8492bebc65
|
||||
426ccc08685def1c9025c94d343de0e59713073984634ee226e69634d6a18fb8
|
||||
552a4b340eb6be03c188cb17628aee0f916b544142465f8e52e7e9d2325206ab
|
||||
556c805068afd3f6784bda2eb59642d7dc03756580e959e1c75262711a30d4d9
|
||||
0642eb9eee222e235de9310b9bd86b42493e5f4d71d430619b5832484dd09cd6
|
||||
645d70ab53013564fd243eae61db209e3793b826a9eeffd77395b18e27458789
|
||||
670e6447721f667df5222cb06a68c2682eb92803184a44faead8aea0dff44df6
|
||||
0717b5f7615c345fe47ab8212c06f506e08585551d93c6a8083b56a216adb71b
|
||||
734e6b63f955d0b508dea39a8637755e75fbb353d54430956a0010f607eb727d
|
||||
748a1560ef6216c8bf80545dbeadd29f99fd5c984b6f34bb678febf8b369adcc
|
||||
748cce6aa136bdc0b3111c73a7c9434508200f8c231933b925d42b37b9cca17c
|
||||
760cd2f9203dc3dc3f933e4a34bbaae857dee0afed66e727846a4c7f750ab5a8
|
||||
762e15e05b2f84f035a7011e10c64558dc6637c9fdde1846dc77f4056f3962d2
|
||||
787dc06078f3718da6faaba82269a7d21b213482c9e45025315bdb6cf59010b5
|
||||
866fc46ac37407bbf74761cecd21fa6322c4e7621f3dfbe963a85671d5753e5c
|
||||
921d92a1c692ea0fd42cbe500bbdb1f3defcd5fb9b51df216451aa260d95ad45
|
||||
932bae2149e79ef113eb4929614a895c642960160d5f5cb2f2e9afac7dfa6af2
|
||||
968a5d7a54fb1d8ef4163f287e128cbd541ab46a2cc54db721dd3eafe1397b65
|
||||
991a5d344112325fb2c40700cd0ee96a784ce2150cba0a7b0be3ed6eb2411878
|
||||
2469a8c88a452ed226dd6d1082f0cbfbe3107d76f919b930b6bc7ce3b839ed90
|
||||
2634a60c3c37d5eba4be4827c08c0e99a951b934604c29d116fcddc27808e480
|
||||
2788da1d8f917a9d5b8acab2c7fe7336b6a7810f1c7edab63ed327a6022ceddf
|
||||
3000c7863b2e80a72329ca1fa0dd03d2fd3994a2391e82450a8c92199cbb56b6
|
||||
3461d48cd9e88caf5363b0e4d96e799f6759ab8b8bc6ba4c148492a1637c01f0
|
||||
4681b8c67ad20fdd3ad04db895084d37a9296ac76b74ea7bc9465bb3ac93df8f
|
||||
4690cfe7c61625190732c028ad4b918930a65e295005171d48114aa8f27b567b
|
||||
5206b60d49110c01e05dc24034e12b767efa739e2ad21c252ab83db47e56d526
|
||||
5234fcace55ee7e44631499ed1f95fd0289918a84bd9f5cf6d9f761d43c34f17
|
||||
07402fbe22b572302f3317958f8cc4c6ca30317e0ee5bbbe9ff51c60b3da5a05
|
||||
7458b4c4b975debbd2bb705e9575d2d1ff5e9308ed9e70c302f5643136cd0cc0
|
||||
8097f97f4ae00b654b1c0b4408ad585f5bfce7999e482f244bd698268e1f6061
|
||||
8158fe1f757d26cadb484ed9690ad4d20d4cb382eeec7c9b0627b365afd6576f
|
||||
9917d608ac0ad05856d31873432d641828ef519825a3f451e6a1ab2519dbf0d1
|
||||
13175cc97740c834e2404c11ba9fa9418d0c390d27a87e6e869977d6a86c20c0
|
||||
36571aa758bc335acc463c3ea074422a5123c309a37e1a7fc818a2069941c493
|
||||
59438a5f6378d218a73d3106ea1984d1c160fe4fcbbb7ec1177067f17430dcb3
|
||||
60496cd3a1e1ad1dcaed30427b70a57b8b14a122bae7f2cd882f8154a4549978
|
||||
61544bee8e2c1cd65e999121bb63011a9c1b4139b20b04fa32e3a850ca8c82b7
|
||||
84046f48a922cd337284bc2448a5d2682ea6a2e3479e71531122aff9ee4cbc93
|
||||
86022b0cd1043facf7b1c933c3b6a6ff36277a76d50bdb2da6a5690d57381b9e
|
||||
98129d711a1e807ab7c3ae4c97187ab8505cebf1012218e8530d0829a2ad436f
|
||||
177451abc7ed0606468250a7793e018725e8c7f370ee8937d4941a11571dcb9d
|
||||
0217869eb1be5181b7053b993443dac73d9cd0804cd5bedb604edeb43f911702
|
||||
219118eb01fa7be03fcce2124b74708cacafe66aea617fd2bafa32f0b2d4fd80
|
||||
394985d0c3db0791b8c7569fccabd36959d00deb285075f9852568c4c81d32eb
|
||||
722380f945884d4dea921ffb958513e02a4802483ec0886712e5ed3fa28d751c
|
||||
774878c58b6a8cae6fbecdf656f2c1f09f37a89190068663e923763dbd6969ce
|
||||
925226a74657c7ed0fcdbcfa9b504a47f7e21d0982c44f7ed9576140fb417146
|
||||
963079a9efb903a575b363b27e15927312569528b9066511b4c282e01e1ec931
|
||||
3830913f0478ec59a7a31d538f858f8ef612e1b2ce3bb9058a2d18f7b43abe9c
|
||||
8579968bcd362a1aa9dcf14e22956b2fd15fd74a2fc2e7369ffb1457c2ce6693
|
||||
8678479fa55ea3871598eb6914eb8428ba599da4c619c62f231e6931f0cb1f54
|
||||
13694424a3895e8e73cc1e06af29aae0c3179c3cb5587b9daadeadcdb29b7cc7
|
||||
67711784cb23efbc0762e83f03365b837ec5216c38f3ba682a82eaff6a7863db
|
||||
92236992ea73ecd9befe3330e88e96a7aedacd9cbca0bb8c32c31d0e2c914e5b
|
||||
134995217d29dc47fb0ca1a274ceb5ea74225d36a98c4bbe6e07bf8c8d6898c7
|
||||
481706314b835ab042583c3107e168932844a162a2adf3b9ef2fa27433a7fd1d
|
||||
4098005900d38c71b152787ce4bdedfb8e2b2b8b061f87645b2b4a14e0004335
|
||||
4699188665aa6eb965b7bb9f4938187f377e7a4ad55fc97e44802ea76f3612e9
|
||||
04762469942ae24c353f929fced777f245bb9f2f888a9f2044ff27fc45933665
|
||||
a6c297b9282b39caca7e513724219af096a3cf6fd453c3736ee7c4c833ef417d
|
||||
a22cb584ba3444bc1dbd90b3723b1ad9424352251f2218a92b1884ef74e63b9f
|
||||
a68fad1bcb1f1030af3811590cc18d74eb1da3c67190af4459fa69118aa8f035
|
||||
a74b9b2333ee936c17edd0525b49fec1337768ccab68f5576a39486c875ff968
|
||||
a74b41238ed9ded9e866679ab00f85813d800f92e4bb2642d3f9de28a1655b81
|
||||
a84ff849188b2bab211fefe8bc61a70f58f48f6a2a6ca25c2645e0bb5c384cbf
|
||||
a2815ec8af59e0a137aa0cdf2baa48ca6d1b86e291c5728fab9449280d15fa75
|
||||
aa5cf2d08c68b8d3dfe3330861935fd97ff8fb9169645f6c632d970ad71b575c
|
||||
aadb8f0be78890f1e08630b7a762a82853dc1402326bd16cd2a2844fa63fef1d
|
||||
ac48d92438e8d19cf546764cc051172af82092223daad8ccfbfb4cb05100f5d6
|
||||
ac78e0a5ac10c7d0239e0e1ba3b604cd6fd418d87a6f32b66f3ce98c674164d8
|
||||
ad79ebffc7d8225cfbee6ad5fec4092ba7376f38dca912d6df8ea932f1eba20b
|
||||
ad7084f002b17a1a27f70c796e3d7424a064b2ef5b71eadb7b3b08dcc9f59042
|
||||
ae2a94eac2960e95f944c9a4a0649c94c405fb0e518adfba2e6fa0f80f2a5af2
|
||||
ae9d5c15cba1adf881044a4449eaf72f5f2ed753c301edc1e43bcfa4560e8d31
|
||||
ae5499adb73000b6f460e63d6e045fe492702bd73187f868d9988690cb72a453
|
||||
b0e341cb8fb7f2fb42f7a8512a9b922591cfa40bd3522f31a422a9fdb4586008
|
||||
b4bc9ff2d9b533e896dfc5082ba865a4ea7676d71259d73aa33d515a5a60dd91
|
||||
b8d2fbe7e6afc931581bf5f1edacaefb9071bcc9f709e294a3f5ead104f73890
|
||||
b8dd8697ad52f98dc44ed3c940752aa52e16d6ab7563d36b1c9e221db22d1e11
|
||||
b21ec6b73a9f1f51623355f2bf8229b5f6f9da1484e1ce105cf8e0363475a5b8
|
||||
b081c3329ed821e22b0ee9cecbc2f52a5e9be70be2b1cb83bb262c21c84640d1
|
||||
b274ece5f6e1d506ca0646077964797c3e7b4413c9a9f1e453d9257e6b4d09f9
|
||||
b676fcf4f5272015b942c8589135a18fa16204c5aec457f1d0898ce6c84638eb
|
||||
b1591e2a2604b6404d0d18e4b2b7599aafbcfb2bbacba06b5de5ff14abf14be3
|
||||
b7000ea518cd327f7dfb42858434fdb80c451aecf0a0325b6ccbff1f30545803
|
||||
beb17a9d330ce361923600aac8e13f8b10e62d0b4067380ad7a95c4d77f0110e
|
||||
bfb91f5666855fd57228801ed0a4294981bd2a6665a93d1093bf3c4a0e89cb2c
|
||||
c0fb7358aabef2c38e6921e2ed06b7fdddddecd29d530814545968323eba964e
|
||||
c1d4fc53203c4b65abb1ff76f9505a46175c57afcd9c95cf445c6a3c443839fb
|
||||
c3fbcb924a928bca859b69c75e4df275eb5e0abf03029785247313b0fa1cdd27
|
||||
c4bccf1c99f4990b0a23848b6fa58977dea0bdbf6ab69b7c0c2b90f7f0a2f5c3
|
||||
c5e875c34f4e4f1fd4edb1c6dbf4862a6543e26387ee480eeba0f5e3f44af7b6
|
||||
c8bb03639a4d7c41c233fa28d5a1c06dee874fe00614f764df9db3df14da86b5
|
||||
c16aa92f949bb89a034c6ef22ea8ece859883ab309086a204e43cedced8dd059
|
||||
c31f2070f92b36708a62c1d53a7aea495fd997440f2c01c851ad83c66357e379
|
||||
c35e0b961694a882dcc4c7bdb8e657d4b0685687cb5c4469559ce5078c95302a
|
||||
c56e78b468dfe3991fc787ab7fe30b4282a73d7fd2e79e4dfc9305bb064685ea
|
||||
c67a888239bc5e11ed44ca31307f3b46f7f76698d706ffc764024110d1a69ceb
|
||||
c424d546895a59e0f88e2296a665c03f4618ac304ef19742b6f1c543665b9bb3
|
||||
c716ebc8ba8fa1556e1dddfaae98c7019618d39ece0e78dbd1b9f58dd0760d94
|
||||
c797d99e68b0efce9b1337dc0e3e9cbf42806bef03021ade362cb4e4d1de4222
|
||||
c5708e0a13ababf73203b1e297b3b73229cd6370e3fec1653b7cf1dc143a0d0a
|
||||
c52417cfa3d923ce7168b7c2b082eb3ab31bc1750ff3f3edfc2c1cd59fb74f3d
|
||||
c53655b36c7abc28db3020511a0ebdffe8fa37d86412753fca86f4caa7ae7a6b
|
||||
c782734d8be184e0721d0a984b6ac3e8e0052206a9e46d22188a7fb2f923face
|
||||
cb71ab036375acb25c69afd915261acf65f919008fbd59417989f6ebef110c7a
|
||||
cb613ceec1bf86b718059ef3e864dce7ed60e5b6a7bd5ab9f7c633350df48f3a
|
||||
cb10732ef556d77ec0a0a5d7b29c0789b47a9757b67bdb3fa9d5be22809debd0
|
||||
cf799a4b43f87d63eba6042a73b9eba3f17b354be1fd2d94dfac2048e13f3fce
|
||||
d3c941cd28b5a692c2532f1381afdc57740d603fbba5eddbe3539f561f26a0ca
|
||||
d5bd7d95fbb1f56d7617d96a5a6deb65417b4448692cb45f4a0bedd824165fc1
|
||||
d6a74fc39944848c79012bb06d915c609b92c1099881b90b4cda59074bd1124f
|
||||
d58b42f643175b6272bb579d354a9510bc34fc88e4e70dda08346997ba3cec13
|
||||
d256b9490d623a136093e459b9f466dedadaa0d00235d6934044e6591ee7e2c5
|
||||
d352bb11678cf6cb47a16740ee9b79cfbf4c19d4da23f4fb95621bbdea6b91b2
|
||||
d053441be51f569ac393c26c220b6369ae3455ae76f3627eba223e8b4e40eacd
|
||||
daa717bf6363a3afe8dbcd3ab863a81ca737e0999242bd99c618daa0e4fa085b
|
||||
db0c37c3ce9df2f2090c622eb4194e6860e26846398d4b4a092b9c4c21441af4
|
||||
db2434b8bc6fb7858aa5eea51b23eb3fe17ed8e8ac025596ebb4f857d00f9ba9
|
||||
db6910df9d6429a8ecffced74f22c26f45a11d48efea51e48e8097bf43331540
|
||||
dbcfeebe1162c457fdf6f7cbfff66c971dfaca8a1a8ae2fb5472fb9007f83ddc
|
||||
dc5c95c9a8f2f01a8eab63c7d591a7d008dbaf5f4b6157380faeeaeab168cd59
|
||||
de81766fb8e806f93c97771ae1a0ff529831bc5e7f50e2187d41dd606b8e23f1
|
||||
e0e25d3ed1797b0092c897ad904a4ac7a2bd77e12b96e1790a939aa009851803
|
||||
e0ee75c206f1b9be41e608c6de05f8fad8532e3bda240cfad0640c673ae46395
|
||||
e1ae949c5339eca43334dd6720b49df19082036b8f53151b8dfac97f2d2c0b6d
|
||||
e2ca0579185aa6d7324f144357323b4d5521c1e94e8dc7929f0ffa9a73e1e4cd
|
||||
e5f9a9a8f7e4b6d097aef1b0493180a547dbf19006920433e7b80aa8aac0026b
|
||||
e6a40ba40f9ac5f43ba2276b3cb6110bb4c0e20003e866a55ca28c954ddd0e9a
|
||||
e9e1b0601616187e8934c7212e2fde5b590b50308aa9c8c5dd12e419c6bebc18
|
||||
e45c71e0680d2856e2945881fd01ad6320868e0a36427c6bca637c53dfecc625
|
||||
e62af03dd8d599dfbdbecb8c6516d95e582ee803f2ddd3798479fbc769229030
|
||||
e72bfb25201770edb1889f678f5a844a57c9f5308acd57a86c5f4936b800d441
|
||||
e311fa91eca24a97e0c558e91d06dca353aa9348a7ea4b111e694e181868fc27
|
||||
e339a495e0c048e30483e4fd3bef66186712975f3c33c90ef63a3c9686da3160
|
||||
e668b1c1f44dae28ca34f530e313c1e544751b63bf0c480300be8e242683fecb
|
||||
e4638cbb5b12548a76a2cb6e87c7f2ada75d61d3c7d1062535ef0bf2b3cde05d
|
||||
e7945baeafe0ac6d9a1e6761786c35fddad76c81d86e544d369188c186a0389b
|
||||
ea35c704202ee36415d4f7a0873bcb254d30268450d7c9ed2dc611c1e2f4d5d1
|
||||
eaeb108b23d455d01ec0c9e44d39ff395522942794d26f3c149e29615be9bf1f
|
||||
eb420ae07f9c459abc101c1264fa8cf779002ef20748ef97abe7c3764c24f032
|
||||
ec30ca07d0b0af16afe60f217dedf15573045ed9adc45e21c3592ee6716fbe39
|
||||
ed02b9c56e48d34910bc97ebff834f1cb082c23ebcae97b0f51f3a2633664466
|
||||
edcc77bb3d279e3dfe7d687fe30e22f95f5d9391b9908e61b2206a6cf309ea91
|
||||
eea963e8499939d36d09e42d8d5cd86798e81e264c4d4c012191e42926e508b8
|
||||
f1b01ee5efb2117033ddcad5f5eafe8e6f65ad6fc3e933902bfc494c13a7d3c0
|
||||
f01b151ee48f9a3a631457069ece3a2ac3e111df28110f3c06f221ee7c54eff8
|
||||
f2ec30f9f836ed26cb13fdb80064d1dcabf953d1952fea2d85bb6410f4cb2271
|
||||
f8a87cab6bc0890c7a2e36f73e533eac1a1cabb13b886b8ed71afa4c8b7d90be
|
||||
f9b0fe025aedb49e94d80119a8a8dc0f85800bc592e568dc95b0f1e5439c54fc
|
||||
f56a1ab986e83e118ff174e9805f6151e47ff693c17a2f475fcedd83cb98e051
|
||||
f82a956615af08264b64e051e09c1441152d92371d4d2db8d5c0ac4ab47a7182
|
||||
f287b03a768126d72af210dceab12e329573779e7aad56c123ac0dbb14fc4f4e
|
||||
f347e779fb8c09daba2a757dfc3ee7c415f8f5a2eaace19535d363cc3ab3171a
|
||||
f547c232f5e1785aa11200ac03dcfda6f056400bf1456a3388db6d73a8158ca0
|
||||
f6822a239614fc82462535db5b2806dd616696c9552e1d469922f4b7cd2bf56e
|
||||
fa453f6fb451a564d31e704bbca0bcbcc9ad1109a25a8e5ad5c704789a424c4b
|
||||
fb5ad60698efb6ee9f45fd440e56c060ba01ba03a854f556e42f8090b2057c7e
|
||||
fbc407de7b12a4975b60dac518ac287f076659213cce708871162342419ab383
|
||||
fbf633eb758b7b66ab761c34eeebc759791e2f73020e839b3cfc2d692c9ba21f
|
||||
fc1bddcbef8c02474db71bf11dace70e1e51d94f3c3fe6d9d45f4a669ac4a9fd
|
||||
fd453dd47dc82ff7625bd4093c62140fe14a348814f094b43b519f7f59293af7
|
||||
fdc05b3bfda3101bec7d66b5e0cfb589e06f1f4b4d6b2f9d84dac470b0abf53b
|
||||
fdebde6f8bcdd5281c50bcb4669c45dc54e33563bec872f8d23d76d2fe3ebfdf
|
||||
fe503d3e9a34b113d4c56b59c5986e13c44b6e401b6e6681eb62712d24368329
|
||||
felhom-controller-data
|
||||
filebrowser_filebrowser_data
|
||||
immich_immich_ml_cache
|
||||
immich_immich_postgres_data
|
||||
immich_immich_redis_data
|
||||
paperless-ngx_paperless_data
|
||||
paperless-ngx_paperless_postgres_data
|
||||
paperless-ngx_paperless_redis_data
|
||||
privatebin_privatebin_data
|
||||
ls: cannot access '/mnt/felhom-drives/scratch_hdd/felhom/backups/primary/': No such file or directory
|
||||
Filesystem Size Used Avail Use% Mounted on
|
||||
/dev/loop1 69G 11G 54G 17% /var/lib/felhom
|
||||
/dev/nvme0n1 938G 210G 681G 24% /mnt/felhom-drives/scratch_hdd
|
||||
@@ -0,0 +1,104 @@
|
||||
10:10:26Z == 1 deploy
|
||||
10:10:27Z deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||
10:11:12Z state: running
|
||||
10:11:42Z == 2 marker A + backup
|
||||
10:11:46Z CREATE TABLE
|
||||
INSERT 0 1
|
||||
10:11:49Z [2] DB markers: 'A'
|
||||
10:11:49Z POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'}
|
||||
10:12:59Z backup finished: {"db_dump": {"count": 3, "duration": "1m9.055678483s", "last_run": "2026-10-20T10:12:58.690358712Z", "success": true}, "enabled": true, "running": false}
|
||||
10:13:04Z unit on disk:
|
||||
UNIT /mnt/sys_drive/felhom-data/backups/primary/rallly
|
||||
2026-10-20 10:12 487 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/app.yaml
|
||||
2026-10-20 10:12 4071 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/docker-compose.yml
|
||||
2026-10-20 10:12 11894 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/.felhom.yml
|
||||
2026-10-20 10:12 52228608 /mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar
|
||||
2026-10-20 10:11 111371 /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql
|
||||
2026-10-20 10:12 3195 /mnt/sys_drive/felhom-data/backups/primary/rallly/manifest.json
|
||||
2026-10-20 10:12 1191 /mnt/sys_drive/felhom-data/backups/primary/rallly/data-stamps.json
|
||||
10:13:04Z == 3 POSITIVE CONTROL: marker B (live) then a normal unit restore
|
||||
10:13:09Z INSERT 0 1
|
||||
10:13:12Z [3-before] DB markers: 'A,B'
|
||||
10:13:12Z [3] restore points offered: [{"time": "2026-10-20T10:11:50Z", "short_id": "helyi", "tier": 1, "drive_label": "Belső SSD (rendszer)"}]
|
||||
10:13:12Z [3] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started']
|
||||
10:13:12Z [3] restore-status: {"running": true, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:13:12.817369857Z", "last": {"op": "offbox-reconstitute", "stack": "rallly", "ok": true, "message": "A(z) rallly: 0 fájl és 1 adatkötet és az adatbázis visszaállítva (mentés: 2026. 10. 20. 09:30) — az alkalmazás újraindult.", "finished_at": "2026-10-20T07:36:57.852528534Z"}}
|
||||
10:13:40Z [3] restore-status: {"running": false, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:13:12.817369857Z", "last": {"op": "restore", "stack": "rallly", "ok": true, "message": "A(z) rallly: 1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-20T10:13:39.161573115Z"}, "last_recent": true}
|
||||
10:14:00Z [3] after: state=running hold=None
|
||||
10:14:18Z [3-after] DB markers: 'A'
|
||||
10:14:18Z == 4 marker C (the newest live row), then THE FAULT: the unit's DB volume tar truncated to half
|
||||
10:14:22Z INSERT 0 1
|
||||
10:14:25Z [4-live] DB markers: 'A,C'
|
||||
10:14:29Z TRUNCATED /mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar 52228608 -> 26114304
|
||||
10:14:29Z == 5 the unit restore over the damaged copy
|
||||
10:14:29Z [5] restore points offered: [{"time": "2026-10-20T10:11:50Z", "short_id": "helyi", "tier": 1, "drive_label": "Belső SSD (rendszer)"}]
|
||||
10:14:29Z [5] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started']
|
||||
10:14:29Z [5] restore-status: {"running": true, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:14:29.288204925Z", "last": {"op": "restore", "stack": "rallly", "ok": true, "message": "A(z) rallly: 1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-20T10:13:39.161573115Z"}, "last_recent": true}
|
||||
10:14:44Z [5] restore-status: {"running": false, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:14:29.288204925Z", "last": {"op": "restore", "stack": "rallly", "ok": false, "message": "Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1\nstderr: ...twork rallly_rallly-internal Creating \n Network rallly_rallly-internal Created \n Network rallly_rallly-internal Created \n Container rallly-postgres Creating \n Container rallly-postgres Created \n Container rallly Creating \n Container rallly Created \n Container rallly-postgres Starting \n Container rallly-p
|
||||
10:15:04Z [5] after: state=degraded hold=None
|
||||
10:15:38Z [5-after] DB markers: 'Error response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running\nError response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running\nError response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running'
|
||||
10:15:41Z containers:
|
||||
rallly Created
|
||||
rallly-postgres Restarting (1) 5 seconds ago
|
||||
10:15:44Z db container log tail:
|
||||
Data page checksums are enabled.
|
||||
|
||||
initdb: error: directory "/var/lib/postgresql/18/docker" exists but is not empty
|
||||
initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/18/docker" or run initdb with an argument other than "/var/lib/postgresql/18/docker".
|
||||
The files belonging to this database system will be owned by user "postgres".
|
||||
This user must also own the server process.
|
||||
|
||||
The database cluster will be initialized with locale "en_US.utf8".
|
||||
The default database encoding has accordingly been set to "UTF8".
|
||||
The default text search configuration will be set to "english".
|
||||
|
||||
Data page checksums are enabled.
|
||||
|
||||
initdb: error: directory "/var/lib/postgresql/18/docker" exists but is not empty
|
||||
initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/18/docker" or run initdb with an argument other than "/var/lib/postgresql/18/docker".
|
||||
10:15:47Z any undo / safety copy of the live state on disk?
|
||||
/mnt/sizing-bench/backups/secondary/paperless-ngx/recovery-unit/db-dumps/pre-restore-20260927T094411Z-paperless-ngx-postgres.sql
|
||||
/mnt/sys_drive/felhom-data/backups/primary/rallly
|
||||
/mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar
|
||||
/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql
|
||||
10:15:51Z controller log (restore lines):
|
||||
Container rallly Created
|
||||
Container rallly-postgres Starting
|
||||
Container rallly-postgres Started
|
||||
Container rallly-postgres Waiting
|
||||
Container rallly-postgres Error dependency rallly-postgres failed to start
|
||||
dependency failed to start: container rallly-postgres is unhealthy
|
||||
2026/10/20 10:14:42 manager.go:1362: [ERROR] [stacks] Stack rallly start failed after 1.1s: exit code 1
|
||||
stderr: ...twork rallly_rallly-internal Creating
|
||||
Network rallly_rallly-internal Created
|
||||
Network rallly_rallly-internal Created
|
||||
Container rallly-postgres Creating
|
||||
Container rallly-postgres Created
|
||||
Container rallly Creating
|
||||
Container rallly Created
|
||||
Container rallly-postgres Starting
|
||||
Container rallly-postgres Started
|
||||
Container rallly-postgres Waiting
|
||||
Container rallly-postgres Error dependency rallly-postgres failed to start
|
||||
dependency failed to start: container rallly-postgres is unhealthy
|
||||
2026/10/20 10:14:42 handlers.go:1881: [ERROR] [web] Restore failed (async): stack=rallly: starting rallly after restore from unit: starting stack rallly: exit code 1
|
||||
stderr: ...twork rallly_rallly-internal Creating
|
||||
Network rallly_rallly-internal Created
|
||||
Network rallly_rallly-internal Created
|
||||
Container rallly-postgres Creating
|
||||
Container rallly-postgres Created
|
||||
Container rallly Creating
|
||||
Container rallly Created
|
||||
Container rallly-postgres Starting
|
||||
Container rallly-postgres Started
|
||||
Container rallly-postgres Waiting
|
||||
Container rallly-postgres Error dependency rallly-postgres failed to start
|
||||
dependency failed to start: container rallly-postgres is unhealthy
|
||||
2026/10/20 10:14:49 manager.go:920: [DEBUG] [stacks] restart-policy of down member "rallly" = "unless-stopped"
|
||||
2026/10/20 10:14:59 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||
2026/10/20 10:15:09 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||
2026/10/20 10:15:09 main.go:4054: [INFO] [deadapp] rallly crossed the crash_loop threshold (8 in 10m0s) while a backup/quiesce/update holds it — NOT stopped
|
||||
2026/10/20 10:15:19 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||
2026/10/20 10:15:29 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||
2026/10/20 10:15:39 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||
2026/10/20 10:15:49 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||
exit 0
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
2026/10/20 10:14:29 handlers.go:1873: [WARN] [web] Restore requested (async): stack=rallly, snapshot=helyi from 172.18.0.3:36322
|
||||
2026/10/20 10:14:29 restore_dbcreds.go:114: [INFO] [backup] Restore rallly: database password(s) [DB_PASSWORD] come from the backup, [] stay as installed (older than the data) — values not logged (R-1106)
|
||||
2026/10/20 10:14:29 restore_unit.go:393: [INFO] [backup] Restoring rallly from recovery unit /mnt/sys_drive/felhom-data/backups/primary/rallly: images=2, secrets recovered=2/2, data_keys=0
|
||||
2026/10/20 10:14:29 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||
2026/10/20 10:14:40 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 10.8s)
|
||||
2026/10/20 10:14:40 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly
|
||||
2026/10/20 10:14:40 restore.go:246: [ERROR] [backup] Failed to populate volume rallly_rallly_postgres_data: tar: short read — exit status 1
|
||||
2026/10/20 10:14:40 restore_unit.go:448: [ERROR] [backup] volume restore for rallly: failed to restore 1 volume(s): [rallly_rallly_postgres_data]
|
||||
2026/10/20 10:14:40 [INFO] [stacks] SaveAppConfig: saved config for rallly
|
||||
2026/10/20 10:14:40 deploy.go:883: [INFO] [stacks] Redeploying rallly from recovery unit with 4 env vars
|
||||
2026/10/20 10:14:40 pin.go:93: [INFO] [stacks] pin rallly: rallly=lukevella/rallly:4.15.3, rallly-postgres=postgres:18-alpine
|
||||
2026/10/20 10:14:40 [INFO] [stacks] SaveAppConfig: saved config for rallly
|
||||
2026/10/20 10:14:40 restore_unit.go:467: [ERROR] [backup] rallly: NOT replaying the database copy — the volume restore failed, so the database is not known to be the copy's own
|
||||
2026/10/20 10:14:40 manager.go:1351: [INFO] [stacks] Starting stack: rallly
|
||||
2026/10/20 10:14:42 manager.go:1704: [ERROR] [stacks] Command failed: docker compose up -d (in /opt/docker/stacks/rallly) — exit code 1 (took 1.1s)
|
||||
2026/10/20 10:14:42 manager.go:1712: [ERROR] [stacks] stderr (last part): ...twork rallly_rallly-internal Creating
|
||||
2026/10/20 10:14:42 manager.go:1362: [ERROR] [stacks] Stack rallly start failed after 1.1s: exit code 1
|
||||
2026/10/20 10:14:42 handlers.go:1881: [ERROR] [web] Restore failed (async): stack=rallly: starting rallly after restore from unit: starting stack rallly: exit code 1
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
200
|
||||
{
|
||||
"ok": true,
|
||||
"message": "Stack rallly stop completed"
|
||||
}
|
||||
200
|
||||
{
|
||||
"ok": true,
|
||||
"data": {
|
||||
"removed": "rallly",
|
||||
"volumes_removed": [
|
||||
"rallly_rallly_postgres_data"
|
||||
],
|
||||
"hdd_paths_removed": [],
|
||||
"hdd_paths_preserved": [],
|
||||
"hdd_note": "Az alkalmazás nem tárolt saját adatot külső meghajtón, így ott nem volt mit törölni.",
|
||||
"userdata_kept": [],
|
||||
"backup_paths_removed": [
|
||||
"/mnt/sys_drive/felhom-data/backups/primary/rallly (26M)"
|
||||
],
|
||||
"verified": true
|
||||
},
|
||||
"message": "Stack rallly removed"
|
||||
}
|
||||
0
|
||||
0
|
||||
ls: cannot access '/mnt/sys_drive/felhom-data/backups/primary/rallly': No such file or directory
|
||||
/opt/docker/stacks/rallly
|
||||
@@ -0,0 +1,25 @@
|
||||
# C3 — R1/D4 proven live: a failed LOCAL restore leaves a broken app and no copy of what it replaced
|
||||
|
||||
Scratch guest 9202 on demo-hp, controller **0.331.0** (the local-restore code is unchanged since v0.328.0 —
|
||||
`git log` on `restore_unit.go`/`restore.go`; 0.336.0 runs the same lines). Hub off on 9202. The dashboard password was
|
||||
reset with the controller's own escape hatch (`--print-reset-code` → `/claim`), held in `sec` only.
|
||||
Architecture read: `07` §6.3 (the ladder „replay → rollback → hold"), §8; `09` §3 decision 154 (R-638 A), D20 (R-893 A).
|
||||
|
||||
| step (UTC) | act | observed |
|
||||
|---|---|---|
|
||||
| 10:10 | Rallly 4.15.3 + postgres:18 deployed through `POST /api/stacks/rallly/deploy` | running |
|
||||
| 10:11 | marker A (psql in the app's own DB container); „Mentés most" (`POST /api/backup/run`) | unit written 10:12 (`rallly_rallly_postgres_data.tar` 52 MB + `rallly-postgres.sql`) |
|
||||
| 10:13 | **positive control:** marker B, then the household's restore (`POST /backup/restore`, point „helyi") | „1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult"; markers **A** (B rewound, as designed) |
|
||||
| 10:14:22 | marker C — the household's newest row | live: **A,C** |
|
||||
| 10:14:29 | **fault:** the unit's DB volume tar truncated to half (a torn / damaged copy) | 52 228 608 → 26 114 304 B |
|
||||
| 10:14:29 | the same restore | stop 10.8 s → `volume rm` → `tar: short read` → `[ERROR] volume restore … failed` → definition rewritten → „NOT replaying the database copy" → start → **postgres crash-loops** (`initdb: directory … exists but is not empty`) |
|
||||
| 10:14:44 | what the household reads | „Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1 stderr: ...twork rallly_rallly-internal Creating …" — **raw English Docker output** |
|
||||
| 10:15:47 | any copy of the live state (marker C)? | **none** — no undo dump, no live-volume copy for Rallly (`find` of `*restore-undo*`/`pre-restore-*` finds only a September paperless file) |
|
||||
| 10:15 | the dead-app guard | `rallly crossed the crash_loop threshold … while a backup/quiesce/update holds it — NOT stopped` |
|
||||
|
||||
**So:** the household's live data (marker C) was destroyed by the restore itself, the app does not start, and nothing
|
||||
on the box can put the pre-restore state back. The off-site restore got exactly this protection in v0.336.0 (D20 = A:
|
||||
„put back exactly as it was"); the LOCAL path — 07 §3's main customer restore — did not. Logs: `01-run.log`,
|
||||
`02-controller-log-fault-window.txt`. Teardown through the product (`03-teardown.txt`): Rallly removed with its data
|
||||
and backups; no container, no volume, no unit (the `/opt/docker/stacks/rallly` folder is the catalog template every
|
||||
app has).
|
||||
@@ -0,0 +1,153 @@
|
||||
#!/usr/bin/env python3
|
||||
"""r1_proof.py — C3 proof of candidate R1/D4 on scratch guest 9202 (data-loss hunt 2026-10-20).
|
||||
|
||||
Claim under test: a LOCAL (Tier-1 unit) restore that fails after it has started replacing the app's volumes
|
||||
restarts the app on whatever is left, and keeps NO copy of the live state it replaced — so the household's newer
|
||||
data is gone and nothing can put it back (unlike the off-site path, which got „put back exactly" in v0.336.0).
|
||||
|
||||
Method (mechanism proof — markers are written with psql inside the app's own database container, as the
|
||||
2026-10-19 R-893 measurement did; the household-facing acts are the dashboard's own endpoints):
|
||||
1 deploy Rallly (throwaway) through POST /api/stacks/rallly/deploy
|
||||
2 marker A; „Mentés most" = POST /api/backup/run (whole box, scratch); wait
|
||||
3 POSITIVE CONTROL: marker B (live), unit restore (POST /backup/restore) -> A present, B gone (expected rewind)
|
||||
4 marker C (the household's newest live row); THE FAULT: truncate the unit's DB volume tar to half
|
||||
5 unit restore again -> observe: the answer, the app's state, what the DB holds, any undo/safety copy on disk
|
||||
6 teardown: remove Rallly with its data and backups through the product; verify nothing left
|
||||
Env: CTL_BASE, CTL_DOMAIN, CTL_PW_ENV (see tools/ctl.py). Prints a timestamped log (no secrets).
|
||||
"""
|
||||
import json, os, re, subprocess, sys, time
|
||||
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/rehearsal-2026-10-20/tools")
|
||||
import ctl # noqa: E402
|
||||
|
||||
APP = "rallly"
|
||||
DOMAIN = os.environ["CTL_DOMAIN"]
|
||||
|
||||
|
||||
def say(*a):
|
||||
print(time.strftime("%H:%M:%S", time.gmtime()) + "Z " + " ".join(str(x) for x in a), flush=True)
|
||||
|
||||
|
||||
def guest(script, timeout=600):
|
||||
r = subprocess.run(["ssh", "demo-hp", "export LC_ALL=C; cat > /tmp/r1p.sh; pct push 9202 /tmp/r1p.sh /tmp/r1p.sh;"
|
||||
" pct exec 9202 -- bash /tmp/r1p.sh; pct exec 9202 -- rm -f /tmp/r1p.sh; rm -f /tmp/r1p.sh"],
|
||||
input=script, capture_output=True, text=True, timeout=timeout)
|
||||
return (r.stdout + r.stderr).strip()
|
||||
|
||||
|
||||
def psql(sql):
|
||||
return guest(f"""C=$(docker ps --format '{{{{.Names}}}}' | grep -E '^rallly.*(postgres|db)' | head -1)
|
||||
U=$(docker exec $C printenv POSTGRES_USER); D=$(docker exec $C printenv POSTGRES_DB)
|
||||
docker exec -i $C psql -v ON_ERROR_STOP=1 -U "$U" -d "$D" -tA <<'SQL'
|
||||
{sql}
|
||||
SQL""")
|
||||
|
||||
|
||||
def stack():
|
||||
code, d = ctl.call("GET", f"/api/stacks/{APP}")
|
||||
return (d.get("data") or {}) if isinstance(d, dict) else {}
|
||||
|
||||
|
||||
def wait_backup(t_cap=1200):
|
||||
t0 = time.time()
|
||||
time.sleep(5)
|
||||
while time.time() - t0 < t_cap:
|
||||
code, d = ctl.call("GET", "/api/backup/status")
|
||||
dd = (d.get("data") or {}) if isinstance(d, dict) else {}
|
||||
if not dd.get("running"):
|
||||
return dd
|
||||
time.sleep(5)
|
||||
return {"timeout": True}
|
||||
|
||||
|
||||
def restore(tag):
|
||||
code, d = ctl.call("GET", f"/api/backup/snapshots?stack={APP}")
|
||||
data = d.get("data") if isinstance(d, dict) else None
|
||||
snaps = data if isinstance(data, list) else next((data[k] for k in ("snapshots", "items", "restore_points")
|
||||
if isinstance(data, dict) and isinstance(data.get(k), list)), [])
|
||||
say(f"[{tag}] restore points offered: {json.dumps(snaps, ensure_ascii=False)[:400]}")
|
||||
sid = snaps[0].get("id") or snaps[0].get("snapshot_id") or snaps[0].get("short_id")
|
||||
out = subprocess.run(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", ctl.HOST, "-H", "Cookie: " + ctl._SESS,
|
||||
"-X", "POST", "--data-urlencode", "_csrf=" + ctl._CSRF, "--data-urlencode", "stack_name=" + APP,
|
||||
"--data-urlencode", "snapshot_id=" + sid, ctl.BASE + "/backup/restore"],
|
||||
capture_output=True, text=True).stdout
|
||||
say(f"[{tag}] POST /backup/restore -> {out.splitlines()[0] if out else '?'} "
|
||||
f"{[l for l in out.splitlines() if l.lower().startswith('location')][:1]}")
|
||||
t0, last = time.time(), None
|
||||
while time.time() - t0 < 900:
|
||||
code, d = ctl.call("GET", "/api/backup/restore-status")
|
||||
dd = (d.get("data") or {}) if isinstance(d, dict) else {}
|
||||
cur = json.dumps(dd, ensure_ascii=False)[:600]
|
||||
if cur != last:
|
||||
say(f"[{tag}] restore-status: {cur}")
|
||||
last = cur
|
||||
if not dd.get("running") and time.time() - t0 > 8:
|
||||
break
|
||||
time.sleep(3)
|
||||
time.sleep(20)
|
||||
st = stack()
|
||||
say(f"[{tag}] after: state={st.get('state')} hold={st.get('hold_reason')!r}")
|
||||
|
||||
|
||||
def markers(tag):
|
||||
say(f"[{tag}] DB markers: {psql('SELECT string_agg(m, \',\' ORDER BY m) FROM r1_marker;')!r}")
|
||||
|
||||
|
||||
def main():
|
||||
say("== 1 deploy")
|
||||
if not stack().get("deployed"):
|
||||
code, d = ctl.call("GET", f"/api/stacks/{APP}/deploy-fields")
|
||||
fields = (((d.get("data") or {}).get("metadata") or {}).get("deploy_fields")) or []
|
||||
vals = {"DOMAIN": DOMAIN, "SUBDOMAIN": "r1poll"}
|
||||
for f in fields:
|
||||
if f.get("env_var") in vals:
|
||||
continue
|
||||
if f.get("type") == "password":
|
||||
import secrets
|
||||
vals[f["env_var"]] = "R1-" + secrets.token_hex(10)
|
||||
elif f.get("required") and f.get("default"):
|
||||
vals[f["env_var"]] = f["default"]
|
||||
code, d = ctl.call("POST", f"/api/stacks/{APP}/deploy", {"values": vals})
|
||||
say("deploy ->", code, str(d)[:160])
|
||||
for _ in range(90):
|
||||
time.sleep(5)
|
||||
st = stack()
|
||||
if st.get("deployed") and (st.get("app_config") or {}).get("pinned_images") and st.get("state") in ("running", "unhealthy", "degraded"):
|
||||
break
|
||||
say("state:", stack().get("state"))
|
||||
time.sleep(30)
|
||||
say("== 2 marker A + backup")
|
||||
say(psql("CREATE TABLE IF NOT EXISTS r1_marker(m text primary key); INSERT INTO r1_marker VALUES ('A') ON CONFLICT DO NOTHING;"))
|
||||
markers("2")
|
||||
code, d = ctl.call("POST", "/api/backup/run")
|
||||
say("POST /api/backup/run ->", code, str(d)[:200])
|
||||
say("backup finished:", json.dumps(wait_backup(), ensure_ascii=False)[:500])
|
||||
unit = guest(f"for d in /mnt/felhom-drives/*/backups/primary/{APP} /mnt/sys_drive/felhom-data/backups/primary/{APP}; do "
|
||||
f"[ -d $d ] && echo UNIT $d && find $d -maxdepth 3 -type f -printf '%TY-%Tm-%Td %TH:%TM %s %p\\n'; done")
|
||||
say("unit on disk:\n" + unit)
|
||||
say("== 3 POSITIVE CONTROL: marker B (live) then a normal unit restore")
|
||||
say(psql("INSERT INTO r1_marker VALUES ('B');"))
|
||||
markers("3-before")
|
||||
restore("3")
|
||||
time.sleep(15)
|
||||
markers("3-after")
|
||||
say("== 4 marker C (the newest live row), then THE FAULT: the unit's DB volume tar truncated to half")
|
||||
say(psql("INSERT INTO r1_marker VALUES ('C');"))
|
||||
markers("4-live")
|
||||
say(guest(f"for d in /mnt/felhom-drives/*/backups/primary/{APP} /mnt/sys_drive/felhom-data/backups/primary/{APP}; do "
|
||||
f"[ -d $d ] || continue; f=$(find $d -name '*.tar' -path '*volume*' | grep -iE 'postgres|db' | head -1); "
|
||||
f"[ -n \"$f\" ] || f=$(find $d -name '*.tar' | head -1); s=$(stat -c %s \"$f\"); "
|
||||
f"truncate -s $((s/2)) \"$f\"; echo \"TRUNCATED $f $s -> $(stat -c %s \"$f\")\"; done"))
|
||||
say("== 5 the unit restore over the damaged copy")
|
||||
restore("5")
|
||||
time.sleep(30)
|
||||
markers("5-after")
|
||||
say("containers:\n" + guest("docker ps -a --format '{{.Names}} {{.Status}}' | grep -i rallly"))
|
||||
say("db container log tail:\n" + guest("C=$(docker ps -a --format '{{.Names}}' | grep -E '^rallly.*(postgres|db)' | head -1); docker logs --tail 15 $C 2>&1"))
|
||||
say("any undo / safety copy of the live state on disk?\n" + guest(
|
||||
"find / -xdev \\( -path /proc -o -path /sys \\) -prune -o \\( -iname '*restore-undo*' -o -iname 'pre-restore-*' \\) -print 2>/dev/null | head; "
|
||||
"for d in /mnt/felhom-drives/*/backups /mnt/sys_drive/felhom-data/backups; do find $d -maxdepth 4 -iname '*rallly*' -newer /etc/hostname 2>/dev/null | head; done"))
|
||||
say("controller log (restore lines):\n" + guest("docker logs --since 30m felhom-controller 2>&1 | grep -iE 'restore|rallly' | grep -vi 'GET /api' | tail -40"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,36 @@
|
||||
10:48:52Z == 1 deploy glance
|
||||
10:48:52Z deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||
10:49:12Z state: running family_gate in app_config: True
|
||||
10:49:34Z [1 POSITIVE CONTROL after install] gate file: /opt/docker/stacks/traefik/dynamic/family-gate-glance.yml
|
||||
10:49:34Z [1 POSITIVE CONTROL after install] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 401
|
||||
10:49:34Z == 2 export to the data drive
|
||||
10:49:34Z export start -> 200 {'ok': True}
|
||||
10:49:34Z /api/export/status {"display_name": "Glance", "done": false, "error": "", "job_type": "export", "ok": true, "output_path": "", "output_size": "", "running": true, "stack_name": "glance", "steps": [{"label": "Előkészítés", "status": "running"}, {"label": "Konfiguráció mentése", "status": "pending"}, {"label": "Adatbázis mentése", "status": "pending"}, {"label": "Felhasználói adatok", "status": "pending"}, {"label": "
|
||||
10:49:37Z /api/export/status {"display_name": "Glance", "done": true, "error": "", "job_type": "export", "ok": true, "output_path": "/mnt/felhom-drives/scratch_hdd/exports/glance_20261020-104935.fab", "output_size": "5.6 KB", "running": false, "stack_name": "glance", "steps": [{"label": "Előkészítés", "status": "done"}, {"label": "Konfiguráció mentése", "status": "done"}, {"label": "Adatbázis mentése", "status": "done"}, {"la
|
||||
10:49:37Z bundles for glance: []
|
||||
10:49:37Z == 3 remove glance (keep nothing)
|
||||
10:50:09Z remove -> 200 {'ok': True, 'data': {'removed': 'glance', 'volumes_removed': ['glance_glance_config'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alka
|
||||
10:50:21Z [3 after remove] gate file: /opt/docker/stacks/traefik/dynamic/family-gate-glance.yml
|
||||
10:50:21Z [3 after remove] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 404
|
||||
10:50:21Z == 4 import the bundle
|
||||
10:50:21Z import start -> 400 {'error': 'Missing path', 'ok': False}
|
||||
10:50:21Z /api/export/import/status {"display_name": "Glance", "done": true, "error": "", "job_type": "export", "ok": true, "output_path": "/mnt/felhom-drives/scratch_hdd/exports/glance_20261020-104935.fab", "output_size": "5.6 KB", "running": false, "stack_name": "glance", "steps": [{"label": "Előkészítés", "status": "done"}, {"label": "Konfiguráció mentése", "status": "done"}, {"label": "Adatbázis mentése", "status": "done"}, {"la
|
||||
10:53:42Z state after import: not_deployed family_gate in app_config: False app_config keys: []
|
||||
10:54:14Z [4 FAULT after import] gate file: (none)
|
||||
10:54:14Z [4 FAULT after import] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 404
|
||||
10:54:15Z controller log: 2026/10/20 10:48:52 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||
2026/10/20 10:48:58 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||
2026/10/20 10:49:34 family_gate.go:325: [DEBUG] [web] family gate glance: GET / without a pass — 401 (visitor 192.168.0.180)
|
||||
2026/10/20 10:49:35 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||
2026/10/20 10:49:35 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||
2026/10/20 10:49:37 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||
2026/10/20 10:49:39 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||
2026/10/20 10:49:43 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||
2026/10/20 10:50:09 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
|
||||
2026/10/20 10:50:09 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
|
||||
2026/10/20 10:50:21 auth.go:144: [DEBUG] [web] auth: valid session for POST /api/export/import
|
||||
2026/10/20 10:50:21 handler_export.go:328: [DEBUG] [web] apiImportStart: path="" hasPassword=false
|
||||
2026/10/20 10:50:29 family_gate.go:190: [INFO] [stacks] glance: removed the family-gate file of an app that is not installed
|
||||
2026/10/20 10:52:09 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
|
||||
2026/10/20 10:54:09 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
|
||||
exit 0
|
||||
@@ -0,0 +1,24 @@
|
||||
10:54:26Z == 4 import the bundle /mnt/felhom-drives/scratch_hdd/exports/glance_20261020-104935.fab
|
||||
10:54:26Z import start -> 200 {'ok': True}
|
||||
10:54:26Z /api/export/import/status {"display_name": "Glance", "done": false, "error": "", "job_type": "import", "ok": true, "output_path": "", "output_size": "", "running": true, "stack_name": "glance", "steps": [{"label": "Csomag megnyitása", "status": "done"}, {"label": "Alkalmazás előkészítése", "status": "done"}, {"label": "Konfiguráció visszaállítása", "status": "running"}, {"label": "Felhasználói adatok visszaállítása", "stat
|
||||
10:54:29Z /api/export/import/status {"display_name": "Glance", "done": false, "error": "", "job_type": "import", "ok": true, "output_path": "", "output_size": "", "running": true, "stack_name": "glance", "steps": [{"label": "Csomag megnyitása", "status": "done"}, {"label": "Alkalmazás előkészítése", "status": "done"}, {"label": "Konfiguráció visszaállítása", "status": "done"}, {"label": "Felhasználói adatok visszaállítása", "status"
|
||||
10:54:32Z /api/export/import/status {"display_name": "Glance", "done": true, "error": "", "job_type": "import", "ok": true, "output_path": "", "output_size": "", "running": false, "stack_name": "glance", "steps": [{"label": "Csomag megnyitása", "status": "done"}, {"label": "Alkalmazás előkészítése", "status": "done"}, {"label": "Konfiguráció visszaállítása", "status": "done"}, {"label": "Felhasználói adatok visszaállítása", "status"
|
||||
10:56:13Z state after import: running family_gate in app_config: False app_config keys: ['deployed', 'deployed_at', 'desired_state', 'env', 'installed_images', 'locked_fields']
|
||||
10:56:55Z [4 FAULT after import] gate file: (none)
|
||||
10:56:55Z [4 FAULT after import] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 200
|
||||
10:56:56Z controller log: 2026/10/20 10:54:33 manager.go:1814: [INFO] [stacks] glance glanceapp/glance:v0.8.5 running Up 3 seconds (health: starting)
|
||||
2026/10/20 10:54:39 healthprobe.go:187: [DEBUG] Health probe glance: HTTP GET :8080/ → 200 (1ms)
|
||||
2026/10/20 10:54:49 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 10s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 10:54:59 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 20s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 10:55:09 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 30s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 10:55:19 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 40s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 10:55:29 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 50s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 10:55:39 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m0s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 10:55:49 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m10s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 10:55:59 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m20s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 10:56:09 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m30s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 10:56:19 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m40s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 10:56:29 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m50s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 10:56:39 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 2m0s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 10:56:49 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 2m10s ago, effective interval 5m0s, healthy=true
|
||||
exit 0
|
||||
+24
@@ -0,0 +1,24 @@
|
||||
200
|
||||
{
|
||||
200
|
||||
{
|
||||
"ok": true,
|
||||
Exception in thread Thread-1 (pump):
|
||||
Traceback (most recent call last):
|
||||
File "/usr/lib/python3.13/threading.py", line 1043, in _bootstrap_inner
|
||||
self.run()
|
||||
~~~~~~~~^^
|
||||
File "/usr/lib/python3.13/threading.py", line 994, in run
|
||||
self._target(*self._args, **self._kwargs)
|
||||
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/mnt/5_hdd/felhom.eu/git/cc-secrets/ccsec/cli.py", line 153, in pump
|
||||
dst.write(m.close()); dst.flush()
|
||||
~~~~~~~~~^^
|
||||
BrokenPipeError: [Errno 32] Broken pipe
|
||||
Exception ignored on flushing sys.stdout:
|
||||
BrokenPipeError: [Errno 32] Broken pipe
|
||||
removed '/mnt/felhom-drives/scratch_hdd/exports/glance_20261020-104935.fab'
|
||||
0
|
||||
0
|
||||
0
|
||||
stranger GET after teardown: 404
|
||||
@@ -0,0 +1,23 @@
|
||||
# C3 — proven live: a `.fab` export → import makes a family-only app public
|
||||
|
||||
Scratch guest 9202, controller 0.331.0 (the `.fab` import code — `cmd/controller/main.go:3517-3530`
|
||||
`SaveEncryptedAppConfig`, `stacks/family_gate.go:173-190` — read the same at 0.336.0, `7ca59e2`). Architecture: `09` §3
|
||||
decisions 63/64 (the family gate), 221 (Glance only behind it), `07` §6.6 („never open by default").
|
||||
|
||||
| UTC | act (the dashboard's own endpoints) | observed (a stranger's GET through traefik; the gate file on disk) |
|
||||
|---|---|---|
|
||||
| 10:48:52 | `POST /api/stacks/glance/deploy` | running, `app_config.family_gate` set |
|
||||
| 10:49:34 | **control** | gate file `traefik/dynamic/family-gate-glance.yml` present; stranger → **401** |
|
||||
| 10:49:35 | `POST /api/export/start` (stop ticked, no password) | `glance_20261020-104935.fab`, 5.6 KB |
|
||||
| 10:50:09 | stop + `POST /api/stacks/glance/remove` (data + backups) | removed; 10:50:29 „removed the family-gate file of an app that is not installed" |
|
||||
| 10:54:26 | `POST /api/export/import` (the household's „Importálás") | import done in 6 s; app running |
|
||||
| 10:56:13 | the app's record | `app_config` keys: deployed, deployed_at, desired_state, env, installed_images, locked_fields — **no family_gate** |
|
||||
| 10:56:55 | **fault** | gate file **none**; stranger → **200** (Glance answers anyone who reaches the address) |
|
||||
|
||||
The first run's bundle-list parse was a harness slip (`01-run.log`, step 4 sent an empty path → 400, nothing done);
|
||||
step 4 re-ran alone with the path the export's own status gave (`02-import.log`, `step4.py`). Teardown
|
||||
(`03-teardown.txt`): Glance removed through the product, the bundle deleted (own file), no container, volume or gate
|
||||
file left; the address answers 404.
|
||||
|
||||
On a household box the address is reachable from the internet through the tunnel. Family-gated apps in the catalog:
|
||||
Glance, Grimmory, MeTube (the last two hold the family's books and videos).
|
||||
+96
@@ -0,0 +1,96 @@
|
||||
#!/usr/bin/env python3
|
||||
"""fab_gate_proof.py — C3 for R-1187 (e) on scratch 9202: does a `.fab` export → remove → import drop the family gate?
|
||||
|
||||
Glance has no login of its own; the catalog puts it ONLY behind the family gate (decision 221). Observables, each
|
||||
read from a DIFFERENT channel than the act: (1) the gate file `family-gate-glance.yml` on the guest's disk; (2) what a
|
||||
stranger's request to the app's host gets back from traefik (302 to the family page vs 200 from Glance).
|
||||
Acts are the dashboard's own endpoints: deploy, /api/export/start, stop+remove, /api/export/import.
|
||||
Env: CTL_BASE, CTL_DOMAIN, CTL_PW_ENV (tools/ctl.py).
|
||||
"""
|
||||
import json, os, subprocess, sys, time
|
||||
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/rehearsal-2026-10-20/tools")
|
||||
import ctl # noqa: E402
|
||||
|
||||
APP, SUB = "glance", "r1187glance"
|
||||
DOM = os.environ["CTL_DOMAIN"]
|
||||
|
||||
|
||||
def say(*a):
|
||||
print(time.strftime("%H:%M:%SZ", time.gmtime()), *a, flush=True)
|
||||
|
||||
|
||||
def guest(cmd):
|
||||
r = subprocess.run(["ssh", "demo-hp", f"export LC_ALL=C; pct exec 9202 -- bash -c {json.dumps(cmd)}"],
|
||||
capture_output=True, text=True, timeout=300)
|
||||
return (r.stdout + r.stderr).strip()
|
||||
|
||||
|
||||
def observe(tag):
|
||||
say(f"[{tag}] gate file:", guest("find / -xdev -name 'family-gate-glance.yml' 2>/dev/null | head -3; true") or "(none)")
|
||||
out = subprocess.run(["curl", "-sk", "-o", "/dev/null", "-w", "%{http_code} %{redirect_url}",
|
||||
"-H", f"Host: {SUB}.{DOM}", ctl.BASE + "/"], capture_output=True, text=True).stdout
|
||||
say(f"[{tag}] a stranger's GET https://{SUB}.{DOM}/ ->", out)
|
||||
|
||||
|
||||
def wait_stack(pred, n=60):
|
||||
for _ in range(n):
|
||||
code, d = ctl.call("GET", f"/api/stacks/{APP}")
|
||||
st = (d.get("data") or {}) if isinstance(d, dict) else {}
|
||||
if pred(st):
|
||||
return st
|
||||
time.sleep(5)
|
||||
return st
|
||||
|
||||
|
||||
def poll(path, n=120):
|
||||
last = None
|
||||
for _ in range(n):
|
||||
code, d = ctl.call("GET", path)
|
||||
dd = (d.get("data") or d) if isinstance(d, dict) else {}
|
||||
s = json.dumps(dd, ensure_ascii=False)[:400]
|
||||
if s != last:
|
||||
say(" ", path, s)
|
||||
last = s
|
||||
if isinstance(dd, dict) and dd.get("status") in ("completed", "failed", "done", "error") or (
|
||||
isinstance(dd, dict) and dd.get("running") is False and dd.get("status") not in ("running", "pending")):
|
||||
return dd
|
||||
time.sleep(3)
|
||||
return dd
|
||||
|
||||
|
||||
say("== 1 deploy glance")
|
||||
code, d = ctl.call("POST", f"/api/stacks/{APP}/deploy", {"values": {"DOMAIN": DOM, "SUBDOMAIN": SUB}})
|
||||
say("deploy ->", code, str(d)[:150])
|
||||
st = wait_stack(lambda s: s.get("deployed") and s.get("state") in ("running", "unhealthy", "degraded"))
|
||||
say("state:", st.get("state"), "family_gate in app_config:", bool((st.get("app_config") or {}).get("family_gate")))
|
||||
time.sleep(20)
|
||||
observe("1 POSITIVE CONTROL after install")
|
||||
|
||||
say("== 2 export to the data drive")
|
||||
drive = "/mnt/felhom-drives/scratch_hdd"
|
||||
code, d = ctl.call("POST", "/api/export/start", {"stack_name": APP, "dest_drive": drive, "password": "", "stop_app": True})
|
||||
say("export start ->", code, str(d)[:200])
|
||||
poll("/api/export/status")
|
||||
code, d = ctl.call("GET", "/api/export/bundles")
|
||||
bundles = [b for b in ((d.get("data") or {}).get("bundles") or d.get("data") or []) if isinstance(b, dict) and APP in json.dumps(b)]
|
||||
say("bundles for glance:", json.dumps(bundles, ensure_ascii=False)[:500])
|
||||
path = (bundles[-1].get("path") if bundles else None)
|
||||
|
||||
say("== 3 remove glance (keep nothing)")
|
||||
ctl.call("POST", f"/api/stacks/{APP}/stop")
|
||||
time.sleep(5)
|
||||
code, d = ctl.call("POST", f"/api/stacks/{APP}/remove", {"remove_hdd_data": True, "remove_backups": True})
|
||||
say("remove ->", code, str(d)[:160])
|
||||
time.sleep(10)
|
||||
observe("3 after remove")
|
||||
|
||||
say("== 4 import the bundle")
|
||||
code, d = ctl.call("POST", "/api/export/import", {"path": path, "password": ""})
|
||||
say("import start ->", code, str(d)[:200])
|
||||
poll("/api/export/import/status")
|
||||
st = wait_stack(lambda s: s.get("deployed") and s.get("state") in ("running", "unhealthy", "degraded"), n=40)
|
||||
say("state after import:", st.get("state"), "family_gate in app_config:", bool((st.get("app_config") or {}).get("family_gate")),
|
||||
"app_config keys:", sorted((st.get("app_config") or {}).keys()))
|
||||
time.sleep(30)
|
||||
observe("4 FAULT after import")
|
||||
say("controller log:", guest("docker logs --since 15m felhom-controller 2>&1 | grep -iE 'family|import' | grep -v 'GET /api' | tail -15"))
|
||||
@@ -0,0 +1,17 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Step 4 of fab_gate_proof.py re-run alone: the first run's bundle-list parse found nothing (harness slip); the
|
||||
export's own status gave the path."""
|
||||
import sys
|
||||
sys.argv = ["x"]
|
||||
exec(open("fab_gate_proof.py").read().split('say("== 1 deploy glance")')[0])
|
||||
path = "/mnt/felhom-drives/scratch_hdd/exports/glance_20261020-104935.fab"
|
||||
say("== 4 import the bundle", path)
|
||||
code, d = ctl.call("POST", "/api/export/import", {"path": path, "password": ""})
|
||||
say("import start ->", code, str(d)[:200])
|
||||
poll("/api/export/import/status")
|
||||
st = wait_stack(lambda s: s.get("deployed") and s.get("state") in ("running", "unhealthy", "degraded"), n=40)
|
||||
say("state after import:", st.get("state"), "family_gate in app_config:", bool((st.get("app_config") or {}).get("family_gate")),
|
||||
"app_config keys:", sorted((st.get("app_config") or {}).keys()))
|
||||
time.sleep(40)
|
||||
observe("4 FAULT after import")
|
||||
say("controller log:", guest("docker logs --since 6m felhom-controller 2>&1 | grep -iE 'family|import|glance' | grep -vE 'GET /api|ScanStacks|Env vars' | tail -15"))
|
||||
@@ -0,0 +1,7 @@
|
||||
# go test at controller 7ca59e2 (v0.336.0), scratch clone; 2026-10-20T11:11:31Z
|
||||
=== RUN TestRehearsalR12_LeftoverTempBlocksEveryRetry
|
||||
zz_rehearsal_r12_test.go:21: retry 1: err=fájlmásolás sikertelen: restoring appdata/app/p1.jpg: open /tmp/TestRehearsalR12_LeftoverTempBlocksEveryRetry3138873193/001/usb/appdata/app/p1.jpg.felhom-restoring: file exists; p1.jpg back=false; app restarted=true
|
||||
zz_rehearsal_r12_test.go:21: retry 2: err=fájlmásolás sikertelen: restoring appdata/app/p1.jpg: open /tmp/TestRehearsalR12_LeftoverTempBlocksEveryRetry3138873193/001/usb/appdata/app/p1.jpg.felhom-restoring: file exists; p1.jpg back=false; app restarted=true
|
||||
--- PASS: TestRehearsalR12_LeftoverTempBlocksEveryRetry (0.01s)
|
||||
PASS
|
||||
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.020s
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
package backup
|
||||
|
||||
// THROWAWAY proof for the 2026-10-20 data-loss hunt (R-1188 c). Never committed. A crash during the whole restore's
|
||||
// file copy leaves `<file>.felhom-restoring` behind (the copy writes it with O_EXCL); the retry then meets it.
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestRehearsalR12_LeftoverTempBlocksEveryRetry(t *testing.T) {
|
||||
f, _, liveLeg := wholeFixture(t)
|
||||
// what a power cut in the middle of copying p1.jpg leaves on the live drive
|
||||
if err := os.MkdirAll(liveLeg, 0o755); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
os.WriteFile(filepath.Join(liveLeg, "p1.jpg.felhom-restoring"), []byte("half"), 0o644)
|
||||
for i := 1; i <= 2; i++ {
|
||||
_, err := f.m.RestoreTier2Whole("app")
|
||||
_, statErr := os.Stat(filepath.Join(liveLeg, "p1.jpg"))
|
||||
t.Logf("retry %d: err=%v; p1.jpg back=%v; app restarted=%v", i, err, statErr == nil, f.fake.fullStarted)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
# go test at controller 7ca59e2 (v0.336.0), scratch clone, fake provider seams; 2026-10-20T10:13:05Z
|
||||
=== RUN TestRehearsalR5_WholeRestoreUnitRefusalAfterFileHalf
|
||||
zz_rehearsal_r5_test.go:22: live files BEFORE: map[p2.jpg:P2 live older]
|
||||
zz_rehearsal_r5_test.go:25: RestoreTier2Whole err = A(z) app adatbázis-másolata csonka: nem ér véget, ahogy egy teljes másolat. A visszaállítás biztonsági okból nem indult el — az alkalmazás és az adatai érintetlenek.
|
||||
zz_rehearsal_r5_test.go:26: file counts: {Restored:1 Replaced:1 KeptNewer:0 Unchanged:0 BytesTotal:17}
|
||||
zz_rehearsal_r5_test.go:27: live files AFTER: map[p1.jpg:P1 p2.jpg:P2 mirror NEWER p2.jpg.felhom-20261020T101306Z:P2 live older]
|
||||
zz_rehearsal_r5_test.go:28: app stopped=true, full start happened=false, provider calls=[stop]
|
||||
--- PASS: TestRehearsalR5_WholeRestoreUnitRefusalAfterFileHalf (0.01s)
|
||||
PASS
|
||||
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.019s
|
||||
+29
@@ -0,0 +1,29 @@
|
||||
package backup
|
||||
|
||||
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate R5). Never committed.
|
||||
import (
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestRehearsalR5_WholeRestoreUnitRefusalAfterFileHalf(t *testing.T) {
|
||||
cut := strings.Replace(pgDump(1), "PostgreSQL database dump complete", "", 1) // a cut-off copy (R-640)
|
||||
f := r102Tier2Fixture(t, []string{"app_data.tar"}, cut)
|
||||
f.m.stackProvider = fileAppProvider{f.fake}
|
||||
mid := time.Date(2026, 9, 22, 10, 0, 0, 0, time.UTC)
|
||||
mirrorLeg := filepath.Join(f.destBase, "hdd", "appdata", "app")
|
||||
liveLeg := filepath.Join(f.liveDrive, "appdata", "app")
|
||||
writeAt(t, filepath.Join(mirrorLeg, "p1.jpg"), "P1", mid)
|
||||
writeAt(t, filepath.Join(mirrorLeg, "p2.jpg"), "P2 mirror NEWER", mid.Add(72*time.Hour))
|
||||
writeAt(t, filepath.Join(liveLeg, "p2.jpg"), "P2 live older", mid.Add(48*time.Hour))
|
||||
before := fingerprint(t, liveLeg)
|
||||
t.Logf("live files BEFORE: %v", before)
|
||||
|
||||
res, err := f.m.RestoreTier2Whole("app")
|
||||
t.Logf("RestoreTier2Whole err = %v", err)
|
||||
t.Logf("file counts: %+v", res.Files)
|
||||
t.Logf("live files AFTER: %v", fingerprint(t, liveLeg))
|
||||
t.Logf("app stopped=%v, full start happened=%v, provider calls=%v", f.fake.stopped, f.fake.fullStarted, f.fake.calls)
|
||||
}
|
||||
+8
@@ -0,0 +1,8 @@
|
||||
# go test at controller 7ca59e2 (v0.336.0), scratch clone; the PRODUCTION rsyncMirror (rsync -a --delete) on local temp dirs; 2026-10-20T11:02:49Z
|
||||
=== RUN TestRehearsalT1_NightlyMirrorRunsDuringARestore
|
||||
zz_rehearsal_t1_test.go:26: POSITIVE CONTROL: mirror after the night = [a.jpg b.jpg c.jpg d.jpg]
|
||||
zz_rehearsal_t1_test.go:34: restore in progress: backup flag held = true
|
||||
zz_rehearsal_t1_test.go:37: FAULT: mirror DURING the restore = [a.jpg b.jpg] (c.jpg, d.jpg — the files the restore is bringing back — are gone from the only copy)
|
||||
--- PASS: TestRehearsalT1_NightlyMirrorRunsDuringARestore (0.22s)
|
||||
PASS
|
||||
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.230s
|
||||
+39
@@ -0,0 +1,39 @@
|
||||
package backup
|
||||
|
||||
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate T1 / R-1180). Never committed.
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type t1Provider struct{ *t2v2Provider }
|
||||
|
||||
func (p t1Provider) ListDeployedStacks() []StackSummary { return []StackSummary{{Name: "app"}} }
|
||||
|
||||
func TestRehearsalT1_NightlyMirrorRunsDuringARestore(t *testing.T) {
|
||||
m, src, target, prov := newTier2V2(t, "app")
|
||||
prov.has["app"] = true
|
||||
prov.binds["app"] = []ClassifiedBind{mHDD("appdata/app/photos")}
|
||||
m.stackProvider = t1Provider{prov}
|
||||
m.tier2Mirror = rsyncMirror // the PRODUCTION mirror (rsync -a --delete), local temp dirs only
|
||||
for _, n := range []string{"a.jpg", "b.jpg", "c.jpg", "d.jpg"} {
|
||||
mustWrite(t, filepath.Join(src, "appdata", "app", "photos", n), n)
|
||||
}
|
||||
m.RunAllTier2()
|
||||
mir := filepath.Join(target, "backups", "secondary", "app", "hdd", "appdata", "app", "photos")
|
||||
ls := func(d string) []string { e, _ := os.ReadDir(d); var o []string; for _, x := range e { o = append(o, x.Name()) }; return o }
|
||||
t.Logf("POSITIVE CONTROL: mirror after the night = %v", ls(mir))
|
||||
|
||||
// The household deleted half the photos and starts the second drive's restore; the restore holds the backup flag.
|
||||
os.Remove(filepath.Join(src, "appdata", "app", "photos", "c.jpg"))
|
||||
os.Remove(filepath.Join(src, "appdata", "app", "photos", "d.jpg"))
|
||||
if err := m.acquireRunning(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Logf("restore in progress: backup flag held = %v", m.acquireRunning() != nil)
|
||||
// 03:30 — the nightly second-drive leg fires while the restore is still copying.
|
||||
m.RunAllTier2()
|
||||
t.Logf("FAULT: mirror DURING the restore = %v (c.jpg, d.jpg — the files the restore is bringing back — are gone from the only copy)", ls(mir))
|
||||
m.releaseRunning()
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
// Replays hub cmd/hub/main.go scheduleDaily's arithmetic (felhom.eu 5c48731, lines 1057-1062) over
|
||||
// 2026-10-23..27 with a simulated clock, for each hub daily job. Prints every fire time.
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
func main() {
|
||||
bud, _ := time.LoadLocation("Europe/Budapest")
|
||||
jobs := []struct{ name string; h, m int }{{"db-snapshot", 2, 0}, {"prune", 4, 30}, {"deadline-check", 5, 0}, {"offsite-key-audit", 7, 10}}
|
||||
for _, j := range jobs {
|
||||
now := time.Date(2026, 10, 23, 12, 0, 0, 0, bud)
|
||||
end := time.Date(2026, 10, 27, 12, 0, 0, 0, bud)
|
||||
fmt.Printf("%s (%02d:%02d):\n", j.name, j.h, j.m)
|
||||
for now.Before(end) {
|
||||
n := now.In(bud)
|
||||
next := time.Date(n.Year(), n.Month(), n.Day(), j.h, j.m, 0, 0, bud)
|
||||
if !next.After(n) {
|
||||
next = next.Add(24 * time.Hour)
|
||||
}
|
||||
fmt.Printf(" fires %s (%s)\n", next.In(bud).Format("2006-01-02 15:04 MST"), next.UTC().Format("15:04Z"))
|
||||
now = next // fn() runs instantly; loop re-reads the clock at the fire instant
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
--- go run, go version go1.26.0 linux/amd64
|
||||
db-snapshot (02:00):
|
||||
fires 2026-10-24 02:00 CEST (00:00Z)
|
||||
fires 2026-10-25 02:00 CEST (00:00Z)
|
||||
fires 2026-10-25 02:00 CET (01:00Z)
|
||||
fires 2026-10-26 02:00 CET (01:00Z)
|
||||
fires 2026-10-27 02:00 CET (01:00Z)
|
||||
fires 2026-10-28 02:00 CET (01:00Z)
|
||||
prune (04:30):
|
||||
fires 2026-10-24 04:30 CEST (02:30Z)
|
||||
fires 2026-10-25 03:30 CET (02:30Z)
|
||||
fires 2026-10-25 04:30 CET (03:30Z)
|
||||
fires 2026-10-26 04:30 CET (03:30Z)
|
||||
fires 2026-10-27 04:30 CET (03:30Z)
|
||||
fires 2026-10-28 04:30 CET (03:30Z)
|
||||
deadline-check (05:00):
|
||||
fires 2026-10-24 05:00 CEST (03:00Z)
|
||||
fires 2026-10-25 04:00 CET (03:00Z)
|
||||
fires 2026-10-25 05:00 CET (04:00Z)
|
||||
fires 2026-10-26 05:00 CET (04:00Z)
|
||||
fires 2026-10-27 05:00 CET (04:00Z)
|
||||
fires 2026-10-28 05:00 CET (04:00Z)
|
||||
offsite-key-audit (07:10):
|
||||
fires 2026-10-24 07:10 CEST (05:10Z)
|
||||
fires 2026-10-25 06:10 CET (05:10Z)
|
||||
fires 2026-10-25 07:10 CET (06:10Z)
|
||||
fires 2026-10-26 07:10 CET (06:10Z)
|
||||
fires 2026-10-27 07:10 CET (06:10Z)
|
||||
fires 2026-10-28 07:10 CET (06:10Z)
|
||||
@@ -0,0 +1,42 @@
|
||||
# Part 1a — hub 0.162.0 deploy (operator yes 2026-10-20), read 2026-10-20T06:19:21Z
|
||||
manifest commit e1d349f1; sync of Deployment/hub only (only it was OutOfSync)
|
||||
argocd sync=Synced health=Healthy rev=b0ae2c574a8870e0e55455ceae77639b68a3d76f
|
||||
pod hub-8c44cf5cb-srth9 start 2026-10-20T06:03:39Z image gitea.dooplex.hu/admin/felhom-hub:0.162.0 imageID gitea.dooplex.hu/admin/felhom-hub@sha256:69285e640a9c5e4ce59b1fbb5cd94623fa47a71798e9d86fa0767ee6c69a6ac8
|
||||
|
||||
## start-up and R-1001 log lines
|
||||
2026/10/20 08:03:42 [INFO] felhom-hub 0.162.0 starting
|
||||
2026/10/20 08:04:19 [INFO] Host staleness checker initialized: 3 ok, 1 already alarmed (seeded silently, R-1001), 0 stale, 0 down (left unseeded → first Check emits; host_stale after 45m0s, host_down after 1h30m0s)
|
||||
2026/10/20 08:04:57 [INFO] Received report from tester-1 (5760 bytes)
|
||||
2026/10/20 08:04:57 [INFO] Received report from demo-felhom (6065 bytes)
|
||||
2026/10/20 08:04:58 [INFO] Received report from demo-hp (10036 bytes)
|
||||
2026/10/20 08:05:24 [INFO] restore-test staleness: Tester-2 pbs tier still unproven, already reported before the restart — not re-sent (R-1001)
|
||||
2026/10/20 08:05:24 [INFO] restore-test staleness: Tester-2 local tier still unproven, already reported before the restart — not re-sent (R-1001)
|
||||
2026/10/20 08:07:33 [INFO] Received report from demo-felhom (6081 bytes)
|
||||
2026/10/20 08:07:34 [INFO] Received report from tester-1 (5755 bytes)
|
||||
2026/10/20 08:07:36 [INFO] Received report from demo-hp (10004 bytes)
|
||||
2026/10/20 08:08:50 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 21418 bytes)
|
||||
2026/10/20 08:09:21 [INFO] host-report from tester-1-d70be4 (1 guests, 3 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 20577 bytes)
|
||||
2026/10/20 08:10:36 [INFO] host-report from demo-felhom-8363b5 (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 3 pbs-snapshots, 21708 bytes)
|
||||
|
||||
## events with id > 6129 (the last id before the restart):
|
||||
(none above = no event)
|
||||
|
||||
## before: each earlier restart re-sent Tester-2's alarms
|
||||
6055|host_down|2026-10-19 02:15:47
|
||||
6056|restore_test_stale|2026-10-19 02:15:51
|
||||
6057|restore_test_stale|2026-10-19 02:15:51
|
||||
6074|host_down|2026-10-19 09:30:04
|
||||
6075|restore_test_stale|2026-10-19 09:30:09
|
||||
6076|restore_test_stale|2026-10-19 09:30:09
|
||||
6087|host_down|2026-10-19 14:12:26
|
||||
6088|restore_test_stale|2026-10-19 14:12:58
|
||||
6089|restore_test_stale|2026-10-19 14:12:59
|
||||
|
||||
## floor
|
||||
min_controller_version|0.335.0
|
||||
min_controller_version_declared_min_agent|0.335.0=0.158.1
|
||||
|
||||
## Day-0 agent dropdown (GET /configuration, CC token)
|
||||
['', '0.160.0', '0.159.2', '0.159.1', '0.159.0', '0.158.1', '0.158.0', '0.157.0', '0.156.1', '0.156.0', '0.155.0', '0.154.0', '0.153.0', '0.152.0', '0.151.0', '0.150.0']
|
||||
|
||||
## mailbox (Gmail, second channel): last Tester-2 host_down / restore_test_stale mails = 2026-10-19T14:12:27Z / 14:12:59Z (the 0.161.0 restart); none after the 06:03 UTC restart; the daily 03:00Z expected_* mails present (mailbox live)
|
||||
@@ -0,0 +1,65 @@
|
||||
# Part 1b — agent 0.161.0 (operator yes 2026-10-20), read 2026-10-20T07:05:57Z
|
||||
|
||||
## release (scripts/release-agent.sh from a clean clone at 5152123)
|
||||
[release-agent] built ok: sha256 ff4e6dc52dd70234913445ecdeb88d31dea8e394d77c48704d26ff19b999d3b5
|
||||
[release-agent] config bundle built: sha256 485f36c957d83f772aa22cac24e5e73f44da5cf8b957b1fd513bec2bc1d12c47
|
||||
[release-agent] tagging v0.161.0 at 5152123 …
|
||||
[publish-agent] upload OK (HTTP 201)
|
||||
[publish-agent] round-trip GET verified (sha256 matches)
|
||||
[release-agent] config bundle published (HTTP 201)
|
||||
RELEASED — and installable, verified by download, not by this script's own say-so.
|
||||
sha256 : ff4e6dc52dd70234913445ecdeb88d31dea8e394d77c48704d26ff19b999d3b5
|
||||
bundle : 485f36c957d83f772aa22cac24e5e73f44da5cf8b957b1fd513bec2bc1d12c47 (felhom-config-bundle.json — vouch it with the agent)
|
||||
* [new tag] v0.161.0 -> v0.161.0
|
||||
|
||||
R-1083 NOT in this release: an agent key for 9202 would make its controller book whole-guest backups through
|
||||
the agent, one tier of which writes to ep0 (demo-hp namespace) — a change on ep0 the brief forbids; 9202 is
|
||||
also outside the felhom pool. Recorded in the R-1083 row.
|
||||
|
||||
## delivery: one signed agent_update job per box (felhom-opsign, key felhom-op-1)
|
||||
### demo-hp (signed 06:24:11Z)
|
||||
Oct 20 08:38:50 demo-hp felhom-agent[2622137]: time=2026-10-20T08:38:50.375+02:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=e3ca40956e0d5fc9 op=agent_update key_id=felhom-op-1 nonce=9e6c1034fbc50692fd1cbade967b32e6
|
||||
Oct 20 08:38:51 demo-hp felhom-selfupdate-guarded[791860]: applied 0.161.0 (prev 0.160.0, sha ff4e6dc52dd70234913445ecdeb88d31dea8e394d77c48704d26ff19b999d3b5); detached restart scheduled
|
||||
Oct 20 08:38:51 demo-hp felhom-agent[2622137]: time=2026-10-20T08:38:51.112+02:00 level=WARN msg="agent_update: signed update verified as root and handed off; restart scheduled" version=0.161.0 report="{\"agent_update\": {\"sha256\": \"ff4e6dc52dd70234913445ec
|
||||
Oct 20 08:38:51 demo-hp felhom-agent[2622137]: time=2026-10-20T08:38:51.112+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=e3ca40956e0d5fc9 op=agent_update
|
||||
Oct 20 08:38:53 demo-hp felhom-agent[791945]: time=2026-10-20T08:38:53.170+02:00 level=INFO msg="felhom-agent daemon starting" version=0.161.0 host_id=demo-hp-bb76ea hub_url=https://hub.felhom.eu interval_s=900
|
||||
Oct 20 08:38:54 demo-hp felhom-agent[791945]: time=2026-10-20T08:38:54.213+02:00 level=INFO msg="capabilities self-check" ok=76 total=76 degraded=0 inactive=0
|
||||
Oct 20 08:39:55 demo-hp felhom-agent[791945]: time=2026-10-20T08:39:55.311+02:00 level=WARN msg="selfupdate: update committed" version=0.161.0 wrapper=""
|
||||
### tester-1 (signed 06:42:27Z)
|
||||
Oct 20 08:54:21 felhom felhom-agent[2443463]: time=2026-10-20T08:54:21.074+02:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=9a6cc5c3d190a44c op=agent_update key_id=felhom-op-1 nonce=afe8d4d0a5389e1520b318ddf22ba9f2
|
||||
Oct 20 08:54:22 felhom felhom-agent[2443463]: time=2026-10-20T08:54:22.211+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=9a6cc5c3d190a44c op=agent_update
|
||||
Oct 20 08:54:24 felhom felhom-agent[560821]: time=2026-10-20T08:54:24.350+02:00 level=INFO msg="felhom-agent daemon starting" version=0.161.0 host_id=tester-1-d70be4 hub_url=https://hub.felhom.eu interval_s=900
|
||||
Oct 20 08:54:25 felhom felhom-agent[560821]: time=2026-10-20T08:54:25.325+02:00 level=INFO msg="capabilities self-check" ok=76 total=76 degraded=0 inactive=0
|
||||
Oct 20 08:55:26 felhom felhom-agent[560821]: time=2026-10-20T08:55:26.096+02:00 level=WARN msg="selfupdate: update committed" version=0.161.0 wrapper=""
|
||||
### demo-felhom (signed 06:42:27Z)
|
||||
Oct 20 08:55:28 demo-felhom felhom-agent[498196]: time=2026-10-20T08:55:28.000+02:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=3c9c9c5b9ab01f25 op=agent_update key_id=felhom-op-1 nonce=84e9007d7d8fc1e4fef0d5d3a5250840
|
||||
Oct 20 08:55:28 demo-felhom felhom-agent[498196]: time=2026-10-20T08:55:28.566+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=3c9c9c5b9ab01f25 op=agent_update
|
||||
Oct 20 08:55:30 demo-felhom felhom-agent[2429752]: time=2026-10-20T08:55:30.694+02:00 level=INFO msg="felhom-agent daemon starting" version=0.161.0 host_id=demo-felhom-8363b5 hub_url=https://hub.felhom.eu interval_s=900
|
||||
Oct 20 08:55:31 demo-felhom felhom-agent[2429752]: time=2026-10-20T08:55:31.379+02:00 level=INFO msg="capabilities self-check" ok=76 total=76 degraded=0 inactive=0
|
||||
Oct 20 08:56:32 demo-felhom felhom-agent[2429752]: time=2026-10-20T08:56:32.693+02:00 level=WARN msg="selfupdate: update committed" version=0.161.0 wrapper=""
|
||||
|
||||
## hub view (latest host report per box)
|
||||
demo-felhom-8363b5|0.161.0|2026-10-20 06:55:44
|
||||
demo-hp-bb76ea|0.161.0|2026-10-20 06:54:06
|
||||
tester-1-d70be4|0.161.0|2026-10-20 06:54:30
|
||||
|
||||
## host_busy, live on demo-hp (R-921): see 1b-host-busy-poll.txt — 3 free, 24 busy (backup:local), 21 free
|
||||
"Mentés most" press 06:41:12Z through POST /api/guest-backup/trigger (the dashboard endpoint), 200 started
|
||||
Oct 20 08:42:05 demo-hp felhom-agent[791945]: time=2026-10-20T08:42:05.179+02:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=local job=backup-9201-1792478523540456451
|
||||
Oct 20 08:49:31 demo-hp felhom-agent[791945]: time=2026-10-20T08:49:31.529+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_10_20-08_42_04.tar.zst size_bytes=7150133935 unco
|
||||
|
||||
## R-1070: the tunnel probe the agent runs (pct exec <vmid> -- docker inspect -f ... cloudflared)
|
||||
9202 (no tunnel): error: no such object: cloudflared rc=1
|
||||
9201 (tunnel): running rc=0
|
||||
fixture row in internal/hub/cloudflared_test.go: 25: {"absent, docker 29", "", "error: no such object: cloudflared", errors.New("exit status 1"), TunnelAbsent, "no tunnel set up"},
|
||||
|
||||
## tests on the released commit (5152123)
|
||||
--- PASS: TestClassifyTunnel (0.00s)
|
||||
ok gitea.dooplex.hu/admin/felhom-agent/internal/hub 0.011s
|
||||
--- PASS: TestHostHealthVerdict_R1070_NoTunnelSetUp (0.00s)
|
||||
ok gitea.dooplex.hu/admin/felhom-agent/internal/osupdate 0.013s
|
||||
--- PASS: TestBackupStatus_R921_HostBusy (0.00s)
|
||||
--- PASS: TestStaleLock_R1010_ForcesOnceOnlyWhenTheVolumeIsGone (0.00s)
|
||||
ok gitea.dooplex.hu/admin/felhom-agent/internal/localapi 0.022s
|
||||
--- PASS: TestDeleteSnapshotForce_R1010 (0.00s)
|
||||
ok gitea.dooplex.hu/admin/felhom-agent/internal/proxmox 0.009s
|
||||
@@ -0,0 +1,48 @@
|
||||
2026-10-20T06:41:12Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:41:30Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:41:48Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:42:06Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:42:24Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:42:42Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:43:02Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:43:20Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:43:38Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:44:11Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:44:28Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:44:46Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:45:04Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:45:26Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:45:44Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:46:05Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:46:23Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:46:42Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:47:00Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:47:18Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:47:36Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:47:54Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:48:12Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:48:30Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:48:48Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:49:05Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:49:23Z path= /backup/status http= 200 pin=ok keys= ['host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": true, "what": "backup:local"} tiers_in_flight= []
|
||||
2026-10-20T06:49:42Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:50:00Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:50:17Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:50:35Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:50:53Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:51:11Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:51:28Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:51:46Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:52:04Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:52:22Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:52:39Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:52:57Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:53:15Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:53:33Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:53:51Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:54:08Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:54:26Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:54:44Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:55:02Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:55:19Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
2026-10-20T06:55:37Z path= /backup/status http= 200 pin=ok keys= ['backup', 'host_busy', 'job_id', 'phase', 'tiers', 'vmid'] host_busy= {"busy": false} tiers_in_flight= []
|
||||
@@ -0,0 +1,11 @@
|
||||
# Part 1c — installer 1.35.0 publish (operator yes 2026-10-20), read 2026-10-20T06:20:23Z
|
||||
tag refs:
|
||||
1aa2e5585b7d13dc7b07fa9cc5e0711c34af291a refs/tags/installer-v1.35.0
|
||||
e1d349f1001b013fbcd2f26b25d71f9c649412a3 refs/tags/installer-v1.35.0^{}
|
||||
pins: commit b0ae2c57 (manifests/webpage.yaml, both git-sync refs); ArgoCD sync of Deployment/felhom-webpage only
|
||||
tag file sha256: f7a196fd5b3389e684aa0766968f29738e0b818d6ac8a79a7e7516fa96c99873
|
||||
served file sha256: f7a196fd5b3389e684aa0766968f29738e0b818d6ac8a79a7e7516fa96c99873
|
||||
served version: SCRIPT_VERSION="1.35.0"
|
||||
die_live_box lines in the served file: 4
|
||||
before (06:16Z): served 1.34.0, sha256 df722ee4... (= the 1.34.0 tag file)
|
||||
felhom-webpage-7f5c7968b5-g62t6 3/3 Running 0 2m49s
|
||||
@@ -0,0 +1,55 @@
|
||||
=== baseline Tue Oct 20 07:25:33 UTC 2026
|
||||
image file: gitea.dooplex.hu/admin/felhom-controller:0.331.0
|
||||
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 7 hours (healthy)
|
||||
paperless-postgres postgres:18-alpine Up 7 hours (healthy)
|
||||
paperless-redis redis:7-alpine Up 7 hours (healthy)
|
||||
immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up 7 hours (healthy)
|
||||
immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up 7 hours (healthy)
|
||||
immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up 7 hours (healthy)
|
||||
immich-redis redis:7-alpine Up 7 hours (healthy)
|
||||
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.331.0 Up 12 hours (healthy)
|
||||
filebrowser gtstef/filebrowser:1.5.6-stable Up 12 hours (healthy)
|
||||
traefik traefik:v3.7.13 Up 4 days
|
||||
--- images present at baseline (names only)
|
||||
alpine:latest
|
||||
curlimages/curl:8.11.1
|
||||
ghcr.io/paperless-ngx/paperless-ngx:2.20.15
|
||||
ghcr.io/puppeteer/puppeteer:latest
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.314.0-rc.1
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.326.0-rc.1
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.326.0-rc.2
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.327.0-rc.2
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.327.0-rc.3
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.327.0-rc.4
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.327.1-rc.1
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.331.0
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.331.0-rc.1
|
||||
gitea.dooplex.hu/admin/felhom-samba:1.2.0
|
||||
gtstef/filebrowser:1.3.3-stable
|
||||
gtstef/filebrowser:1.5.6-stable
|
||||
postgres:18-alpine
|
||||
redis:7-alpine
|
||||
redis:7.4-alpine
|
||||
traefik:v3.6.7
|
||||
traefik:v3.7.13
|
||||
--- rallly template dir (pre-existing)
|
||||
.felhom.yml
|
||||
docker-compose.yml
|
||||
hold-logs
|
||||
--- rallly volumes: 0
|
||||
/dev/loop1 69G 11G 54G 17% /var/lib/felhom
|
||||
--- offbox folder (listing only)
|
||||
total 20
|
||||
drwx------ 2 root root 4096 Sep 30 08:40 .
|
||||
drwxr-xr-x 4 root root 4096 Oct 20 07:25 ..
|
||||
-rw-r--r-- 1 root root 91 Sep 30 08:40 known_hosts
|
||||
-rw------- 1 root root 64 Sep 30 08:40 repo_password
|
||||
-rw------- 1 root root 411 Sep 30 08:40 ssh_key
|
||||
cb6b18f87662cf43d7e100f37366ce639ee5593fbf505751f5c1d831f8053fd1 settings.json
|
||||
-rw-r--r-- 1 root root 10240 Oct 20 07:25 /root/r893b-offbox.before.tar
|
||||
-rw-r--r-- 1 root root 27003 Oct 20 01:30 /root/r893b-settings.json.before
|
||||
offbox target set: False
|
||||
per-app offbox: None
|
||||
restore holds: ['chaoscrash', 'chaosoom', 'chaosoomb', 'gokapi']
|
||||
top-level keys: ['app_backup', 'app_update', 'app_update_events_seeded', 'backup_window_start', 'claim_consumed_generation', 'claimed', 'db_validations', 'filebrowser_admin_decided_at', 'filebrowser_admin_password_enc', 'filebrowser_admin_state', 'language', 'offbox_enlarge_notice_seeded', 'password_hash', 'restore_holds', 'storage_paths', 'unhealthy_stop_event_seeded', 'unhealthy_stops']
|
||||
--- controller [ERROR] lines since container start: 0
|
||||
@@ -0,0 +1,16 @@
|
||||
=== upgrade to 0.336.0-rc.1 Tue Oct 20 07:25:47 UTC 2026
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.336.0-rc.1
|
||||
|
||||
template parsing error: template: :1:23: executing "" at <.Config.Labels>: map has no entry for key "Labels"
|
||||
restart rc=0
|
||||
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.336.0-rc.1 Up 8 seconds (healthy)
|
||||
2026/10/20 07:25:49 undo.go:676: [INFO] [stacks] applied-meta backfill (R-646): recorded 0 []; skipped 0 [] — not current with the catalog, their pinned version's .felhom.yml is no longer on the box
|
||||
2026/10/20 07:25:49 scheduler.go:120: [INFO] [scheduler] Registered periodic job: conversion-copy-release (every 1h0m0s)
|
||||
2026/10/20 07:25:49 scheduler.go:85: [DEBUG] [scheduler] periodic job registered: name="conversion-copy-release" interval=1h0m0s totalJobs=13
|
||||
[ERROR] lines after start: 0
|
||||
image id sha256:0756d4d2b1c83a666c0d8b40b526983ac13fa792111a0ddbb8f430ba2f9588c3
|
||||
2026/10/20 07:25:49 main.go:353: [INFO] felhom-controller 0.336.0-rc.1 starting (customer: demo-hp, domain: enkisfelhom.hu)
|
||||
2026/10/20 07:25:49 server.go:315: [DEBUG] [web] NewServer: initializing web server v0.336.0-rc.1
|
||||
2026/10/20 07:25:50 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 7ea0989d5c8d felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.336.0-rc.1
|
||||
{"ok":true,"message":"felhom-controller is healthy"}
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
=== scratch SFTP target Tue Oct 20 07:27:37 UTC 2026
|
||||
bridge=172.17.0.2 felhom-filebrowser=172.21.0.3 traefik-public=172.18.0.2
|
||||
docker.io/atmoz/sftp:alpine
|
||||
5167fd39cf22cc30c1f0507e59377916d921640e12e08ddb4f6318db2c42fc34
|
||||
r893b-sftp atmoz/sftp:alpine Up 6 seconds 0.0.0.0:2222->22/tcp, [::]:2222->22/tcp
|
||||
[172.17.0.1]:2222 ssh-ed25519 AAAAC3NzaC
|
||||
=== off-site target configured through the household's endpoint 07:27:57
|
||||
07:27:57 POST /backup/offbox/config -> 302 Location: /backups/remote?flash_error=flash.offbox.target_saved_escrow_agent_down
|
||||
07:27:57 POST /backup/offbox/confirm-escrow -> 302 Location: /backups/remote?flash=flash.offbox.escrow_confirmed
|
||||
07:27:58 POST /backup/offbox/toggle -> 302 Location: /backups/remote?flash=flash.offbox.app_setting_updated
|
||||
@@ -0,0 +1,3 @@
|
||||
09:28:03 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||
09:28:53 [1] deployed, controller state=running, pinned={'rallly': 'lukevella/rallly:4.15.3', 'rallly-postgres': 'postgres:18-alpine'}
|
||||
deploy: True
|
||||
@@ -0,0 +1,23 @@
|
||||
=== A-written
|
||||
=== marker A written 07:29:13 UTC
|
||||
rallly lukevella/rallly:4.15.3 Up 43 seconds (healthy)
|
||||
rallly-postgres postgres:18-alpine Up 48 seconds (healthy)
|
||||
markers (phase=rows): A-before-snapshot-1=1
|
||||
r893_after exists: 0
|
||||
r893_only_old exists: 0
|
||||
r893_new2 exists: 0
|
||||
public tables: 34
|
||||
prisma migrations: 153
|
||||
db volume created: 2026-10-20T07:28:21Z
|
||||
schema-only md5: 8c473c01fccf52724e62a1d73867b2cc (control, second dump: 8c473c01fccf52724e62a1d73867b2cc)
|
||||
pg_dump: warning: there are circular foreign-key constraints among these tables:
|
||||
pg_dump: detail: users
|
||||
pg_dump: detail: provider_calendars
|
||||
pg_dump: detail: calendar_connections
|
||||
pg_dump: hint: You might not be able to restore the dump without using --disable-triggers or temporarily dropping the constraints.
|
||||
pg_dump: hint: Consider using a full dump instead of a --data-only dump to avoid this problem.
|
||||
data-only md5: b914f0553434b56f021043dd5946c2fe
|
||||
per-table row counts md5: 72061ad756a2f2d3842ad58d92879db2
|
||||
per-table row counts: _prisma_migrations=153 accounts=0 calendar_connections=0 comments=0 credentials=0 event_types=0 instance_licenses=0 instance_settings=1 license_validations=0 licenses=0 options=0 participants=0 payment_methods=0 poll_activities=0 poll_invites=0 polls=0 provider_calendars=0 r893_marker=1 registered_instances=0 rescheduled_event_dates=0 scheduled_event_invites=0 scheduled_events=0 sessions=0 sheet_slots=0 sheets=0 space_api_keys=0 space_member_invites=0 space_members=0 spaces=0 subscriptions=0 user_notification_preferences=0 users=0 verifications=0 votes=0
|
||||
definition: image: lukevella/rallly:4.15.3@sha256:8cd979aefe8d06e1822bc67054eb2d31088cdb831aa3f7b5615b3768f35431d9 image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
restore-undo folder: ls: cannot access '/mnt/sys_drive/felhom-data/backups/restore-undo/rallly': No such file or directory
|
||||
@@ -0,0 +1,18 @@
|
||||
07:29:29 POST /backup/offbox/run -> 302 Location: /backups/remote?flash=flash.offbox.run_started
|
||||
07:30:39 {"status": "ok", "snapshots": 1, "last_run": "2026-10-20T07:30:38Z", "last_duration": "1m8s", "repo_size_human": "7.2 MB", "last_error": ""}
|
||||
=== before snapshot 2 (markers B, C; r893_after, r893_only_old) 07:30:50 UTC
|
||||
rallly lukevella/rallly:4.15.3 Up 20 seconds (healthy)
|
||||
rallly-postgres postgres:18-alpine Up 25 seconds (healthy)
|
||||
markers (phase=rows): A-before-snapshot-1=1, B-after-snapshot-1=1, C-in-snapshot-2=1
|
||||
r893_after exists: 1 rows: 1
|
||||
r893_only_old exists: 1 rows: 1
|
||||
r893_new2 exists: 0
|
||||
public tables: 36
|
||||
prisma migrations: 153
|
||||
db volume created: 2026-10-20T07:28:21Z
|
||||
schema-only md5: f872bd54b5e09991876d465efb13d754 (control, second dump: f872bd54b5e09991876d465efb13d754)
|
||||
data-only md5: db5903abeeb4c3a6da32016ad71f01bb
|
||||
per-table row counts md5: c032377e7ce421083e40fb875cfadb34
|
||||
per-table row counts: _prisma_migrations=153 accounts=0 calendar_connections=0 comments=0 credentials=0 event_types=0 instance_licenses=0 instance_settings=1 license_validations=0 licenses=0 options=0 participants=0 payment_methods=0 poll_activities=0 poll_invites=0 polls=0 provider_calendars=0 r893_after=1 r893_marker=3 r893_only_old=1 registered_instances=0 rescheduled_event_dates=0 scheduled_event_invites=0 scheduled_events=0 sessions=0 sheet_slots=0 sheets=0 space_api_keys=0 space_member_invites=0 space_members=0 spaces=0 subscriptions=0 user_notification_preferences=0 users=0 verifications=0 votes=0
|
||||
definition: image: lukevella/rallly:4.15.3@sha256:8cd979aefe8d06e1822bc67054eb2d31088cdb831aa3f7b5615b3768f35431d9 image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
restore-undo folder: ls: cannot access '/mnt/sys_drive/felhom-data/backups/restore-undo/rallly': No such file or directory
|
||||
@@ -0,0 +1,2 @@
|
||||
07:30:56 POST /backup/offbox/run -> 302 Location: /backups/remote?flash=flash.offbox.run_started
|
||||
07:32:07 {"status": "ok", "snapshots": 2, "last_run": "2026-10-20T07:32:04Z", "last_duration": "1m6s", "repo_size_human": "10.1 MB", "last_error": ""}
|
||||
@@ -0,0 +1,21 @@
|
||||
=== LIVE before the restore 07:32:38 UTC
|
||||
rallly lukevella/rallly:4.15.3 Up 40 seconds (healthy)
|
||||
rallly-postgres postgres:18-alpine Up 45 seconds (healthy)
|
||||
markers (phase=rows): A-before-snapshot-1=1, B-after-snapshot-1=1, C-in-snapshot-2=1, D-after-snapshot-2=1
|
||||
r893_after exists: 1 rows: 1
|
||||
r893_only_old exists: 0
|
||||
r893_new2 exists: 1 rows: 1
|
||||
public tables: 36
|
||||
prisma migrations: 153
|
||||
db volume created: 2026-10-20T07:28:21Z
|
||||
schema-only md5: 05828c7714605cb5f2a6b3b691b52a22 (control, second dump: 05828c7714605cb5f2a6b3b691b52a22)
|
||||
data-only md5: e5c1a72ffb7107fef1d06e918b78194b
|
||||
per-table row counts md5: d8bcbc4ba364fe3fdee69a2e773cf5be
|
||||
per-table row counts: _prisma_migrations=153 accounts=0 calendar_connections=0 comments=0 credentials=0 event_types=0 instance_licenses=0 instance_settings=1 license_validations=0 licenses=0 options=0 participants=0 payment_methods=0 poll_activities=0 poll_invites=0 polls=0 provider_calendars=0 r893_after=1 r893_marker=4 r893_new2=1 registered_instances=0 rescheduled_event_dates=0 scheduled_event_invites=0 scheduled_events=0 sessions=0 sheet_slots=0 sheets=0 space_api_keys=0 space_member_invites=0 space_members=0 spaces=0 subscriptions=0 user_notification_preferences=0 users=0 verifications=0 votes=0
|
||||
definition: image: lukevella/rallly:4.15.3@sha256:8cd979aefe8d06e1822bc67054eb2d31088cdb831aa3f7b5615b3768f35431d9 image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
restore-undo folder: ls: cannot access '/mnt/sys_drive/felhom-data/backups/restore-undo/rallly': No such file or directory
|
||||
markers (phase=rows): A-before-snapshot-1=1, B-after-snapshot-1=1, C-in-snapshot-2=1, D-after-snapshot-2=1
|
||||
public tables: 36
|
||||
schema-only md5: 05828c7714605cb5f2a6b3b691b52a22 (control, second dump: 05828c7714605cb5f2a6b3b691b52a22)
|
||||
data-only md5: e5c1a72ffb7107fef1d06e918b78194b
|
||||
per-table row counts md5: d8bcbc4ba364fe3fdee69a2e773cf5be
|
||||
@@ -0,0 +1,27 @@
|
||||
07:32:57 POST /backup/offbox/restore -> 302 Location: /backups/restore/app?name=rallly&full_prep=rallly&full_size=66.1 MB
|
||||
07:32:57 POST /backup/offbox/restore -> 302 Location: /backups/restore/app?name=rallly&flash=flash.offbox.restore_started
|
||||
07:33:01 200 {"running": false, "op": "offbox-restore", "stack": "rallly", "started_at": "2026-10-20T07:32:57.875089287Z", "last": {"op": "offbox-restore", "stack": "rallly", "ok": true, "message": "A(z) rallly teljes mentése visszaállítva ellenőrző mappába: /mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly — a saját fájljaiddal együtt. A meglévő adatok változatlanok.", "finished_at": "2026-10-20T07:33:00.816791263Z"}, "last_recent": true}
|
||||
=== prepared scratch 07:33:13
|
||||
total 16
|
||||
drwxr-xr-x 3 root root 4096 Oct 20 07:33 .
|
||||
drwxr-xr-x 3 root root 4096 Oct 20 07:32 ..
|
||||
-rw------- 1 root root 86 Oct 20 07:33 .felhom-restore-complete.json
|
||||
drwxr-xr-x 3 root root 4096 Oct 14 14:05 mnt
|
||||
/mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps:
|
||||
total 120
|
||||
drwxr-xr-x 2 root root 4096 Oct 20 07:30 .
|
||||
drwxr-xr-x 5 root root 4096 Oct 20 07:31 ..
|
||||
-rw-r--r-- 1 root root 111959 Oct 20 07:30 rallly-postgres.sql
|
||||
|
||||
/mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly/mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps:
|
||||
total 67580
|
||||
drwxr-xr-x 2 root root 4096 Oct 20 07:31 .
|
||||
drwxr-xr-x 5 root root 4096 Oct 20 07:31 ..
|
||||
-rw-r--r-- 1 root root 69190656 Oct 20 07:31 rallly_rallly_postgres_data.tar
|
||||
snapshot-2 content check: r893_only_old mentions=5 r893_new2 mentions=0 marker C row=1 marker D row=0
|
||||
completion markers before: 1
|
||||
injected before line 2700:
|
||||
2699---
|
||||
2700:SELECT r893_forced_replay_failure(); -- R-893 option A proof 2026-10-20: forced error AFTER the volumes are replaced
|
||||
2701--- PostgreSQL database dump complete
|
||||
lines: 2705
|
||||
@@ -0,0 +1,5 @@
|
||||
07:33:40.967 restore-undo: rallly 4096;rallly/20261020T073340Z 4096;rallly/20261020T073340Z/rallly_rallly_postgres_data.tar.tmp 48553984;
|
||||
07:33:41.275 restore-undo: rallly 4096;rallly/20261020T073340Z 4096;rallly/20261020T073340Z/rallly_rallly_postgres_data.tar.tmp 69198848;
|
||||
07:33:41.583 restore-undo: rallly 4096;rallly/20261020T073340Z 4096;rallly/20261020T073340Z/rallly_rallly_postgres_data.tar 69198848;
|
||||
07:33:54.453 restore-undo: <absent or empty>
|
||||
watch end 07:35:57
|
||||
@@ -0,0 +1,3 @@
|
||||
=== reconstitute (forced failure) T0=2026-10-20T07:33:27Z
|
||||
07:33:28 POST /backup/offbox/reconstitute -> 302 Location: /backups/restore/app?name=rallly&flash=flash.offbox.full_restore_started
|
||||
07:33:55 200 {"running": false, "op": "offbox-reconstitute", "stack": "rallly", "started_at": "2026-10-20T07:33:28.242768479Z", "last": {"op": "offbox-reconstitute", "stack": "rallly", "ok": false, "message": "A teljes visszaállítás sikertelen: a(z) rallly visszaállítása sikertelen. Nem változott semmi: az alkalmazás adatai visszakerültek a visszaállítás előtti állapotba, és az alkalmazás újra fut", "finished_at": "2026-10-20T07:33:54.173657757Z"}, "last_recent": true}
|
||||
@@ -0,0 +1,307 @@
|
||||
2026/10/20 07:33:28 auth.go:160: [DEBUG] [web] login attempt: visitor 192.168.0.180 (peer 172.18.0.3:34320, X-Forwarded-For "192.168.0.180", CF-Connecting-IP "")
|
||||
2026/10/20 07:33:28 auth.go:192: [DEBUG] [web] login successful from 192.168.0.180, creating session
|
||||
2026/10/20 07:33:28 session_store.go:140: [DEBUG] [web] sessions: saved 145 session(s) to dashboard-sessions.json
|
||||
2026/10/20 07:33:28 auth.go:316: [DEBUG] [web] session created, expires=2026-10-27T07:33:28Z, active_sessions=145
|
||||
2026/10/20 07:33:28 auth.go:212: [INFO] [web] Login from 192.168.0.180
|
||||
2026/10/20 07:33:28 auth.go:144: [DEBUG] [web] auth: valid session for GET /
|
||||
2026/10/20 07:33:28 server.go:637: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.3:34320
|
||||
2026/10/20 07:33:28 auth.go:144: [DEBUG] [web] auth: valid session for GET /launcher
|
||||
2026/10/20 07:33:28 server.go:637: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.3:34320
|
||||
2026/10/20 07:33:28 auth.go:144: [DEBUG] [web] auth: valid session for POST /backup/offbox/reconstitute
|
||||
2026/10/20 07:33:28 server.go:637: [DEBUG] [web] ServeHTTP: POST /backup/offbox/reconstitute from 172.18.0.3:34320
|
||||
2026/10/20 07:33:28 auth.go:160: [DEBUG] [web] login attempt: visitor 192.168.0.180 (peer 172.18.0.3:34320, X-Forwarded-For "192.168.0.180", CF-Connecting-IP "")
|
||||
2026/10/20 07:33:28 auth.go:192: [DEBUG] [web] login successful from 192.168.0.180, creating session
|
||||
2026/10/20 07:33:28 session_store.go:140: [DEBUG] [web] sessions: saved 146 session(s) to dashboard-sessions.json
|
||||
2026/10/20 07:33:28 auth.go:316: [DEBUG] [web] session created, expires=2026-10-27T07:33:28Z, active_sessions=146
|
||||
2026/10/20 07:33:28 auth.go:212: [INFO] [web] Login from 192.168.0.180
|
||||
2026/10/20 07:33:28 auth.go:144: [DEBUG] [web] auth: valid session for GET /
|
||||
2026/10/20 07:33:28 server.go:637: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.3:34320
|
||||
2026/10/20 07:33:28 auth.go:144: [DEBUG] [web] auth: valid session for GET /launcher
|
||||
2026/10/20 07:33:28 server.go:637: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.3:34320
|
||||
2026/10/20 07:33:28 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:33:28 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:33:28 [WARN] [backup] ValidateDump: /mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql is structurally valid (36 tables) but its accounts table has NO rows — the dump may predate the customer's data
|
||||
2026/10/20 07:33:28 offbox_reconstitute.go:1253: [WARN] [offbox] rallly: the snapshot dump rallly-postgres.sql has no account rows — it may predate the customer's data
|
||||
2026/10/20 07:33:29 restore_dbcreds.go:114: [INFO] [backup] Restore rallly: database password(s) [DB_PASSWORD] come from the backup, [] stay as installed (older than the data) — values not logged (R-1106)
|
||||
2026/10/20 07:33:29 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers
|
||||
2026/10/20 07:33:29 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 74f5e7510c45 rallly rallly lukevella/rallly:4.15.3
|
||||
d0758337136e rallly-postgres rallly postgres:18-alpine
|
||||
2d8767d09c82 paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15
|
||||
5ac453c47c19 paperless-postgres paperless-ngx postgres:18-alpine
|
||||
ac5fbd6a551f paperless-redis paperless-ngx redis:7-alpine
|
||||
a5743e7ad6c9 immich-server immich ghcr.io/immich-app/immich-server:v3.2.4
|
||||
66a2711b7a0d immich-postgres immich ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0
|
||||
4e0f1ce...
|
||||
2026/10/20 07:33:29 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container rallly (image=lukevella/rallly:4.15.3, not a database)
|
||||
2026/10/20 07:33:29 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=d0758337136e)
|
||||
2026/10/20 07:33:29 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly
|
||||
2026/10/20 07:33:29 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database)
|
||||
2026/10/20 07:33:29 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5ac453c47c19)
|
||||
2026/10/20 07:33:29 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
|
||||
2026/10/20 07:33:29 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
|
||||
2026/10/20 07:33:29 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/20 07:33:29 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-server (image=ghcr.io/immich-app/immich-server:v3.2.4, not a database)
|
||||
2026/10/20 07:33:29 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: immich-postgres (id=66a2711b7a0d)
|
||||
2026/10/20 07:33:29 dbdump.go:198: [DEBUG] DiscoverDatabases: immich-postgres → stack=immich, dbUser=immich, dbName=immich
|
||||
2026/10/20 07:33:29 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-machine-learning (image=ghcr.io/immich-app/immich-machine-learning:v3.2.4, not a database)
|
||||
2026/10/20 07:33:29 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/20 07:33:29 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container r893b-sftp (image=atmoz/sftp:alpine, not a database)
|
||||
2026/10/20 07:33:29 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.6-stable, not a database)
|
||||
2026/10/20 07:33:29 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.336.0-rc.1, not a database)
|
||||
2026/10/20 07:33:29 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.13, not a database)
|
||||
2026/10/20 07:33:29 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 10 non-DB container(s)
|
||||
2026/10/20 07:33:29 dbdump.go:208: [INFO] [backup] Discovered 3 databases
|
||||
2026/10/20 07:33:29 dbdump.go:277: [DEBUG] DumpOne: starting dump for container=rallly-postgres, stack=rallly, dbType=postgres, dest=/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261020T073329Z-rallly-postgres.sql
|
||||
2026/10/20 07:33:29 dbdump.go:314: [DEBUG] DumpOne: pg_dump command: docker exec d0758337136e pg_dump -U rallly -d rallly --clean --if-exists --no-owner --no-privileges
|
||||
2026/10/20 07:33:29 [WARN] [backup] ValidateDump: /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261020T073329Z-rallly-postgres.sql is structurally valid (36 tables) but its accounts table has NO rows — the dump may predate the customer's data
|
||||
2026/10/20 07:33:29 dbdump.go:424: [DEBUG] DumpOne: completed rallly-postgres → pre-restore-20261020T073329Z-rallly-postgres.sql (size=109.4 KB, valid=true, tables=36, duration=298ms)
|
||||
2026/10/20 07:33:29 dbdump.go:429: [INFO] [backup] DB dump: rallly-postgres → pre-restore-20261020T073329Z-rallly-postgres.sql (109.4 KB, 298ms, 36 tables)
|
||||
2026/10/20 07:33:29 offbox_reconstitute.go:262: [INFO] [offbox] rallly: pre-restore safety dump written → pre-restore-20261020T073329Z-rallly-postgres.sql (109.4 KB)
|
||||
2026/10/20 07:33:29 manager.go:1437: [DEBUG] [stacks] StopStack rallly: current state=running deployed=true containers=2
|
||||
2026/10/20 07:33:29 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||
2026/10/20 07:33:29 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, SECRET_PASSWORD, SUBDOMAIN, IMPORT_PATH] (10 app + 0 system)
|
||||
2026/10/20 07:33:29 manager.go:1695: [DEBUG] Running: docker compose down (in /opt/docker/stacks/rallly)
|
||||
2026/10/20 07:33:29 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
|
||||
2026/10/20 07:33:29 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping immich — last check 1m50s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:33:29 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
|
||||
2026/10/20 07:33:29 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m10s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:33:29 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 1m10s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:33:29 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (3 skipped not due, 0 skipped no container)
|
||||
2026/10/20 07:33:30 manager.go:838: [INFO] [stacks] Status refresh: 11 containers across 70 stacks
|
||||
2026/10/20 07:33:31 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:33:31 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:33:34 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:33:34 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:33:34 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
|
||||
2026/10/20 07:33:37 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:33:37 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:33:39 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
|
||||
2026/10/20 07:33:39 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
|
||||
2026/10/20 07:33:40 manager.go:838: [INFO] [stacks] Status refresh: 11 containers across 70 stacks
|
||||
2026/10/20 07:33:40 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping immich — last check 2m0s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:33:40 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m20s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:33:40 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 1m20s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:33:40 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (3 skipped not due, 0 skipped no container)
|
||||
2026/10/20 07:33:40 manager.go:1717: [DEBUG] Command completed: docker compose down (took 10.8s)
|
||||
2026/10/20 07:33:40 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 10.8s)
|
||||
2026/10/20 07:33:40 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:33:40 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:33:40 manager.go:838: [INFO] [stacks] Status refresh: 9 containers across 70 stacks
|
||||
2026/10/20 07:33:41 offbox_restore_undo.go:154: [INFO] [offbox] rallly: the live volumes are copied before the restore (1 copied, 0 new from the snapshot) in 867ms — /mnt/sys_drive/felhom-data/backups/restore-undo/rallly/20261020T073340Z (R-893)
|
||||
2026/10/20 07:33:41 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly
|
||||
2026/10/20 07:33:42 restore.go:253: [DEBUG] [backup] Volume rallly_rallly_postgres_data restored successfully
|
||||
2026/10/20 07:33:42 restore.go:258: [INFO] [backup] Restored 1 Docker volume(s) for rallly
|
||||
2026/10/20 07:33:42 manager.go:1404: [INFO] [stacks] Starting stack rallly services only: [rallly-postgres]
|
||||
2026/10/20 07:33:42 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, SECRET_PASSWORD, SUBDOMAIN, IMPORT_PATH] (10 app + 0 system)
|
||||
2026/10/20 07:33:42 manager.go:1695: [DEBUG] Running: docker compose up -d rallly-postgres (in /opt/docker/stacks/rallly)
|
||||
2026/10/20 07:33:43 manager.go:1717: [DEBUG] Command completed: docker compose up -d rallly-postgres (took 0.4s)
|
||||
2026/10/20 07:33:43 manager.go:1415: [INFO] [stacks] Stack rallly services [rallly-postgres] started (took 0.4s)
|
||||
2026/10/20 07:33:43 manager.go:838: [INFO] [stacks] Status refresh: 10 containers across 70 stacks
|
||||
2026/10/20 07:33:43 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers
|
||||
2026/10/20 07:33:43 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 306e430406e4 rallly-postgres rallly postgres:18-alpine
|
||||
2d8767d09c82 paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15
|
||||
5ac453c47c19 paperless-postgres paperless-ngx postgres:18-alpine
|
||||
ac5fbd6a551f paperless-redis paperless-ngx redis:7-alpine
|
||||
a5743e7ad6c9 immich-server immich ghcr.io/immich-app/immich-server:v3.2.4
|
||||
66a2711b7a0d immich-postgres immich ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0
|
||||
4e0f1ce12bf2 immich-machine-learning immich ghcr.io/immich...
|
||||
2026/10/20 07:33:43 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=306e430406e4)
|
||||
2026/10/20 07:33:43 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly
|
||||
2026/10/20 07:33:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database)
|
||||
2026/10/20 07:33:43 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5ac453c47c19)
|
||||
2026/10/20 07:33:43 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
|
||||
2026/10/20 07:33:43 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
|
||||
2026/10/20 07:33:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/20 07:33:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-server (image=ghcr.io/immich-app/immich-server:v3.2.4, not a database)
|
||||
2026/10/20 07:33:43 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: immich-postgres (id=66a2711b7a0d)
|
||||
2026/10/20 07:33:43 dbdump.go:198: [DEBUG] DiscoverDatabases: immich-postgres → stack=immich, dbUser=immich, dbName=immich
|
||||
2026/10/20 07:33:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-machine-learning (image=ghcr.io/immich-app/immich-machine-learning:v3.2.4, not a database)
|
||||
2026/10/20 07:33:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/20 07:33:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container r893b-sftp (image=atmoz/sftp:alpine, not a database)
|
||||
2026/10/20 07:33:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.6-stable, not a database)
|
||||
2026/10/20 07:33:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.336.0-rc.1, not a database)
|
||||
2026/10/20 07:33:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.13, not a database)
|
||||
2026/10/20 07:33:43 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 9 non-DB container(s)
|
||||
2026/10/20 07:33:43 dbdump.go:208: [INFO] [backup] Discovered 3 databases
|
||||
2026/10/20 07:33:43 restore_db.go:87: [INFO] [backup] Restore rallly: replaying DB dump into rallly-postgres (postgres)
|
||||
2026/10/20 07:33:43 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/20 07:33:43 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:33:43 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:33:44 dbdump.go:814: [ERROR] [backup] postgres import into rallly-postgres FAILED: exit status 3 — engine output follows:
|
||||
ERROR: function r893_forced_replay_failure() does not exist
|
||||
LINE 1: SELECT r893_forced_replay_failure();
|
||||
^
|
||||
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
|
||||
2026/10/20 07:33:44 offbox_reconstitute.go:1100: [ERROR] [offbox] rallly: database replay failed, rolling back to the pre-restore state: importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3
|
||||
2026/10/20 07:33:44 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers
|
||||
2026/10/20 07:33:44 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 306e430406e4 rallly-postgres rallly postgres:18-alpine
|
||||
2d8767d09c82 paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15
|
||||
5ac453c47c19 paperless-postgres paperless-ngx postgres:18-alpine
|
||||
ac5fbd6a551f paperless-redis paperless-ngx redis:7-alpine
|
||||
a5743e7ad6c9 immich-server immich ghcr.io/immich-app/immich-server:v3.2.4
|
||||
66a2711b7a0d immich-postgres immich ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0
|
||||
4e0f1ce12bf2 immich-machine-learning immich ghcr.io/immich...
|
||||
2026/10/20 07:33:44 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=306e430406e4)
|
||||
2026/10/20 07:33:44 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly
|
||||
2026/10/20 07:33:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database)
|
||||
2026/10/20 07:33:44 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5ac453c47c19)
|
||||
2026/10/20 07:33:44 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
|
||||
2026/10/20 07:33:44 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
|
||||
2026/10/20 07:33:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/20 07:33:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-server (image=ghcr.io/immich-app/immich-server:v3.2.4, not a database)
|
||||
2026/10/20 07:33:44 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: immich-postgres (id=66a2711b7a0d)
|
||||
2026/10/20 07:33:44 dbdump.go:198: [DEBUG] DiscoverDatabases: immich-postgres → stack=immich, dbUser=immich, dbName=immich
|
||||
2026/10/20 07:33:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-machine-learning (image=ghcr.io/immich-app/immich-machine-learning:v3.2.4, not a database)
|
||||
2026/10/20 07:33:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/20 07:33:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container r893b-sftp (image=atmoz/sftp:alpine, not a database)
|
||||
2026/10/20 07:33:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.6-stable, not a database)
|
||||
2026/10/20 07:33:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.336.0-rc.1, not a database)
|
||||
2026/10/20 07:33:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.13, not a database)
|
||||
2026/10/20 07:33:44 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 9 non-DB container(s)
|
||||
2026/10/20 07:33:44 dbdump.go:208: [INFO] [backup] Discovered 3 databases
|
||||
2026/10/20 07:33:44 offbox_reconstitute.go:603: [DEBUG] [offbox] rallly: rallly-postgres was re-created during the restore (d0758337136e → 306e430406e4) — rolling back into the live container
|
||||
2026/10/20 07:33:44 offbox_reconstitute.go:606: [INFO] [offbox] rallly: rolling back to the pre-restore state from pre-restore-20261020T073329Z-rallly-postgres.sql
|
||||
2026/10/20 07:33:44 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261020T073329Z-rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/20 07:33:45 dbdump.go:820: [INFO] [backup] Imported DB dump pre-restore-20261020T073329Z-rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/20 07:33:45 offbox_reconstitute.go:611: [INFO] [offbox] rallly: rollback complete — 1 database(s) returned to the pre-restore state
|
||||
2026/10/20 07:33:45 manager.go:1437: [DEBUG] [stacks] StopStack rallly: current state=starting deployed=true containers=1
|
||||
2026/10/20 07:33:45 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||
2026/10/20 07:33:45 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DB_PASSWORD, DOMAIN, SECRET_PASSWORD, IMPORT_PATH] (10 app + 0 system)
|
||||
2026/10/20 07:33:45 manager.go:1695: [DEBUG] Running: docker compose down (in /opt/docker/stacks/rallly)
|
||||
2026/10/20 07:33:46 manager.go:1717: [DEBUG] Command completed: docker compose down (took 0.9s)
|
||||
2026/10/20 07:33:46 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 0.9s)
|
||||
2026/10/20 07:33:46 manager.go:838: [INFO] [stacks] Status refresh: 9 containers across 70 stacks
|
||||
2026/10/20 07:33:46 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly
|
||||
2026/10/20 07:33:46 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:33:46 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:33:47 restore.go:253: [DEBUG] [backup] Volume rallly_rallly_postgres_data restored successfully
|
||||
2026/10/20 07:33:47 restore.go:258: [INFO] [backup] Restored 1 Docker volume(s) for rallly
|
||||
2026/10/20 07:33:47 offbox_restore_undo.go:232: [WARN] [offbox] rallly: the restore failed (importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3) — the live volumes and definition are put back; starting the app (R-893)
|
||||
2026/10/20 07:33:47 notifier.go:1373: [WARN] notifier disabled (no hub configured): DROPPED event backup_run_failures (severity error) — further backup_run_failures events are logged at DEBUG only
|
||||
2026/10/20 07:33:47 manager.go:1348: [DEBUG] [stacks] StartStack rallly: current state=stopped deployed=true
|
||||
2026/10/20 07:33:47 manager.go:1351: [INFO] [stacks] Starting stack: rallly
|
||||
2026/10/20 07:33:47 manager.go:1358: [DEBUG] [stacks] StartStack rallly: prepared 10 env vars for compose
|
||||
2026/10/20 07:33:47 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, SECRET_PASSWORD, SUBDOMAIN, IMPORT_PATH] (10 app + 0 system)
|
||||
2026/10/20 07:33:47 manager.go:1695: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/rallly)
|
||||
2026/10/20 07:33:49 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:33:49 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:33:49 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true)
|
||||
2026/10/20 07:33:49 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714348KB) → used≈12498MB avail≈13400MB
|
||||
2026/10/20 07:33:49 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714348 availKB=15892428 → total=25898MB avail=13400MB used=12498MB (48.3%)
|
||||
2026/10/20 07:33:49 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=12.1GB avail=52.7GB (18.6%)
|
||||
2026/10/20 07:33:49 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=193.7GB avail=696.4GB (21.8%)
|
||||
2026/10/20 07:33:49 info.go:13: [DEBUG] [system] readLoadAvg: raw="1.95 2.29 1.70 5/1649 4507" → 1m=1.95 5m=2.29 15m=1.70
|
||||
2026/10/20 07:33:49 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones
|
||||
2026/10/20 07:33:49 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 51.2°C (hwmon2)
|
||||
2026/10/20 07:33:49 info.go:13: [DEBUG] [system] GetInfo done in 57ms — mem=12498MB/25898MB (48.3%), rootDisk=12.1GB/68.4GB (18.6%), load=1.95/2.29/1.70, temp=51.2°C (hwmon2), cpu=19.4%
|
||||
2026/10/20 07:33:49 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
|
||||
2026/10/20 07:33:49 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
|
||||
2026/10/20 07:33:49 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
|
||||
2026/10/20 07:33:49 scheduler.go:450: [INFO] [scheduler] Running job: stack-scan
|
||||
2026/10/20 07:33:49 scheduler.go:85: [DEBUG] [scheduler] job stack-scan: execution starting
|
||||
2026/10/20 07:33:49 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
|
||||
2026/10/20 07:33:49 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
|
||||
2026/10/20 07:33:49 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
|
||||
2026/10/20 07:33:50 manager.go:838: [INFO] [stacks] Status refresh: 11 containers across 70 stacks
|
||||
2026/10/20 07:33:50 manager.go:920: [DEBUG] [stacks] restart-policy of down member "rallly" = "unless-stopped"
|
||||
2026/10/20 07:33:50 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping immich — last check 2m10s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:33:50 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m30s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "actualbudget" deployed=false composePath=/opt/docker/stacks/actualbudget/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "adventurelog" deployed=false composePath=/opt/docker/stacks/adventurelog/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "audiobookshelf" deployed=false composePath=/opt/docker/stacks/audiobookshelf/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "bentopdf" deployed=false composePath=/opt/docker/stacks/bentopdf/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "bookstack" deployed=false composePath=/opt/docker/stacks/bookstack/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "calcom" deployed=false composePath=/opt/docker/stacks/calcom/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "calibre-web" deployed=false composePath=/opt/docker/stacks/calibre-web/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "chaoscrash" deployed=false composePath=/opt/docker/stacks/chaoscrash/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "chaosoom" deployed=false composePath=/opt/docker/stacks/chaosoom/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "chaosoomb" deployed=false composePath=/opt/docker/stacks/chaosoomb/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "claper" deployed=false composePath=/opt/docker/stacks/claper/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "code-server" deployed=false composePath=/opt/docker/stacks/code-server/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "crafty-controller" deployed=false composePath=/opt/docker/stacks/crafty-controller/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "dawarich" deployed=false composePath=/opt/docker/stacks/dawarich/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "docmost" deployed=false composePath=/opt/docker/stacks/docmost/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "emby" deployed=false composePath=/opt/docker/stacks/emby/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "filebrowser" deployed=false composePath=/opt/docker/stacks/filebrowser/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "ghost" deployed=false composePath=/opt/docker/stacks/ghost/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "gitea" deployed=false composePath=/opt/docker/stacks/gitea/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "glance" deployed=false composePath=/opt/docker/stacks/glance/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "gokapi" deployed=false composePath=/opt/docker/stacks/gokapi/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "grafana" deployed=false composePath=/opt/docker/stacks/grafana/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "gramps-web" deployed=false composePath=/opt/docker/stacks/gramps-web/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "grimmory" deployed=false composePath=/opt/docker/stacks/grimmory/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "grocy" deployed=false composePath=/opt/docker/stacks/grocy/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "home-assistant" deployed=false composePath=/opt/docker/stacks/home-assistant/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "homebox" deployed=false composePath=/opt/docker/stacks/homebox/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "homepage" deployed=false composePath=/opt/docker/stacks/homepage/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "immich" deployed=true composePath=/opt/docker/stacks/immich/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "jellyfin" deployed=false composePath=/opt/docker/stacks/jellyfin/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "karakeep" deployed=false composePath=/opt/docker/stacks/karakeep/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "kimai" deployed=false composePath=/opt/docker/stacks/kimai/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "komga" deployed=false composePath=/opt/docker/stacks/komga/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "lubelogger" deployed=false composePath=/opt/docker/stacks/lubelogger/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "mealie" deployed=false composePath=/opt/docker/stacks/mealie/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "metube" deployed=false composePath=/opt/docker/stacks/metube/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "n8n" deployed=false composePath=/opt/docker/stacks/n8n/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "navidrome" deployed=false composePath=/opt/docker/stacks/navidrome/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "nextcloud" deployed=false composePath=/opt/docker/stacks/nextcloud/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "nzbget" deployed=false composePath=/opt/docker/stacks/nzbget/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "onlyoffice" deployed=false composePath=/opt/docker/stacks/onlyoffice/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "opengist" deployed=false composePath=/opt/docker/stacks/opengist/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "outline" deployed=false composePath=/opt/docker/stacks/outline/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "paperless-ngx" deployed=true composePath=/opt/docker/stacks/paperless-ngx/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "papra" deployed=false composePath=/opt/docker/stacks/papra/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "plant-it" deployed=false composePath=/opt/docker/stacks/plant-it/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "plex" deployed=false composePath=/opt/docker/stacks/plex/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "privatebin" deployed=true composePath=/opt/docker/stacks/privatebin/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "prowlarr" deployed=false composePath=/opt/docker/stacks/prowlarr/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "qbittorrent" deployed=false composePath=/opt/docker/stacks/qbittorrent/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "radarr" deployed=false composePath=/opt/docker/stacks/radarr/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "radicale" deployed=false composePath=/opt/docker/stacks/radicale/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "rallly" deployed=true composePath=/opt/docker/stacks/rallly/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "romm" deployed=false composePath=/opt/docker/stacks/romm/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "sabnzbd" deployed=false composePath=/opt/docker/stacks/sabnzbd/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "seerr" deployed=false composePath=/opt/docker/stacks/seerr/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "sonarr" deployed=false composePath=/opt/docker/stacks/sonarr/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "sparkyfitness" deployed=false composePath=/opt/docker/stacks/sparkyfitness/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "tandoor" deployed=false composePath=/opt/docker/stacks/tandoor/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "termix" deployed=false composePath=/opt/docker/stacks/termix/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "traefik" deployed=false composePath=/opt/docker/stacks/traefik/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "transmission" deployed=false composePath=/opt/docker/stacks/transmission/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "uptime-kuma" deployed=false composePath=/opt/docker/stacks/uptime-kuma/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "vaultwarden" deployed=false composePath=/opt/docker/stacks/vaultwarden/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "vikunja" deployed=false composePath=/opt/docker/stacks/vikunja/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "wanderer" deployed=false composePath=/opt/docker/stacks/wanderer/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "wger" deployed=false composePath=/opt/docker/stacks/wger/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "wishlist" deployed=false composePath=/opt/docker/stacks/wishlist/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "zipline" deployed=false composePath=/opt/docker/stacks/zipline/docker-compose.yml
|
||||
2026/10/20 07:33:50 manager.go:1958: [DEBUG] [stacks] getCatalogTemplateSlugs: found 65 template slugs in /opt/docker/felhom-controller/data/catalog-cache/templates
|
||||
2026/10/20 07:33:50 manager.go:756: [DEBUG] [stacks] ScanStacks: catalog has 65 template slugs for orphan detection
|
||||
2026/10/20 07:33:50 manager.go:787: [INFO] [stacks] ScanStacks complete: 70 stacks found (4 deployed, 66 available)
|
||||
2026/10/20 07:33:50 manager.go:838: [INFO] [stacks] Status refresh: 11 containers across 70 stacks
|
||||
2026/10/20 07:33:50 scheduler.go:467: [INFO] [scheduler] Job stack-scan completed (took 181ms)
|
||||
2026/10/20 07:33:50 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(starting)]
|
||||
2026/10/20 07:33:50 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (2 skipped not due, 1 skipped no container)
|
||||
2026/10/20 07:33:52 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:33:52 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:33:53 manager.go:1717: [DEBUG] Command completed: docker compose up -d (took 6.2s)
|
||||
2026/10/20 07:33:53 manager.go:1366: [INFO] [stacks] Stack rallly started successfully (took 6.2s)
|
||||
2026/10/20 07:33:54 installed.go:409: [DEBUG] [stacks] installed-images rallly: unchanged (2 service(s)) — app.yaml not rewritten
|
||||
2026/10/20 07:33:54 manager.go:838: [INFO] [stacks] Status refresh: 11 containers across 70 stacks
|
||||
2026/10/20 07:33:54 offbox_handlers.go:547: [ERROR] [web] off-box reconstitute rallly (async): a(z) rallly visszaállítása sikertelen. Nem változott semmi: az alkalmazás adatai visszakerültek a visszaállítás előtti állapotba, és az alkalmazás újra fut
|
||||
2026/10/20 07:33:55 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:33:55 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:33:57 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, SECRET_PASSWORD, SUBDOMAIN, IMPORT_PATH] (10 app + 0 system)
|
||||
2026/10/20 07:33:57 manager.go:1695: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/rallly)
|
||||
2026/10/20 07:33:57 manager.go:1717: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.1s)
|
||||
2026/10/20 07:33:57 manager.go:1811: [INFO] [stacks] Stack rallly post-start status:
|
||||
2026/10/20 07:33:57 manager.go:1814: [INFO] [stacks] rallly lukevella/rallly:4.15.3@sha256:8cd979aefe8d06e1822bc67054eb2d31088cdb831aa3f7b5615b3768f35431d9 running Up 3 seconds (health: starting)
|
||||
2026/10/20 07:33:57 manager.go:1814: [INFO] [stacks] rallly-postgres postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873 running Up 9 seconds (healthy)
|
||||
2026/10/20 07:33:59 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
|
||||
2026/10/20 07:33:59 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
|
||||
2026/10/20 07:33:59 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping immich — last check 2m20s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:33:59 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m40s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:33:59 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (2 skipped not due, 0 skipped no container)
|
||||
2026/10/20 07:34:00 manager.go:838: [INFO] [stacks] Status refresh: 11 containers across 70 stacks
|
||||
2026/10/20 07:34:04 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
|
||||
@@ -0,0 +1,29 @@
|
||||
2026/10/20 07:33:28 auth.go:144: [DEBUG] [web] auth: valid session for POST /backup/offbox/reconstitute
|
||||
2026/10/20 07:33:28 server.go:637: [DEBUG] [web] ServeHTTP: POST /backup/offbox/reconstitute from 172.18.0.3:34320
|
||||
2026/10/20 07:33:28 offbox_reconstitute.go:1253: [WARN] [offbox] rallly: the snapshot dump rallly-postgres.sql has no account rows — it may predate the customer's data
|
||||
2026/10/20 07:33:29 restore_dbcreds.go:114: [INFO] [backup] Restore rallly: database password(s) [DB_PASSWORD] come from the backup, [] stay as installed (older than the data) — values not logged (R-1106)
|
||||
2026/10/20 07:33:29 offbox_reconstitute.go:262: [INFO] [offbox] rallly: pre-restore safety dump written → pre-restore-20261020T073329Z-rallly-postgres.sql (109.4 KB)
|
||||
2026/10/20 07:33:29 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||
2026/10/20 07:33:40 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 10.8s)
|
||||
2026/10/20 07:33:41 offbox_restore_undo.go:154: [INFO] [offbox] rallly: the live volumes are copied before the restore (1 copied, 0 new from the snapshot) in 867ms — /mnt/sys_drive/felhom-data/backups/restore-undo/rallly/20261020T073340Z (R-893)
|
||||
2026/10/20 07:33:41 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly
|
||||
2026/10/20 07:33:42 restore.go:253: [DEBUG] [backup] Volume rallly_rallly_postgres_data restored successfully
|
||||
2026/10/20 07:33:42 restore.go:258: [INFO] [backup] Restored 1 Docker volume(s) for rallly
|
||||
2026/10/20 07:33:42 manager.go:1404: [INFO] [stacks] Starting stack rallly services only: [rallly-postgres]
|
||||
2026/10/20 07:33:43 manager.go:1415: [INFO] [stacks] Stack rallly services [rallly-postgres] started (took 0.4s)
|
||||
2026/10/20 07:33:43 restore_db.go:87: [INFO] [backup] Restore rallly: replaying DB dump into rallly-postgres (postgres)
|
||||
2026/10/20 07:33:43 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/20 07:33:44 dbdump.go:814: [ERROR] [backup] postgres import into rallly-postgres FAILED: exit status 3 — engine output follows:
|
||||
2026/10/20 07:33:44 offbox_reconstitute.go:1100: [ERROR] [offbox] rallly: database replay failed, rolling back to the pre-restore state: importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3
|
||||
2026/10/20 07:33:44 offbox_reconstitute.go:603: [DEBUG] [offbox] rallly: rallly-postgres was re-created during the restore (d0758337136e → 306e430406e4) — rolling back into the live container
|
||||
2026/10/20 07:33:44 offbox_reconstitute.go:606: [INFO] [offbox] rallly: rolling back to the pre-restore state from pre-restore-20261020T073329Z-rallly-postgres.sql
|
||||
2026/10/20 07:33:44 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261020T073329Z-rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/20 07:33:45 offbox_reconstitute.go:611: [INFO] [offbox] rallly: rollback complete — 1 database(s) returned to the pre-restore state
|
||||
2026/10/20 07:33:45 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||
2026/10/20 07:33:46 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 0.9s)
|
||||
2026/10/20 07:33:46 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly
|
||||
2026/10/20 07:33:47 restore.go:253: [DEBUG] [backup] Volume rallly_rallly_postgres_data restored successfully
|
||||
2026/10/20 07:33:47 restore.go:258: [INFO] [backup] Restored 1 Docker volume(s) for rallly
|
||||
2026/10/20 07:33:47 offbox_restore_undo.go:232: [WARN] [offbox] rallly: the restore failed (importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3) — the live volumes and definition are put back; starting the app (R-893)
|
||||
2026/10/20 07:33:47 manager.go:1351: [INFO] [stacks] Starting stack: rallly
|
||||
2026/10/20 07:33:54 offbox_handlers.go:547: [ERROR] [web] off-box reconstitute rallly (async): a(z) rallly visszaállítása sikertelen. Nem változott semmi: az alkalmazás adatai visszakerültek a visszaállítás előtti állapotba, és az alkalmazás újra fut
|
||||
@@ -0,0 +1,24 @@
|
||||
=== AFTER the failed restore + put-back 07:34:25 UTC
|
||||
rallly lukevella/rallly:4.15.3 Up 31 seconds (healthy)
|
||||
rallly-postgres postgres:18-alpine Up 37 seconds (healthy)
|
||||
markers (phase=rows): A-before-snapshot-1=1, B-after-snapshot-1=1, C-in-snapshot-2=1, D-after-snapshot-2=1
|
||||
r893_after exists: 1 rows: 1
|
||||
r893_only_old exists: 0
|
||||
r893_new2 exists: 1 rows: 1
|
||||
public tables: 36
|
||||
prisma migrations: 153
|
||||
db volume created: 2026-10-20T07:33:46Z
|
||||
schema-only md5: 05828c7714605cb5f2a6b3b691b52a22 (control, second dump: 05828c7714605cb5f2a6b3b691b52a22)
|
||||
data-only md5: e5c1a72ffb7107fef1d06e918b78194b
|
||||
per-table row counts md5: d8bcbc4ba364fe3fdee69a2e773cf5be
|
||||
per-table row counts: _prisma_migrations=153 accounts=0 calendar_connections=0 comments=0 credentials=0 event_types=0 instance_licenses=0 instance_settings=1 license_validations=0 licenses=0 options=0 participants=0 payment_methods=0 poll_activities=0 poll_invites=0 polls=0 provider_calendars=0 r893_after=1 r893_marker=4 r893_new2=1 registered_instances=0 rescheduled_event_dates=0 scheduled_event_invites=0 scheduled_events=0 sessions=0 sheet_slots=0 sheets=0 space_api_keys=0 space_member_invites=0 space_members=0 spaces=0 subscriptions=0 user_notification_preferences=0 users=0 verifications=0 votes=0
|
||||
definition: image: lukevella/rallly:4.15.3@sha256:8cd979aefe8d06e1822bc67054eb2d31088cdb831aa3f7b5615b3768f35431d9 image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
restore-undo folder: ls: cannot access '/mnt/sys_drive/felhom-data/backups/restore-undo/rallly': No such file or directory
|
||||
restore-undo root: total 8 drwx------ 2 root root 4096 Oct 20 07:33 . drwxr-xr-x 5 root root 4096 Oct 20 07:33 ..
|
||||
live unit db-dumps: pre-restore-20261020T073329Z-rallly-postgres.sql rallly-postgres.sql
|
||||
restore holds: ['chaoscrash', 'chaosoom', 'chaosoomb', 'gokapi'] | rallly hold: None
|
||||
=== product view 07:34:31
|
||||
200 {"ok": true, "data": {"name": "rallly", "meta": {"display_name": "Rallly", "description": "Időpont szavazás (Doodle alternatíva)", "category": "productivity", "subdomain": "poll", "slug": "rallly", "catalog_since": "2026-09-30", "resources": {"mem_request": "384M", "mem_limit": "1024M", "pi_compatible": true, "needs_hdd": false, "hungarian_ui": false}, "deploy_fields": [{"env_var": "DOMAIN", "label": "Domain", "type": "domain", "generate": "", "default": "", "required": true, "placeholder": "", "description": "A szerver domain neve", "locked_after_deploy": true}, {"env_var": "SUBDOMAIN", "labe
|
||||
stack: {'state': 'running', 'deployed': True, 'status': None, 'held': None, 'hold': None, 'restore_hold': None, 'health': None}
|
||||
09:34:40 gate: poll is gated — passed as the household (cookie set)
|
||||
front door poll.enkisfelhom.hu/ -> 0 200
|
||||
@@ -0,0 +1,5 @@
|
||||
07:36:09 POST /backup/offbox/restore -> 302 Location: /backups/restore/app?name=rallly&full_prep=rallly&full_size=66.1 MB
|
||||
07:36:09 POST /backup/offbox/restore -> 302 Location: /backups/restore/app?name=rallly&flash=flash.offbox.restore_started
|
||||
07:36:13 200 {"running": false, "op": "offbox-restore", "stack": "rallly", "started_at": "2026-10-20T07:36:09.798980152Z", "last": {"op": "offbox-restore", "stack": "rallly", "ok": true, "message": "A(z) rallly teljes mentése visszaállítva ellenőrző mappába: /mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly — a saját fájljaiddal együtt. A meglévő adatok változatlanok.", "finished_at": "2026-10-20T07:36:12.377386791Z"}, "last_recent": true}
|
||||
=== re-prepared scratch 07:36:15: injected line count=0 (must be 0); completion markers=1; r893_only_old mentions=5; r893_new2 mentions=0; lines=2704
|
||||
rallly-postgres.sql
|
||||
@@ -0,0 +1,5 @@
|
||||
07:36:41.401 restore-undo: rallly 4096;rallly/20261020T073641Z 4096;
|
||||
07:36:41.709 restore-undo: rallly 4096;rallly/20261020T073641Z 4096;rallly/20261020T073641Z/rallly_rallly_postgres_data.tar.tmp 69198848;
|
||||
07:36:42.322 restore-undo: rallly 4096;rallly/20261020T073641Z 4096;rallly/20261020T073641Z/rallly_rallly_postgres_data.tar 69198848;
|
||||
07:36:57.965 restore-undo: <absent or empty>
|
||||
watch end 07:38:27
|
||||
@@ -0,0 +1,3 @@
|
||||
=== reconstitute (normal) T1=2026-10-20T07:36:28Z
|
||||
07:36:28 POST /backup/offbox/reconstitute -> 302 Location: /backups/restore/app?name=rallly&flash=flash.offbox.full_restore_started
|
||||
07:36:59 200 {"running": false, "op": "offbox-reconstitute", "stack": "rallly", "started_at": "2026-10-20T07:36:28.909561139Z", "last": {"op": "offbox-reconstitute", "stack": "rallly", "ok": true, "message": "A(z) rallly: 0 fájl és 1 adatkötet és az adatbázis visszaállítva (mentés: 2026. 10. 20. 09:30) — az alkalmazás újraindult.", "finished_at": "2026-10-20T07:36:57.852528534Z"}, "last_recent": true}
|
||||
@@ -0,0 +1,191 @@
|
||||
2026/10/20 07:36:28 auth.go:160: [DEBUG] [web] login attempt: visitor 192.168.0.180 (peer 172.18.0.3:34320, X-Forwarded-For "192.168.0.180", CF-Connecting-IP "")
|
||||
2026/10/20 07:36:28 auth.go:192: [DEBUG] [web] login successful from 192.168.0.180, creating session
|
||||
2026/10/20 07:36:28 session_store.go:140: [DEBUG] [web] sessions: saved 152 session(s) to dashboard-sessions.json
|
||||
2026/10/20 07:36:28 auth.go:316: [DEBUG] [web] session created, expires=2026-10-27T07:36:28Z, active_sessions=152
|
||||
2026/10/20 07:36:28 auth.go:212: [INFO] [web] Login from 192.168.0.180
|
||||
2026/10/20 07:36:28 auth.go:144: [DEBUG] [web] auth: valid session for GET /
|
||||
2026/10/20 07:36:28 server.go:637: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.3:34320
|
||||
2026/10/20 07:36:28 auth.go:144: [DEBUG] [web] auth: valid session for GET /launcher
|
||||
2026/10/20 07:36:28 server.go:637: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.3:34320
|
||||
2026/10/20 07:36:28 auth.go:144: [DEBUG] [web] auth: valid session for POST /backup/offbox/reconstitute
|
||||
2026/10/20 07:36:28 server.go:637: [DEBUG] [web] ServeHTTP: POST /backup/offbox/reconstitute from 172.18.0.3:34320
|
||||
2026/10/20 07:36:29 auth.go:160: [DEBUG] [web] login attempt: visitor 192.168.0.180 (peer 172.18.0.3:34320, X-Forwarded-For "192.168.0.180", CF-Connecting-IP "")
|
||||
2026/10/20 07:36:29 auth.go:192: [DEBUG] [web] login successful from 192.168.0.180, creating session
|
||||
2026/10/20 07:36:29 session_store.go:140: [DEBUG] [web] sessions: saved 153 session(s) to dashboard-sessions.json
|
||||
2026/10/20 07:36:29 auth.go:316: [DEBUG] [web] session created, expires=2026-10-27T07:36:29Z, active_sessions=153
|
||||
2026/10/20 07:36:29 auth.go:212: [INFO] [web] Login from 192.168.0.180
|
||||
2026/10/20 07:36:29 auth.go:144: [DEBUG] [web] auth: valid session for GET /
|
||||
2026/10/20 07:36:29 server.go:637: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.3:34320
|
||||
2026/10/20 07:36:29 auth.go:144: [DEBUG] [web] auth: valid session for GET /launcher
|
||||
2026/10/20 07:36:29 server.go:637: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.3:34320
|
||||
2026/10/20 07:36:29 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:36:29 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:36:29 [WARN] [backup] ValidateDump: /mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql is structurally valid (36 tables) but its accounts table has NO rows — the dump may predate the customer's data
|
||||
2026/10/20 07:36:29 offbox_reconstitute.go:1253: [WARN] [offbox] rallly: the snapshot dump rallly-postgres.sql has no account rows — it may predate the customer's data
|
||||
2026/10/20 07:36:29 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
|
||||
2026/10/20 07:36:29 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
|
||||
2026/10/20 07:36:30 manager.go:838: [INFO] [stacks] Status refresh: 11 containers across 70 stacks
|
||||
2026/10/20 07:36:30 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m10s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:36:30 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 2m10s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:36:30 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping immich — last check 4m50s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:36:30 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (3 skipped not due, 0 skipped no container)
|
||||
2026/10/20 07:36:30 restore_dbcreds.go:114: [INFO] [backup] Restore rallly: database password(s) [DB_PASSWORD] come from the backup, [] stay as installed (older than the data) — values not logged (R-1106)
|
||||
2026/10/20 07:36:30 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers
|
||||
2026/10/20 07:36:30 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 0be48f112fd3 rallly rallly lukevella/rallly:4.15.3
|
||||
28f60240bf89 rallly-postgres rallly postgres:18-alpine
|
||||
2d8767d09c82 paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15
|
||||
5ac453c47c19 paperless-postgres paperless-ngx postgres:18-alpine
|
||||
ac5fbd6a551f paperless-redis paperless-ngx redis:7-alpine
|
||||
a5743e7ad6c9 immich-server immich ghcr.io/immich-app/immich-server:v3.2.4
|
||||
66a2711b7a0d immich-postgres immich ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0
|
||||
4e0f1ce...
|
||||
2026/10/20 07:36:30 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container rallly (image=lukevella/rallly:4.15.3, not a database)
|
||||
2026/10/20 07:36:30 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=28f60240bf89)
|
||||
2026/10/20 07:36:30 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly
|
||||
2026/10/20 07:36:30 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database)
|
||||
2026/10/20 07:36:30 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5ac453c47c19)
|
||||
2026/10/20 07:36:30 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
|
||||
2026/10/20 07:36:30 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
|
||||
2026/10/20 07:36:30 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/20 07:36:30 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-server (image=ghcr.io/immich-app/immich-server:v3.2.4, not a database)
|
||||
2026/10/20 07:36:30 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: immich-postgres (id=66a2711b7a0d)
|
||||
2026/10/20 07:36:30 dbdump.go:198: [DEBUG] DiscoverDatabases: immich-postgres → stack=immich, dbUser=immich, dbName=immich
|
||||
2026/10/20 07:36:30 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-machine-learning (image=ghcr.io/immich-app/immich-machine-learning:v3.2.4, not a database)
|
||||
2026/10/20 07:36:30 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/20 07:36:30 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container r893b-sftp (image=atmoz/sftp:alpine, not a database)
|
||||
2026/10/20 07:36:30 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.6-stable, not a database)
|
||||
2026/10/20 07:36:30 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.336.0-rc.1, not a database)
|
||||
2026/10/20 07:36:30 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.13, not a database)
|
||||
2026/10/20 07:36:30 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 10 non-DB container(s)
|
||||
2026/10/20 07:36:30 dbdump.go:208: [INFO] [backup] Discovered 3 databases
|
||||
2026/10/20 07:36:30 dbdump.go:277: [DEBUG] DumpOne: starting dump for container=rallly-postgres, stack=rallly, dbType=postgres, dest=/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261020T073630Z-rallly-postgres.sql
|
||||
2026/10/20 07:36:30 dbdump.go:314: [DEBUG] DumpOne: pg_dump command: docker exec 28f60240bf89 pg_dump -U rallly -d rallly --clean --if-exists --no-owner --no-privileges
|
||||
2026/10/20 07:36:30 [WARN] [backup] ValidateDump: /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261020T073630Z-rallly-postgres.sql is structurally valid (36 tables) but its accounts table has NO rows — the dump may predate the customer's data
|
||||
2026/10/20 07:36:30 dbdump.go:424: [DEBUG] DumpOne: completed rallly-postgres → pre-restore-20261020T073630Z-rallly-postgres.sql (size=109.4 KB, valid=true, tables=36, duration=294ms)
|
||||
2026/10/20 07:36:30 dbdump.go:429: [INFO] [backup] DB dump: rallly-postgres → pre-restore-20261020T073630Z-rallly-postgres.sql (109.4 KB, 294ms, 36 tables)
|
||||
2026/10/20 07:36:30 offbox_reconstitute.go:262: [INFO] [offbox] rallly: pre-restore safety dump written → pre-restore-20261020T073630Z-rallly-postgres.sql (109.4 KB)
|
||||
2026/10/20 07:36:30 manager.go:1437: [DEBUG] [stacks] StopStack rallly: current state=running deployed=true containers=2
|
||||
2026/10/20 07:36:30 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||
2026/10/20 07:36:30 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DB_PASSWORD, DOMAIN, SECRET_PASSWORD, IMPORT_PATH] (10 app + 0 system)
|
||||
2026/10/20 07:36:30 manager.go:1695: [DEBUG] Running: docker compose down (in /opt/docker/stacks/rallly)
|
||||
2026/10/20 07:36:32 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:36:32 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:36:34 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
|
||||
2026/10/20 07:36:35 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:36:35 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:36:38 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:36:38 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:36:39 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
|
||||
2026/10/20 07:36:39 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
|
||||
2026/10/20 07:36:40 manager.go:838: [INFO] [stacks] Status refresh: 11 containers across 70 stacks
|
||||
2026/10/20 07:36:40 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m20s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:36:40 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 2m20s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:36:40 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (2 skipped not due, 0 skipped no container)
|
||||
2026/10/20 07:36:40 healthprobe.go:187: [DEBUG] Health probe immich: API GET :2283/api/server/ping → 200 (2ms)
|
||||
2026/10/20 07:36:40 healthprobe.go:167: [INFO] Health probes: 1 ok (of 1 probed)
|
||||
2026/10/20 07:36:41 manager.go:1717: [DEBUG] Command completed: docker compose down (took 10.7s)
|
||||
2026/10/20 07:36:41 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 10.7s)
|
||||
2026/10/20 07:36:41 manager.go:838: [INFO] [stacks] Status refresh: 9 containers across 70 stacks
|
||||
2026/10/20 07:36:41 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:36:41 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:36:42 offbox_restore_undo.go:154: [INFO] [offbox] rallly: the live volumes are copied before the restore (1 copied, 0 new from the snapshot) in 875ms — /mnt/sys_drive/felhom-data/backups/restore-undo/rallly/20261020T073641Z (R-893)
|
||||
2026/10/20 07:36:42 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly
|
||||
2026/10/20 07:36:43 restore.go:253: [DEBUG] [backup] Volume rallly_rallly_postgres_data restored successfully
|
||||
2026/10/20 07:36:43 restore.go:258: [INFO] [backup] Restored 1 Docker volume(s) for rallly
|
||||
2026/10/20 07:36:43 manager.go:1404: [INFO] [stacks] Starting stack rallly services only: [rallly-postgres]
|
||||
2026/10/20 07:36:43 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, SECRET_PASSWORD, SUBDOMAIN, IMPORT_PATH] (10 app + 0 system)
|
||||
2026/10/20 07:36:43 manager.go:1695: [DEBUG] Running: docker compose up -d rallly-postgres (in /opt/docker/stacks/rallly)
|
||||
2026/10/20 07:36:43 manager.go:1717: [DEBUG] Command completed: docker compose up -d rallly-postgres (took 0.4s)
|
||||
2026/10/20 07:36:43 manager.go:1415: [INFO] [stacks] Stack rallly services [rallly-postgres] started (took 0.4s)
|
||||
2026/10/20 07:36:43 manager.go:838: [INFO] [stacks] Status refresh: 10 containers across 70 stacks
|
||||
2026/10/20 07:36:43 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers
|
||||
2026/10/20 07:36:43 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: a449dce25682 rallly-postgres rallly postgres:18-alpine
|
||||
2d8767d09c82 paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15
|
||||
5ac453c47c19 paperless-postgres paperless-ngx postgres:18-alpine
|
||||
ac5fbd6a551f paperless-redis paperless-ngx redis:7-alpine
|
||||
a5743e7ad6c9 immich-server immich ghcr.io/immich-app/immich-server:v3.2.4
|
||||
66a2711b7a0d immich-postgres immich ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0
|
||||
4e0f1ce12bf2 immich-machine-learning immich ghcr.io/immich...
|
||||
2026/10/20 07:36:43 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=a449dce25682)
|
||||
2026/10/20 07:36:44 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly
|
||||
2026/10/20 07:36:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database)
|
||||
2026/10/20 07:36:44 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=5ac453c47c19)
|
||||
2026/10/20 07:36:44 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
|
||||
2026/10/20 07:36:44 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
|
||||
2026/10/20 07:36:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/20 07:36:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-server (image=ghcr.io/immich-app/immich-server:v3.2.4, not a database)
|
||||
2026/10/20 07:36:44 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: immich-postgres (id=66a2711b7a0d)
|
||||
2026/10/20 07:36:44 dbdump.go:198: [DEBUG] DiscoverDatabases: immich-postgres → stack=immich, dbUser=immich, dbName=immich
|
||||
2026/10/20 07:36:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-machine-learning (image=ghcr.io/immich-app/immich-machine-learning:v3.2.4, not a database)
|
||||
2026/10/20 07:36:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/20 07:36:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container r893b-sftp (image=atmoz/sftp:alpine, not a database)
|
||||
2026/10/20 07:36:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.6-stable, not a database)
|
||||
2026/10/20 07:36:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.336.0-rc.1, not a database)
|
||||
2026/10/20 07:36:44 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.13, not a database)
|
||||
2026/10/20 07:36:44 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 9 non-DB container(s)
|
||||
2026/10/20 07:36:44 dbdump.go:208: [INFO] [backup] Discovered 3 databases
|
||||
2026/10/20 07:36:44 restore_db.go:87: [INFO] [backup] Restore rallly: replaying DB dump into rallly-postgres (postgres)
|
||||
2026/10/20 07:36:44 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/20 07:36:44 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:36:44 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:36:45 dbdump.go:820: [INFO] [backup] Imported DB dump rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/20 07:36:45 restore_db.go:97: [INFO] [backup] Restore rallly: replayed 1 DB dump(s)
|
||||
2026/10/20 07:36:45 manager.go:1348: [DEBUG] [stacks] StartStack rallly: current state=starting deployed=true
|
||||
2026/10/20 07:36:45 manager.go:1351: [INFO] [stacks] Starting stack: rallly
|
||||
2026/10/20 07:36:45 manager.go:1358: [DEBUG] [stacks] StartStack rallly: prepared 10 env vars for compose
|
||||
2026/10/20 07:36:45 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, SECRET_PASSWORD, SUBDOMAIN, IMPORT_PATH] (10 app + 0 system)
|
||||
2026/10/20 07:36:45 manager.go:1695: [DEBUG] Running: docker compose up -d (in /opt/docker/stacks/rallly)
|
||||
2026/10/20 07:36:47 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:36:47 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:36:49 manager.go:1717: [DEBUG] Command completed: docker compose up -d (took 4.4s)
|
||||
2026/10/20 07:36:49 manager.go:1366: [INFO] [stacks] Stack rallly started successfully (took 4.4s)
|
||||
2026/10/20 07:36:49 installed.go:409: [DEBUG] [stacks] installed-images rallly: unchanged (2 service(s)) — app.yaml not rewritten
|
||||
2026/10/20 07:36:49 manager.go:838: [INFO] [stacks] Status refresh: 11 containers across 70 stacks
|
||||
2026/10/20 07:36:49 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true)
|
||||
2026/10/20 07:36:49 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
|
||||
2026/10/20 07:36:49 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
|
||||
2026/10/20 07:36:49 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
|
||||
2026/10/20 07:36:49 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping immich — last check 10s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:36:49 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m30s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:36:49 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (2 skipped not due, 0 skipped no container)
|
||||
2026/10/20 07:36:49 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
|
||||
2026/10/20 07:36:49 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
|
||||
2026/10/20 07:36:49 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
|
||||
2026/10/20 07:36:50 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714348KB) → used≈12410MB avail≈13488MB
|
||||
2026/10/20 07:36:50 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714348 availKB=15996140 → total=25898MB avail=13488MB used=12410MB (47.9%)
|
||||
2026/10/20 07:36:50 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=12.1GB avail=52.7GB (18.7%)
|
||||
2026/10/20 07:36:50 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=193.7GB avail=696.4GB (21.8%)
|
||||
2026/10/20 07:36:50 info.go:13: [DEBUG] [system] readLoadAvg: raw="1.48 1.89 1.65 6/1675 5879" → 1m=1.48 5m=1.89 15m=1.65
|
||||
2026/10/20 07:36:50 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones
|
||||
2026/10/20 07:36:50 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 51.1°C (hwmon2)
|
||||
2026/10/20 07:36:50 info.go:13: [DEBUG] [system] GetInfo done in 77ms — mem=12410MB/25898MB (47.9%), rootDisk=12.1GB/68.4GB (18.7%), load=1.48/1.89/1.65, temp=51.1°C (hwmon2), cpu=16.9%
|
||||
2026/10/20 07:36:50 manager.go:838: [INFO] [stacks] Status refresh: 11 containers across 70 stacks
|
||||
2026/10/20 07:36:50 main.go:2287: [INFO] [deadapp] check alive: 20 scans since boot, 4 deployed app(s) evaluated, 0 currently down
|
||||
2026/10/20 07:36:50 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:36:50 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:36:52 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DB_PASSWORD, DOMAIN, SECRET_PASSWORD, SUBDOMAIN, IMPORT_PATH] (10 app + 0 system)
|
||||
2026/10/20 07:36:52 manager.go:1695: [DEBUG] Running: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (in /opt/docker/stacks/rallly)
|
||||
2026/10/20 07:36:52 manager.go:838: [INFO] [stacks] Status refresh: 11 containers across 70 stacks
|
||||
2026/10/20 07:36:52 restore.go:298: [DEBUG] [backup] Post-restore health check: rallly not yet running, waiting...
|
||||
2026/10/20 07:36:52 manager.go:1717: [DEBUG] Command completed: docker compose ps -a --format table {{.Name}} {{.Image}} {{.State}} {{.Status}} (took 0.1s)
|
||||
2026/10/20 07:36:52 manager.go:1811: [INFO] [stacks] Stack rallly post-start status:
|
||||
2026/10/20 07:36:52 manager.go:1814: [INFO] [stacks] rallly lukevella/rallly:4.15.3@sha256:8cd979aefe8d06e1822bc67054eb2d31088cdb831aa3f7b5615b3768f35431d9 running Up 3 seconds (health: starting)
|
||||
2026/10/20 07:36:52 manager.go:1814: [INFO] [stacks] rallly-postgres postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873 running Up 9 seconds (healthy)
|
||||
2026/10/20 07:36:53 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:36:53 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:36:56 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:36:56 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:36:57 manager.go:838: [INFO] [stacks] Status refresh: 11 containers across 70 stacks
|
||||
2026/10/20 07:36:57 restore.go:293: [DEBUG] [backup] Post-restore health check: rallly is running
|
||||
2026/10/20 07:36:57 offbox_reconstitute.go:1158: [INFO] [offbox] reconstituted rallly from snapshot c3d94d03: 0 file(s) placed, 1 volume(s) replayed, 1 DB dump(s) replayed, safety dump=pre-restore-20261020T073630Z-rallly-postgres.sql, skewed=false
|
||||
2026/10/20 07:36:57 offbox_handlers.go:551: [INFO] [web] off-box reconstitute rallly completed (async): files=0 dbs=1 snapshot=c3d94d03
|
||||
2026/10/20 07:36:59 auth.go:144: [DEBUG] [web] auth: valid session for GET /api/backup/restore-status
|
||||
2026/10/20 07:36:59 router.go:78: [DEBUG] [api] GET /api/backup/restore-status (path=/backup/restore-status)
|
||||
2026/10/20 07:36:59 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
|
||||
2026/10/20 07:36:59 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
|
||||
2026/10/20 07:37:00 manager.go:838: [INFO] [stacks] Status refresh: 11 containers across 70 stacks
|
||||
2026/10/20 07:37:00 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping immich — last check 20s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:37:00 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 4m40s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/20 07:37:00 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 1 targets (2 skipped not due, 0 skipped no container)
|
||||
2026/10/20 07:37:00 healthprobe.go:187: [DEBUG] Health probe rallly: HTTP GET :3000/ → 200 (54ms)
|
||||
2026/10/20 07:37:00 healthprobe.go:167: [INFO] Health probes: 1 ok (of 1 probed)
|
||||
@@ -0,0 +1,23 @@
|
||||
2026/10/20 07:36:28 auth.go:144: [DEBUG] [web] auth: valid session for POST /backup/offbox/reconstitute
|
||||
2026/10/20 07:36:28 server.go:637: [DEBUG] [web] ServeHTTP: POST /backup/offbox/reconstitute from 172.18.0.3:34320
|
||||
2026/10/20 07:36:29 offbox_reconstitute.go:1253: [WARN] [offbox] rallly: the snapshot dump rallly-postgres.sql has no account rows — it may predate the customer's data
|
||||
2026/10/20 07:36:30 restore_dbcreds.go:114: [INFO] [backup] Restore rallly: database password(s) [DB_PASSWORD] come from the backup, [] stay as installed (older than the data) — values not logged (R-1106)
|
||||
2026/10/20 07:36:30 offbox_reconstitute.go:262: [INFO] [offbox] rallly: pre-restore safety dump written → pre-restore-20261020T073630Z-rallly-postgres.sql (109.4 KB)
|
||||
2026/10/20 07:36:30 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||
2026/10/20 07:36:41 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 10.7s)
|
||||
2026/10/20 07:36:42 offbox_restore_undo.go:154: [INFO] [offbox] rallly: the live volumes are copied before the restore (1 copied, 0 new from the snapshot) in 875ms — /mnt/sys_drive/felhom-data/backups/restore-undo/rallly/20261020T073641Z (R-893)
|
||||
2026/10/20 07:36:42 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly
|
||||
2026/10/20 07:36:43 restore.go:253: [DEBUG] [backup] Volume rallly_rallly_postgres_data restored successfully
|
||||
2026/10/20 07:36:43 restore.go:258: [INFO] [backup] Restored 1 Docker volume(s) for rallly
|
||||
2026/10/20 07:36:43 manager.go:1404: [INFO] [stacks] Starting stack rallly services only: [rallly-postgres]
|
||||
2026/10/20 07:36:43 manager.go:1415: [INFO] [stacks] Stack rallly services [rallly-postgres] started (took 0.4s)
|
||||
2026/10/20 07:36:44 restore_db.go:87: [INFO] [backup] Restore rallly: replaying DB dump into rallly-postgres (postgres)
|
||||
2026/10/20 07:36:44 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/20 07:36:45 dbdump.go:820: [INFO] [backup] Imported DB dump rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/20 07:36:45 restore_db.go:97: [INFO] [backup] Restore rallly: replayed 1 DB dump(s)
|
||||
2026/10/20 07:36:45 manager.go:1351: [INFO] [stacks] Starting stack: rallly
|
||||
2026/10/20 07:36:49 manager.go:1366: [INFO] [stacks] Stack rallly started successfully (took 4.4s)
|
||||
2026/10/20 07:36:52 restore.go:298: [DEBUG] [backup] Post-restore health check: rallly not yet running, waiting...
|
||||
2026/10/20 07:36:57 restore.go:293: [DEBUG] [backup] Post-restore health check: rallly is running
|
||||
2026/10/20 07:36:57 offbox_reconstitute.go:1158: [INFO] [offbox] reconstituted rallly from snapshot c3d94d03: 0 file(s) placed, 1 volume(s) replayed, 1 DB dump(s) replayed, safety dump=pre-restore-20261020T073630Z-rallly-postgres.sql, skewed=false
|
||||
2026/10/20 07:36:57 offbox_handlers.go:551: [INFO] [web] off-box reconstitute rallly completed (async): files=0 dbs=1 snapshot=c3d94d03
|
||||
@@ -0,0 +1,20 @@
|
||||
=== AFTER the normal restore of snapshot 2 07:38:31 UTC
|
||||
rallly lukevella/rallly:4.15.3 Up About a minute (healthy)
|
||||
rallly-postgres postgres:18-alpine Up About a minute (healthy)
|
||||
markers (phase=rows): A-before-snapshot-1=1, B-after-snapshot-1=1, C-in-snapshot-2=1
|
||||
r893_after exists: 1 rows: 1
|
||||
r893_only_old exists: 1 rows: 1
|
||||
r893_new2 exists: 0
|
||||
public tables: 36
|
||||
prisma migrations: 153
|
||||
db volume created: 2026-10-20T07:36:42Z
|
||||
schema-only md5: f872bd54b5e09991876d465efb13d754 (control, second dump: f872bd54b5e09991876d465efb13d754)
|
||||
data-only md5: 1e688d1bc96fba7e5140f391707a762a
|
||||
per-table row counts md5: c032377e7ce421083e40fb875cfadb34
|
||||
per-table row counts: _prisma_migrations=153 accounts=0 calendar_connections=0 comments=0 credentials=0 event_types=0 instance_licenses=0 instance_settings=1 license_validations=0 licenses=0 options=0 participants=0 payment_methods=0 poll_activities=0 poll_invites=0 polls=0 provider_calendars=0 r893_after=1 r893_marker=3 r893_only_old=1 registered_instances=0 rescheduled_event_dates=0 scheduled_event_invites=0 scheduled_events=0 sessions=0 sheet_slots=0 sheets=0 space_api_keys=0 space_member_invites=0 space_members=0 spaces=0 subscriptions=0 user_notification_preferences=0 users=0 verifications=0 votes=0
|
||||
definition: image: lukevella/rallly:4.15.3@sha256:8cd979aefe8d06e1822bc67054eb2d31088cdb831aa3f7b5615b3768f35431d9 image: postgres:18-alpine@sha256:77f585114c32fbca283dc835b0596f4e52b51b4c6662d7810b2f4084f60a1873
|
||||
restore-undo folder: ls: cannot access '/mnt/sys_drive/felhom-data/backups/restore-undo/rallly': No such file or directory
|
||||
restore holds: ['chaoscrash', 'chaosoom', 'chaosoomb', 'gokapi']
|
||||
stack: {'state': 'running', 'deployed': True, 'status': None, 'held': None, 'hold': None, 'restore_hold': None, 'health': None}
|
||||
09:38:37 gate: poll is gated — passed as the household (cookie set)
|
||||
front door poll.enkisfelhom.hu/ -> 0 200
|
||||
@@ -0,0 +1,79 @@
|
||||
10:43:11 [X] stop -> 200 {'ok': True, 'message': 'Stack rallly stop completed'}
|
||||
10:43:43 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'rallly', 'volumes_removed': ['rallly_rallly_postgres_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': '
|
||||
10:43:52 [X] after remove: deployed=False leftovers='/opt/docker/stacks/rallly'
|
||||
remove: 200
|
||||
=== machine teardown 08:44:07
|
||||
rallly stack dir: .felhom.yml docker-compose.yml hold-logs (template dir pre-existed with the same three entries)
|
||||
rallly containers: 0; rallly volumes: 0
|
||||
ls: cannot access '/mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly': No such file or directory
|
||||
ls: cannot access '/mnt/sys_drive/felhom-data/backups/primary/rallly': No such file or directory
|
||||
r893b-sftp
|
||||
2
|
||||
restore-undo (empty, created by this drill) removed
|
||||
cb6b18f87662cf43d7e100f37366ce639ee5593fbf505751f5c1d831f8053fd1 settings.json
|
||||
total 20
|
||||
drwx------ 2 root root 4096 Sep 30 08:40 .
|
||||
drwxr-xr-x 4 root root 4096 Oct 20 08:44 ..
|
||||
-rw-r--r-- 1 root root 91 Sep 30 08:40 known_hosts
|
||||
-rw------- 1 root root 64 Sep 30 08:40 repo_password
|
||||
-rw------- 1 root root 411 Sep 30 08:40 ssh_key
|
||||
restart rc=0
|
||||
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.331.0 Up 8 seconds (healthy)
|
||||
2026/10/20 08:44:08 main.go:353: [INFO] felhom-controller 0.331.0 starting (customer: demo-hp, domain: enkisfelhom.hu)
|
||||
[ERROR] lines after start: 0
|
||||
2
|
||||
0
|
||||
=== after teardown 08:45:17
|
||||
image file: gitea.dooplex.hu/admin/felhom-controller:0.331.0
|
||||
filebrowser gtstef/filebrowser:1.5.6-stable Up About a minute (healthy)
|
||||
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.331.0 Up About a minute (healthy)
|
||||
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up About an hour (healthy)
|
||||
paperless-postgres postgres:18-alpine Up About an hour (healthy)
|
||||
paperless-redis redis:7-alpine Up About an hour (healthy)
|
||||
immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up About an hour (healthy)
|
||||
immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up About an hour (healthy)
|
||||
immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up About an hour (healthy)
|
||||
immich-redis redis:7-alpine Up About an hour (healthy)
|
||||
traefik traefik:v3.7.13 Up 4 days
|
||||
--- images now
|
||||
alpine:latest
|
||||
curlimages/curl:8.11.1
|
||||
ghcr.io/paperless-ngx/paperless-ngx:2.20.15
|
||||
ghcr.io/puppeteer/puppeteer:latest
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.314.0-rc.1
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.326.0-rc.1
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.326.0-rc.2
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.327.0-rc.2
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.327.0-rc.3
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.327.0-rc.4
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.327.1-rc.1
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.331.0
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.331.0-rc.1
|
||||
gitea.dooplex.hu/admin/felhom-samba:1.2.0
|
||||
gtstef/filebrowser:1.3.3-stable
|
||||
gtstef/filebrowser:1.5.6-stable
|
||||
postgres:18-alpine
|
||||
redis:7-alpine
|
||||
redis:7.4-alpine
|
||||
traefik:v3.6.7
|
||||
traefik:v3.7.13
|
||||
offbox target set: False
|
||||
per-app offbox: None
|
||||
restore holds: ['chaoscrash', 'chaosoom', 'chaosoomb', 'gokapi']
|
||||
[ERROR] lines since start: 0
|
||||
/mnt/felhom-drives/scratch_hdd/backups/offsite-restore/:
|
||||
total 8
|
||||
drwxr-xr-x 2 root root 4096 Oct 20 08:43 .
|
||||
drwxr-xr-x 5 root root 4096 Oct 19 18:52 ..
|
||||
|
||||
/mnt/sys_drive/felhom-data/backups/:
|
||||
total 16
|
||||
drwxr-xr-x 4 root root 4096 Oct 20 08:44 .
|
||||
drwxr-xr-x 4 root root 4096 Sep 13 20:26 ..
|
||||
drwxr-xr-x 3 root root 4096 Oct 20 08:43 primary
|
||||
drwxr-xr-x 3 root root 4096 Oct 12 20:51 secondary
|
||||
IMAGE LIST IDENTICAL TO BASELINE
|
||||
=== host layer (demo-hp)
|
||||
0
|
||||
DooPlex scratch: throwaway key shredded
|
||||
=== hub layer: nothing — 9202 has no hub
|
||||
@@ -0,0 +1,161 @@
|
||||
# P2 live: R-893 option A, proved on scratch guest 9202 (2026-10-20, by day)
|
||||
|
||||
Night brief 2026-10-20 part 2, Part 2 (operator answer „D20 A"). Controller **0.336.0-rc.1** (commit `93fb251`,
|
||||
image id `sha256:0756d4d2…`) on Tier 0 only: demo-hp, scratch guest 9202. Code read live:
|
||||
`felhom-controller/controller/internal/backup/offbox_restore_undo.go` and `offbox_reconstitute.go` at `93fb251`.
|
||||
Architecture: `07-backup-architecture.md` §6.3 (R-893); `09` §3 decision 192 and D20 = A.
|
||||
The method repeats `night-2026-10-19/P3/` (the measurement of the old behaviour) step for step, so the two sets of
|
||||
numbers can be compared.
|
||||
|
||||
## Method (endpoint level, the household's buttons; no browser)
|
||||
|
||||
- 9202 was on **0.331.0**. For the run it went to 0.336.0-rc.1: pull, write `/etc/felhom-controller-image`,
|
||||
restart `felhom-controller-bootstrap.service`. Healthy, 0 `[ERROR]` (`01-upgrade.txt`). It was put back after the run.
|
||||
- **Scratch off-site target:** `atmoz/sftp:alpine` container `r893b-sftp` on 9202 itself (port 2222). The repo is on
|
||||
9202's own disk at `/var/lib/felhom/r893b-sftp` (nvme-scratch, not `pve/data`). The controller reached it at
|
||||
`172.17.0.1:2222`, which does not depend on DHCP. Set up through `POST /backup/offbox/config` with a throwaway
|
||||
ed25519 key and the server's own host key pinned. 9202 has no agent, so the escrow was confirmed with the legacy
|
||||
manual `POST /backup/offbox/confirm-escrow`. Only `rallly` was switched on (`POST /backup/offbox/toggle`).
|
||||
- **Throwaway app:** Rallly 4.15.3 + postgres:18-alpine. Its DB is in the named volume `rallly_rallly_postgres_data`.
|
||||
Deployed through the product (`box_walk.deploy`).
|
||||
- **Observables, all positive** (`tool-obs.sh.txt`):
|
||||
- the `r893_marker` rows;
|
||||
- tables that exist only at certain times;
|
||||
- the public base-table count;
|
||||
- an md5 of `pg_dump --schema-only`, with pg 18's random `\restrict` lines removed and **a second dump as its
|
||||
control** (equal every time);
|
||||
- an md5 of the per-table row counts;
|
||||
- an md5 of `pg_dump --data-only`;
|
||||
- the definition's image lines;
|
||||
- the `backups/restore-undo/rallly` folder.
|
||||
- **Forced error:** the same as last night. One line, `SELECT r893_forced_replay_failure();`, goes into the PREPARED
|
||||
scratch copy of the snapshot's dump, just before its completion marker. The completeness gate still passes, and the
|
||||
replay fails **after the volume was replaced**.
|
||||
- **Restore:** `POST /backup/offbox/reconstitute` with `app=rallly confirm=1`. The outcome was read from
|
||||
`/api/backup/restore-status`, the source the restore page polls.
|
||||
- **Watch on the undo folder:** while each restore ran, a watcher on 9202 listed
|
||||
`/mnt/sys_drive/felhom-data/backups/restore-undo` every 0.3 s. This shows the copy from a different channel than
|
||||
the controller's log.
|
||||
|
||||
## Timeline (UTC)
|
||||
|
||||
| Time | Act | File |
|
||||
|---|---|---|
|
||||
| 07:25:33 | baseline: 0.331.0, no off-site target, `settings.json` and the `offbox/` folder copied | `00-baseline.txt` |
|
||||
| 07:25:47 | 9202 on 0.336.0-rc.1, healthy | `01-upgrade.txt` |
|
||||
| 07:27:37–07:27:58 | SFTP target up; off-site configured, escrow confirmed, `rallly` switched on | `02-sftp-target.txt` |
|
||||
| 07:28:03–07:28:53 | Rallly deployed | `03-deploy.txt` |
|
||||
| 07:29:13 | marker **A** (34 tables) | `04-marker-A.txt` |
|
||||
| 07:29:29–07:30:38 | off-site run → snapshot 1 | `05-snapshot-1.txt` |
|
||||
| 07:30:50 | marker **B** + `r893_after`; `r893_only_old` + marker **C** (36 tables) | `05-snapshot-1.txt` |
|
||||
| 07:30:56–07:32:04 | off-site run → **snapshot 2** (`c3d94d03`) | `06-snapshot-2.txt` |
|
||||
| 07:32:38 | `r893_only_old` DROPPED, marker **D**, `r893_new2`: the LIVE record | `07-live-before-restore.txt` |
|
||||
| 07:32:57–07:33:13 | full restore of snapshot 2 prepared (66.1 MB; content checked: C present, D absent); the failing line injected | `08-prepare-and-inject.txt` |
|
||||
| 07:33:28 | reconstitute (forced failure) | `10-forced-failure-http.txt`, `11-controller-log-window*.txt` |
|
||||
| 07:33:29 | undo dump of the live DB; app stopped 07:33:29–07:33:40 | |
|
||||
| 07:33:40.9–41.6 | **live volume copied** to `restore-undo/rallly/20261020T073340Z/rallly_rallly_postgres_data.tar` (69 198 848 B), tmp then rename | `09-undo-folder-watch.txt` |
|
||||
| 07:33:42 | snapshot's volume in place (`Restored 1 Docker volume(s)`) | |
|
||||
| 07:33:44 | replay FAILED (`function r893_forced_replay_failure() does not exist`, exit 3); rows rolled back 07:33:45 | |
|
||||
| 07:33:46–47 | app stopped; **the copy poured back** (`Restored 1 Docker volume(s)`, second time); „the live volumes and definition are put back; starting the app (R-893)" | |
|
||||
| 07:33:53 | `Stack rallly started successfully` | |
|
||||
| 07:33:54 | restore op ends ok=false; the copy folder **gone** (watcher: absent at 07:33:54.45) | |
|
||||
| 07:34:25 | after state read: identical to 07:32:38 | `12-after-failed-restore.txt` |
|
||||
| 07:36:09–07:36:15 | full restore prepared again, with no injection (injected-line count 0) | `13-normal-prepare.txt` |
|
||||
| 07:36:28–07:36:57 | normal reconstitute: copy made 07:36:41–42, volume + dump replayed, app restarted, copy removed 07:36:57.9 | `14-…`, `15-…`, `16-controller-log-normal*.txt` |
|
||||
| 07:38:31 | after state read = snapshot 2 | `17-after-normal-restore.txt` |
|
||||
| (stop) | the session was paused by an API rate limit. When it resumed, 9202 was re-read first: nothing was in progress, no restore-undo copy, step 4 already recorded | |
|
||||
| 08:43–08:45 | teardown, three layers | `18-teardown.txt` |
|
||||
|
||||
## Step 3: the forced failure, before and after
|
||||
|
||||
| | LIVE before the restore (07:32:38) | After the failed restore + put-back (07:34:25) | 2026-10-19 on 0.335.0 (old behaviour) |
|
||||
|---|---|---|---|
|
||||
| marker rows | A, B, C, D | **A, B, C, D** | A, B, C, D |
|
||||
| `r893_only_old` (in snapshot 2 only) | absent | **absent** | present, 1 row |
|
||||
| `r893_new2` (written after snapshot 2) | present, 1 row | **present, 1 row** | present, 1 row |
|
||||
| public base tables | 36 | **36** | 37 |
|
||||
| schema-only md5 | `05828c77…` | **`05828c77…`** (control equal) | — |
|
||||
| per-table row-count md5 | `d8bcbc4b…` | **`d8bcbc4b…`** | — |
|
||||
| data-only md5 | `e5c1a72f…` | **`e5c1a72f…`** | — |
|
||||
| definition | rallly 4.15.3, postgres 18 (digest pins) | same | same |
|
||||
| app | running | **running, healthy**; controller `state=running`; front door 200 | stopped, held (`restore_mixed`) |
|
||||
| restore hold for rallly | none | **none** (holds: chaoscrash, chaosoom, chaosoomb, gokapi, as at baseline) | `restore_mixed` |
|
||||
| `backups/restore-undo/rallly` | absent | **absent** (it existed 07:33:40–07:33:54) | (did not exist) |
|
||||
|
||||
The DB volume's `CreatedAt` moved from 07:28:21 to 07:33:46. That is expected: the put-back recreates the volume
|
||||
from the copy. Every content observable is equal, the data-only dump included.
|
||||
|
||||
## Step 4: the normal restore of snapshot 2
|
||||
|
||||
| | Recorded just before snapshot 2 (07:30:50) | After the normal restore (07:38:31) |
|
||||
|---|---|---|
|
||||
| marker rows | A, B, C | **A, B, C** |
|
||||
| `r893_only_old` | present, 1 row | **present, 1 row** |
|
||||
| `r893_new2` | absent | **absent** |
|
||||
| public base tables | 36 | **36** |
|
||||
| schema-only md5 | `f872bd54…` | **`f872bd54…`** |
|
||||
| per-table row-count md5 | `c032377e…` | **`c032377e…`** |
|
||||
| data-only md5 | `db5903ab…` | `1e688d1b…` (differs; see below) |
|
||||
| app | running | **running, healthy**; front door 200; no hold |
|
||||
| restore-undo folder | — | **absent** (existed 07:36:41–07:36:57) |
|
||||
|
||||
- **Table count.** Snapshot 2 has **36** base tables, not 37. It held `r893_only_old` but not `r893_new2`. The
|
||||
„37" in the brief is last night's MIXED state (36 live tables + the stray `r893_only_old`), not the snapshot's.
|
||||
- **The data-only md5.** It differs from the read taken 6 s before the snapshot's own dump started, while every row
|
||||
count and the schema are equal. Not explained here. It may be a row value that Rallly touched between that read
|
||||
and the dump (the off-site run restarted the app), or row order after the reload. It was not part of the
|
||||
acceptance list, and no row count moved. In step 3 the same observable was byte-equal, because there the volume
|
||||
itself was put back.
|
||||
|
||||
## The sentences the household sees
|
||||
|
||||
- **Failed restore, put back** (`err.backup.restore_failed_put_back`, from `/api/backup/restore-status`): „A teljes
|
||||
visszaállítás sikertelen: a(z) rallly visszaállítása sikertelen. Nem változott semmi: az alkalmazás adatai
|
||||
visszakerültek a visszaállítás előtti állapotba, és az alkalmazás újra fut". Measured **true**: the data is
|
||||
identical and the app runs. It does **not** contain „vedd fel velünk a kapcsolatot".
|
||||
- **Normal restore:** „A(z) rallly: 0 fájl és 1 adatkötet és az adatbázis visszaállítva (mentés: 2026. 10. 20.
|
||||
09:30) — az alkalmazás újraindult." True. 09:30 is local time (CEST) for snapshot 2's dump at 07:30 UTC.
|
||||
- Flash keys seen on the way: `flash.offbox.target_saved_escrow_agent_down` (no agent on 9202, expected),
|
||||
`escrow_confirmed`, `run_started`, `restore_started`, `full_restore_started`.
|
||||
|
||||
## Not measured (said plainly)
|
||||
|
||||
- **The version-change branch** (the snapshot's older definition written, then the live one written back). It needs
|
||||
an app at an older catalog step, the same limit as last night. Pinned by `r893_putback_test.go`.
|
||||
- **The `_files` sentence** (`restore_failed_put_back_files`). Rallly has no drive files, so 0 files were placed.
|
||||
- **The refusals before the stop:** the fit check refusing (`restore_undo_no_space`), an unmeasurable volume
|
||||
(`restore_undo_unmeasured`) and a failed copy (`restore_undo_copy_failed`). Only the positive path of the fit
|
||||
check ran: it passed and the app was stopped after it.
|
||||
- **A put-back that itself fails** (hold + kept copy).
|
||||
- **The operator notification** for a put-back. 9202 has no hub, so it was not observable.
|
||||
|
||||
## Side observations
|
||||
|
||||
- Each manual off-site run restarted Rallly (its dump leg), as last night.
|
||||
- The controller logged one expected `[ERROR]` pair per failure (the import and the rollback line) and one
|
||||
`[ERROR] [web] off-box reconstitute … (async)` line that carries the household sentence. The sentence is good
|
||||
news, so ERROR is a loud level for a put-back that worked. This is noted for the parent, not changed.
|
||||
- The empty folder `backups/restore-undo/` is left behind after the per-app folder is removed (the code removes the
|
||||
per-app folder only). This drill removed it by name.
|
||||
|
||||
## Teardown (three layers)
|
||||
|
||||
- **Machine (9202):**
|
||||
- Rallly was removed through the product (`POST /api/stacks/rallly/stop`, then `/remove` with `remove_hdd_data` +
|
||||
`remove_backups`). Result: `volumes_removed: rallly_rallly_postgres_data`; 0 containers, 0 volumes; no
|
||||
`backups/primary/rallly`, no `offsite-restore/rallly`. The template folder `/opt/docker/stacks/rallly`
|
||||
(`.felhom.yml`, `docker-compose.yml`, `hold-logs`) was there before and stays.
|
||||
- `r893b-sftp` was removed. So were the images `atmoz/sftp:alpine` and `lukevella/rallly:4.15.3` (both absent at
|
||||
baseline), the repo folder, the key and the empty `backups/restore-undo/`.
|
||||
- `settings.json` was put back from the 07:25 copy (sha256 `cb6b18f8…` = baseline) and `offbox/` from its tar,
|
||||
with the controller stopped.
|
||||
- The controller is back on **0.331.0**: healthy, 0 `[ERROR]`. Off-site target unset, no per-app off-site switch,
|
||||
restore holds as at baseline.
|
||||
- The 0.336.0-rc.1 image was removed by name. The image list is **identical to the baseline**. All
|
||||
`/root/r893b-*` and `/tmp/r893b-*` files are gone.
|
||||
- Not put back: `restore-status.json`, the controller's record of the last op. It now names this drill's last
|
||||
restore, as it named last night's at baseline.
|
||||
- **Host (demo-hp):** only the per-call `/tmp/r893b-g-*` script copies, each removed after its call; 0 left. Nothing
|
||||
on `pve/data`. No whole-guest backup taken.
|
||||
- **Hub:** nothing. 9202 has no hub.
|
||||
- **DooPlex:** the throwaway key was shredded from the session scratchpad.
|
||||
@@ -0,0 +1,22 @@
|
||||
#!/bin/bash
|
||||
# r893b-obs.sh LABEL — positive observables of the Rallly database (read-only)
|
||||
P="docker exec rallly-postgres psql -U rallly -d rallly -Atq"
|
||||
echo "=== $1 $(date -u +%T) UTC"
|
||||
docker ps -a --filter label=com.docker.compose.project=rallly --format '{{.Names}} {{.Image}} {{.Status}}'
|
||||
echo "markers (phase=rows): $($P -c "select string_agg(phase||'='||n, ', ' order by phase) from (select phase, count(*) n from r893_marker group by phase) x" 2>&1)"
|
||||
for t in r893_after r893_only_old r893_new2; do
|
||||
ex=$($P -c "select count(*) from information_schema.tables where table_schema='public' and table_name='$t'")
|
||||
if [ "$ex" = 1 ]; then echo "$t exists: 1 rows: $($P -c "select count(*) from $t")"; else echo "$t exists: 0"; fi
|
||||
done
|
||||
echo "public tables: $($P -c "select count(*) from information_schema.tables where table_schema='public' and table_type='BASE TABLE'")"
|
||||
echo "prisma migrations: $($P -c 'select count(*) from _prisma_migrations' 2>&1)"
|
||||
echo "db volume created: $(docker volume inspect -f '{{.CreatedAt}}' rallly_rallly_postgres_data 2>&1)"
|
||||
S1=$(docker exec rallly-postgres pg_dump -U rallly -d rallly --schema-only | grep -Ev '^\\(un)?restrict ' | md5sum | cut -c1-32)
|
||||
S2=$(docker exec rallly-postgres pg_dump -U rallly -d rallly --schema-only | grep -Ev '^\\(un)?restrict ' | md5sum | cut -c1-32)
|
||||
echo "schema-only md5: $S1 (control, second dump: $S2)"
|
||||
echo "data-only md5: $(docker exec rallly-postgres pg_dump -U rallly -d rallly --data-only | grep -Ev '^\\(un)?restrict ' | md5sum | cut -c1-32)"
|
||||
RC=$(for t in $($P -c "select table_name from information_schema.tables where table_schema='public' and table_type='BASE TABLE' order by table_name"); do echo "$t=$($P -c "select count(*) from \"$t\"")"; done)
|
||||
echo "per-table row counts md5: $(echo "$RC" | md5sum | cut -c1-32)"
|
||||
echo "per-table row counts: $(echo "$RC" | tr '\n' ' ')"
|
||||
echo "definition: $(grep -E '^\s*image:' /opt/docker/stacks/rallly/docker-compose.yml | tr -s ' ' | tr '\n' ' ')"
|
||||
echo "restore-undo folder: $(ls -A /mnt/sys_drive/felhom-data/backups/restore-undo/rallly 2>&1 | tr '\n' ' ')"
|
||||
@@ -0,0 +1,10 @@
|
||||
put-back skipped RED (good) r893_putback_test.go:135: the live volume was poured back as "" — want the exact pre-restore copy
|
||||
volume-replay branch without undo RED (good) r893_putback_test.go:170: the replay removes a volume before it recreates it; the copy must be poured back even at a count of 0, got ""
|
||||
fit check without the copy RED (good) r893_putback_test.go:187: the refusal must come before the stop — StopStack ran 2 time(s)
|
||||
probe error ignored RED (good) r893_putback_test.go:213: StopStack ran 2 time(s) on an unmeasured copy
|
||||
no drop on exit RED (good) r893_putback_test.go:232: the copy must be removed after a restore that worked, left [/tmp/TestR893A_ASuccessfulRestoreRemovesTheCopy3779926857/001/backups/restore-undo/immich/20261020T065144Z/immich_pgdata.tar]
|
||||
copy not kept on a failed put-back RED (good) r893_putback_test.go:259: the copy must stay on disk for support, found []
|
||||
copy error ignored RED (good) r893_putback_test.go:276: nothing may move after a failed copy: replays 1, rollbacks 1, definition writes []
|
||||
slice-1 hold after the rollback RED (good) r893_mixed_restore_test.go:170: option A: the app must be started again after the put-back — calls [stop recreate startsvc:app-db recreate stop]
|
||||
DB-start branch only restarts RED (good) r893_mixed_restore_test.go:202: the live definition must be written back BEFORE the start — calls [stop recreate startsvc:app-db start]
|
||||
restored; failures-to-go-red: 0
|
||||
@@ -0,0 +1,50 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Red-proof the R-893 option A tests: apply one mutation, run its test, expect FAIL, restore the file."""
|
||||
import subprocess, sys
|
||||
|
||||
ROOT = "/mnt/5_hdd/felhom.eu/git/felhom-controller/controller"
|
||||
U = ROOT + "/internal/backup/offbox_restore_undo.go"
|
||||
R = ROOT + "/internal/backup/offbox_reconstitute.go"
|
||||
|
||||
CASES = [
|
||||
("put-back skipped", U, " if volumesMoved {\n if err := m.putBack(stack, c); err != nil {",
|
||||
" if false {\n if err := m.putBack(stack, c); err != nil {", "TestR893A_AFailedReplayPutsTheLiveVolumesBackExactly"),
|
||||
("volume-replay branch without undo", R, "if definitionMoved || nVols > 0 || undo != nil {", "if definitionMoved || nVols > 0 {",
|
||||
"TestR893A_AFailedVolumeReplayStillPutsBackWhenItCountedNothing"),
|
||||
("fit check without the copy", R, "if free < need+volNeed {", "if free < need {", "TestR893A_TheCopyMustFitBeforeAnythingStops"),
|
||||
("probe error ignored", U, " if pErr != nil {\n return 0, nil, pErr\n }", " if pErr != nil {\n continue\n }",
|
||||
"TestR893A_AnUnmeasurableVolumeRefusesBeforeAnythingStops"),
|
||||
("no drop on exit", R, " defer undo.drop(m, stack)\n", "", "TestR893A_ASuccessfulRestoreRemovesTheCopy"),
|
||||
("copy not kept on a failed put-back", U, " c.keep = true\n", "", "TestR893A_AFailedPutBackHoldsTheAppAndKeepsTheCopy"),
|
||||
("copy error ignored", R, " if cErr != nil {\n m.logger.Printf(\"[ERROR] [offbox] %s: the copy of the live volumes failed",
|
||||
" if cErr != nil && false {\n m.logger.Printf(\"[ERROR] [offbox] %s: the copy of the live volumes failed",
|
||||
"TestR893A_AFailedCopyRestoresNothingAndStartsTheApp"),
|
||||
("slice-1 hold after the rollback", R,
|
||||
" return res, m.putBackAfterFailedRestore(stack, iErr, liveDef, undo, res.VolumesReplayed > 0, res.FilesPlaced, restartStack, \"err.backup.db_restore_failed_held_mixed\")",
|
||||
" m.holdAppAfterMixedRestore(stack, iErr, liveDef)\n return res, util.MsgError(\"err.backup.db_restore_failed_held_mixed\", stack)",
|
||||
"TestR893_AVersionChangeThenAFailedReplayStartsTheAppAtItsLiveDefinition"),
|
||||
("DB-start branch only restarts", R,
|
||||
" return res, m.putBackAfterFailedRestore(stack, err, liveDef, undo, res.VolumesReplayed > 0, res.FilesPlaced, restartStack, \"err.backup.restore_failed_held_mixed\")",
|
||||
" _ = restartStack()\n return res, err",
|
||||
"TestR893_AVersionChangeThenAFailedDBStartStartsTheAppAtItsLiveDefinition"),
|
||||
]
|
||||
|
||||
bad = 0
|
||||
for name, path, old, new, test in CASES:
|
||||
src = open(path).read()
|
||||
if src.count(old) != 1:
|
||||
print("SETUP-ERROR", name, "match count", src.count(old)); bad += 1; continue
|
||||
open(path, "w").write(src.replace(old, new))
|
||||
try:
|
||||
p = subprocess.run(["go", "test", "./internal/backup/", "-run", "^" + test + "$", "-count=1"], cwd=ROOT,
|
||||
capture_output=True, text=True)
|
||||
out = p.stdout + p.stderr
|
||||
line = next((l.strip() for l in out.splitlines() if "_test.go:" in l), out.strip().splitlines()[-1] if out.strip() else "")
|
||||
verdict = "RED (good)" if p.returncode != 0 else "STILL GREEN (BAD)"
|
||||
if p.returncode == 0:
|
||||
bad += 1
|
||||
print("%-38s %-18s %s" % (name, verdict, line[:220]))
|
||||
finally:
|
||||
open(path, "w").write(src)
|
||||
print("restored; failures-to-go-red:", bad)
|
||||
sys.exit(1 if bad else 0)
|
||||
@@ -0,0 +1,94 @@
|
||||
# R-1019 — the PRODUCTION classifier (controller main 93fb251) over the templates; hddPath=/mnt/drive
|
||||
# BEFORE (catalog HEAD 5e37aa6):
|
||||
=== RUN TestZZ_R1019_Probe
|
||||
== qbittorrent hasClassification=true
|
||||
bind root=userdata rel="media/torrents" ro=false class="optional" origin=explicit
|
||||
secondary copies 1 path(s) [/mnt/drive/userdata/media/torrents]
|
||||
offsite copies 0 path(s) []
|
||||
== transmission hasClassification=true
|
||||
bind root=userdata rel="media/transmission" ro=false class="optional" origin=explicit
|
||||
bind root=userdata rel="media/transmission/watch" ro=false class="optional" origin=explicit
|
||||
secondary copies 1 path(s) [/mnt/drive/userdata/media/transmission]
|
||||
offsite copies 0 path(s) []
|
||||
== nzbget hasClassification=true
|
||||
bind root=userdata rel="media/nzbget" ro=false class="optional" origin=explicit
|
||||
secondary copies 1 path(s) [/mnt/drive/userdata/media/nzbget]
|
||||
offsite copies 0 path(s) []
|
||||
== sabnzbd hasClassification=true
|
||||
bind root=userdata rel="media/usenet" ro=false class="optional" origin=explicit
|
||||
secondary copies 1 path(s) [/mnt/drive/userdata/media/usenet]
|
||||
offsite copies 0 path(s) []
|
||||
== metube hasClassification=true
|
||||
bind root=userdata rel="media/metube" ro=false class="optional" origin=explicit
|
||||
secondary copies 1 path(s) [/mnt/drive/userdata/media/metube]
|
||||
offsite copies 0 path(s) []
|
||||
== sonarr hasClassification=true
|
||||
bind root=userdata rel="media" ro=false class="excluded" origin=explicit
|
||||
secondary copies 0 path(s) []
|
||||
offsite copies 0 path(s) []
|
||||
== radarr hasClassification=true
|
||||
bind root=userdata rel="media" ro=false class="excluded" origin=explicit
|
||||
secondary copies 0 path(s) []
|
||||
offsite copies 0 path(s) []
|
||||
== prowlarr hasClassification=true
|
||||
secondary copies 0 path(s) []
|
||||
offsite copies 0 path(s) []
|
||||
== jellyfin hasClassification=true
|
||||
bind root=userdata rel="media" ro=true class="excluded" origin=explicit
|
||||
secondary copies 0 path(s) []
|
||||
offsite copies 0 path(s) []
|
||||
== plex hasClassification=true
|
||||
bind root=userdata rel="media" ro=true class="excluded" origin=explicit
|
||||
secondary copies 0 path(s) []
|
||||
offsite copies 0 path(s) []
|
||||
== emby hasClassification=true
|
||||
bind root=userdata rel="media" ro=true class="excluded" origin=explicit
|
||||
secondary copies 0 path(s) []
|
||||
offsite copies 0 path(s) []
|
||||
|
||||
# AFTER (the working tree with the five download folders excluded):
|
||||
=== RUN TestZZ_R1019_Probe
|
||||
== qbittorrent hasClassification=true
|
||||
bind root=userdata rel="media/torrents" ro=false class="excluded" origin=explicit
|
||||
secondary copies 0 path(s) []
|
||||
offsite copies 0 path(s) []
|
||||
== transmission hasClassification=true
|
||||
bind root=userdata rel="media/transmission" ro=false class="excluded" origin=explicit
|
||||
bind root=userdata rel="media/transmission/watch" ro=false class="excluded" origin=explicit
|
||||
secondary copies 0 path(s) []
|
||||
offsite copies 0 path(s) []
|
||||
== nzbget hasClassification=true
|
||||
bind root=userdata rel="media/nzbget" ro=false class="excluded" origin=explicit
|
||||
secondary copies 0 path(s) []
|
||||
offsite copies 0 path(s) []
|
||||
== sabnzbd hasClassification=true
|
||||
bind root=userdata rel="media/usenet" ro=false class="excluded" origin=explicit
|
||||
secondary copies 0 path(s) []
|
||||
offsite copies 0 path(s) []
|
||||
== metube hasClassification=true
|
||||
bind root=userdata rel="media/metube" ro=false class="excluded" origin=explicit
|
||||
secondary copies 0 path(s) []
|
||||
offsite copies 0 path(s) []
|
||||
== sonarr hasClassification=true
|
||||
bind root=userdata rel="media" ro=false class="excluded" origin=explicit
|
||||
secondary copies 0 path(s) []
|
||||
offsite copies 0 path(s) []
|
||||
== radarr hasClassification=true
|
||||
bind root=userdata rel="media" ro=false class="excluded" origin=explicit
|
||||
secondary copies 0 path(s) []
|
||||
offsite copies 0 path(s) []
|
||||
== prowlarr hasClassification=true
|
||||
secondary copies 0 path(s) []
|
||||
offsite copies 0 path(s) []
|
||||
== jellyfin hasClassification=true
|
||||
bind root=userdata rel="media" ro=true class="excluded" origin=explicit
|
||||
secondary copies 0 path(s) []
|
||||
offsite copies 0 path(s) []
|
||||
== plex hasClassification=true
|
||||
bind root=userdata rel="media" ro=true class="excluded" origin=explicit
|
||||
secondary copies 0 path(s) []
|
||||
offsite copies 0 path(s) []
|
||||
== emby hasClassification=true
|
||||
bind root=userdata rel="media" ro=true class="excluded" origin=explicit
|
||||
secondary copies 0 path(s) []
|
||||
offsite copies 0 path(s) []
|
||||
@@ -0,0 +1,45 @@
|
||||
# R-49 — Immich's off-site copy: what it holds, and two ways to shrink it (2026-10-20)
|
||||
|
||||
Status: **measure and plan only — nothing changed.** Night brief part 2 (run by day), Part 4. Architecture: `07-backup-architecture.md`
|
||||
(the off-site tier = the recovery unit + the app's mandatory data). Measured read-only on scratch guest 9202 (demo-hp), Immich
|
||||
v3.2.4 + `ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0`, data on the drive `/mnt/sizing-bench`, 09:0x UTC.
|
||||
|
||||
## What today's off-site copy holds
|
||||
|
||||
The off-site tier copies the recovery unit (`backups/primary/immich/`, 1 160 553 825 B) and the mandatory drive folder
|
||||
`appdata/immich/` (166 077 159 B) — about **1.33 GB** in all. The external library `media/photos` is `optional` (not off-site;
|
||||
0 B here anyway).
|
||||
|
||||
| Part | Where | Bytes | Share | Can it come back without the copy? |
|
||||
|---|---|---:|---:|---|
|
||||
| ML model cache | unit `volume-dumps/immich_immich_ml_cache.tar` (volume 823 835 298 B) | 823 764 480 | 62 % | **Yes** — Immich downloads the models again when a smart-search or face job first needs them |
|
||||
| Raw database folder | unit `volume-dumps/immich_immich_postgres_data.tar` (volume 282 169 461 B) | 283 049 472 | 21 % | **Yes, if** a restore from the logical dump alone is proven (below) |
|
||||
| Immich's own database backups | `appdata/immich/backups/` | 115 367 547 | 9 % | Yes — a second copy of what the logical dump already holds; it grows every night |
|
||||
| Logical database dump | unit `db-dumps/immich-postgres.sql` | 53 711 763 | 4 % | **No** — this IS the household's albums, people, metadata |
|
||||
| Originals | `appdata/immich/upload/` | 48 673 663 | 4 % | **No** — the photos themselves |
|
||||
| Thumbnails | `appdata/immich/thumbs/` | 2 035 910 | <1 % | Yes — Immich regenerates them |
|
||||
| Redis | volume `immich_immich_redis_data` (31 594 479 B) | — | — | not in the unit (a queue cache) |
|
||||
|
||||
**What must survive: 102 MB of 1.33 GB (8 %).** The 2026-07-19 measurement (R-49) said the same shape: ~60 % model cache.
|
||||
|
||||
## Two options
|
||||
|
||||
| | A — leave out all three re-fetchable parts | B — leave out only the model cache |
|
||||
|---|---|---|
|
||||
| Off-site size here | ~105 MB (dump + originals + thumbnails + unit metadata) — **−92 %** | ~510 MB — **−62 %** |
|
||||
| What a restore then needs | the models download again (~824 MB, from Immich's model host) before smart search and face recognition work again — the photos, albums and timeline work at once; the database is rebuilt from the logical dump alone into an EMPTY volume (no raw folder to lay under it) | the models download again, as in A; the database comes back as today (volume, then dump) |
|
||||
| The proof it needs | (1) one off-site restore of Immich onto a box with an empty model cache: photos and albums read back, then a smart search answers after the download (time measured); (2) **one off-site restore from the dump alone** into an empty database volume, on this vectorchord image: the extensions and vector indexes come back, the row counts match, Immich starts and searches — today's restore always lays the raw folder first, so this has never been run | only proof (1) |
|
||||
| Risk | the dump-only database restore is the unmeasured part; if the vector extension needs more than the dump gives, a restore would fail where it works today (option A's put-back keeps the app as it was, but the backup would be weaker) | none new: the cache is a cache |
|
||||
| Also | Immich's own `backups/` folder needs a catalog class (`excluded`) — a template change in the `hdd` block | a catalog change: the model-cache volume marked as not captured |
|
||||
|
||||
Not measured today: the model download's time and size on a real line (the 824 MB is the cache as it stands, which holds every
|
||||
model this box used); the off-site bytes after restic's compression and de-duplication (the table is the plain size).
|
||||
|
||||
## Pick: B now, then A's other two parts after proof (2)
|
||||
|
||||
B removes 62 % of the copy with no new risk — the strongest case R-49 named first — and needs only proof (1). The raw database
|
||||
folder and Immich's own backups are another 30 %, but leaving out the raw folder changes HOW the database comes back, and that
|
||||
path has never run for this extension image; it should be a separate step with its own proof, not ride along.
|
||||
|
||||
**If the operator does nothing:** every Immich household keeps sending ~12× its real data off-site each night, against its
|
||||
quota and transfer.
|
||||
@@ -0,0 +1,36 @@
|
||||
# Part 5 — R-462: the update test for the installed apps (read 2026-10-20T08:45:32Z, catalog c50adfc)
|
||||
|
||||
## installed apps on the test boxes (hub, latest controller report per customer)
|
||||
bentopdf 2, opengist 2, privatebin 2, bookstack 2, paperless-ngx 2; 1 each: prowlarr radarr qbittorrent sonarr home-assistant romm docmost plex calibre-web adventurelog kimai
|
||||
|
||||
## fixture coverage (upgrade_boxport.available() + upgrade_fixtures.FIXTURES)
|
||||
box-ported fixtures: 58 ; bench-only fixtures: 0
|
||||
installed apps with a fixture: ['opengist', 'privatebin', 'bookstack', 'paperless-ngx', 'prowlarr', 'radarr', 'qbittorrent', 'sonarr', 'home-assistant', 'romm', 'docmost', 'plex', 'calibre-web', 'adventurelog', 'kimai']
|
||||
installed apps without one: ['bentopdf'] (bentopdf keeps no data: nothing to seed)
|
||||
|
||||
## upstream currency (scripts/check-currency.py, registries read)
|
||||
opengist opengist 1.15 same-shape: major 1.15, all 1.15 — SHAPE-SWITCH? 1.15.2
|
||||
privatebin privatebin 2.0.6 same-shape: major 2.1.0, all 2.1.0 — behind
|
||||
bookstack bookstack 26.09.1 same-shape: major 26.09.1, all 26.09.1 — current
|
||||
bookstack bookstack-db 12.3 same-shape: major 12.3, all 13.0 — behind
|
||||
paperless-ngx paperless-webserver 2.20.15 same-shape: major 2.20.15, all 3.3.3 — behind
|
||||
paperless-ngx paperless-postgres 18-alpine same-shape: major 18-alpine, all 18-alpine — SHAPE-SWITCH? 18.6
|
||||
paperless-ngx paperless-redis 7-alpine same-shape: major 7-alpine, all 8-alpine — behind
|
||||
prowlarr prowlarr version-2.6.5.5623 same-shape: major version-2.6.5.5623, all version-2.6.5.5623 — current
|
||||
radarr radarr 6.4.4 same-shape: major 6.4.4, all 6.4.4 — current
|
||||
qbittorrent qbittorrent 5.2.4-1 same-shape: major 5.2.4-1, all 5.2.4-1 — current
|
||||
sonarr sonarr 4.0.20 same-shape: major 4.0.20, all 4.0.20 — SHAPE-SWITCH? 5.14
|
||||
home-assistant home-assistant 2026.9.4 same-shape: major 2026.10.2, all 2026.10.2 — behind
|
||||
romm romm 5.3.1 same-shape: major 5.4.0, all 5.4.0 — behind
|
||||
romm romm-db 11.8 same-shape: major 11.8, all 13.0 — behind
|
||||
romm romm-redis 7-alpine same-shape: major 7-alpine, all 8-alpine — behind
|
||||
docmost docmost 0.96.0 same-shape: major 0.96.0, all 0.96.0 — current
|
||||
docmost docmost-postgres 18-alpine same-shape: major 18-alpine, all 18-alpine — SHAPE-SWITCH? 18.6
|
||||
docmost docmost-redis 7-alpine same-shape: major 7-alpine, all 8-alpine — behind
|
||||
plex plex 1.41.4.9463-630c9f557 same-shape: major 1.43.4.10903-e5521bd8c, all 1.43.4.10903-e5521bd8c — behind
|
||||
calibre-web calibre-web v4.0.8 same-shape: major v4.0.8, all v4.0.8 — current
|
||||
adventurelog adventurelog v0.13.0 same-shape: major v0.13.0, all v0.13.0 — current
|
||||
adventurelog adventurelog-postgres 16-3.5-alpine same-shape: major 16-3.6-alpine, all 18-3.6-alpine — behind
|
||||
adventurelog adventurelog-frontend v0.13.0 same-shape: major v0.13.0, all v0.13.0 — current
|
||||
kimai kimai 2.67.0 same-shape: major 2.70.0, all 2.70.0 — behind
|
||||
kimai kimai-db 11.8 same-shape: major 11.8, all 13.0 — behind
|
||||
@@ -0,0 +1,4 @@
|
||||
# per-box floors 0.336.0 / MinAgent 0.158.1 set 08:51:06Z; image sha256:93cd67d871125877e935a024765075b5a0b7c0e7683e1165976ff7a654fe8d73 (clean clone 93fb251; rc.1 on 9202 first)
|
||||
demo-felhom|0.336.0|2026-10-20 08:54:55
|
||||
demo-hp|0.336.0|2026-10-20 08:54:56
|
||||
tester-1|0.336.0|2026-10-20 08:54:54
|
||||
@@ -0,0 +1,27 @@
|
||||
=== demo-hp (enkisfelhom.hu) 08:53:38
|
||||
containers: 24 running; not healthy/running: traefik Up 4 days;
|
||||
unhealthy: 0
|
||||
2026/10/20 08:51:19 updater.go:855: [INFO] [selfupdate] Post-update startup: update successful (0.335.0 → 0.336.0)
|
||||
allow-list refusals since start: 0
|
||||
2026/10/20 08:51:19 sync.go:142: [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
|
||||
installed apps (stacks with app.yaml): 12
|
||||
public https://felhom.enkisfelhom.hu/: 302
|
||||
public https://files.enkisfelhom.hu/: 200
|
||||
=== demo-felhom (demo-felhom.eu) 08:53:41
|
||||
containers: 9 running; not healthy/running: traefik Up 4 days;
|
||||
unhealthy: 0
|
||||
2026/10/20 08:51:08 [INFO] [selfupdate] Post-update startup: update successful (0.335.0 → 0.336.0)
|
||||
allow-list refusals since start: 0
|
||||
2026/10/20 08:51:11 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
|
||||
installed apps (stacks with app.yaml): 5
|
||||
public https://felhom.demo-felhom.eu/: 302
|
||||
public https://files.demo-felhom.eu/: 200
|
||||
=== tester-1 (enkicsifelhom.hu) 08:53:42
|
||||
containers: 10 running; not healthy/running: traefik Up 4 days;
|
||||
unhealthy: 0
|
||||
2026/10/20 08:51:18 [INFO] [selfupdate] Post-update startup: update successful (0.335.0 → 0.336.0)
|
||||
allow-list refusals since start: 0
|
||||
2026/10/20 08:51:19 [INFO] [sync] Initial sync: Sablonok naprakészek — nincs változás
|
||||
installed apps (stacks with app.yaml): 3
|
||||
public https://felhom.enkicsifelhom.hu/: 302
|
||||
public https://files.enkicsifelhom.hu/: 200
|
||||
@@ -0,0 +1,33 @@
|
||||
control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
|
||||
== tester-1 (enkicsifelhom.hu): 6 public name(s)
|
||||
OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
|
||||
OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
|
||||
== demo-hp (enkisfelhom.hu): 14 public name(s)
|
||||
OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
|
||||
OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
|
||||
== demo-felhom (demo-felhom.eu): 8 public name(s)
|
||||
OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
|
||||
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
|
||||
== every public name redirects http -> https
|
||||
@@ -0,0 +1,16 @@
|
||||
== demo-hp
|
||||
dashboard sign-in: 302
|
||||
the Fajlok button goes to: https://files.enkisfelhom.hu/_felhom/sso
|
||||
the pass answer goes to: https://files.enkisfelhom.hu/files/; files cookie set: 1
|
||||
FileBrowser with the files cookie: felhom-household admin: False modify: True
|
||||
FileBrowser without it, a fresh browser: HTTP 401
|
||||
dashboard sign-out: 302
|
||||
after the sign-out, the files cookie sends the browser to: https://felhom.enkisfelhom.hu/__files/open?to=%2Ffiles%2F
|
||||
== tester-1
|
||||
dashboard sign-in: 302
|
||||
the Fajlok button goes to: https://files.enkicsifelhom.hu/_felhom/sso
|
||||
the pass answer goes to: https://files.enkicsifelhom.hu/files/; files cookie set: 1
|
||||
FileBrowser with the files cookie: felhom-household admin: False modify: True
|
||||
FileBrowser without it, a fresh browser: HTTP 401
|
||||
dashboard sign-out: 302
|
||||
after the sign-out, the files cookie sends the browser to: https://felhom.enkicsifelhom.hu/__files/open?to=%2Ffiles%2F
|
||||
@@ -0,0 +1,22 @@
|
||||
2026-10-20T08:54:35Z
|
||||
global floor:
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /configuration?flash=floor_set
|
||||
read back (hub DB with WAL):
|
||||
min_controller_version|0.336.0
|
||||
min_controller_version_declared_min_agent|0.336.0=0.158.1
|
||||
clear the per-box floors:
|
||||
== tester-1
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/tester-1?flash=floor_set
|
||||
== demo-hp
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/demo-hp?flash=floor_set
|
||||
== demo-felhom
|
||||
HTTP/1.1 303 See Other
|
||||
Location: /customers/demo-felhom?flash=floor_set
|
||||
Error: in prepare, no such table: customers
|
||||
per-box floors after the clear (customer_configs):
|
||||
demo-felhom|''|''
|
||||
demo-hp|''|''
|
||||
tester-1|''|''
|
||||
@@ -0,0 +1,17 @@
|
||||
# Versions measured at the start of the rehearsal (2026-10-20 11:40 CEST = 09:40 UTC)
|
||||
# Method: git origin/main per repo; hub pod image (kubectl); hub /configuration page (token read); hub /hosts and /configs pages;
|
||||
# installer read live from https://felhom.eu/scripts/felhom-host-install.sh; ISO link + checksum read from https://felhom.eu/letoltes
|
||||
felhom-controller main 7ca59e2d (v0.336.0 released 10:56 CEST; brief said ae2bfb2f / v0.335.0 — CHANGED before the run started)
|
||||
felhom-agent main 4e1621c6 (v0.161.0 released + delivered to the 3 test boxes; brief said 51521230)
|
||||
felhom.eu main 5c487313 (brief said b0ae2c57)
|
||||
app-catalog main c50adfc6 (brief said 5e37aa6e)
|
||||
templates in catalog: 65 (ls templates/)
|
||||
hub pod image: gitea.dooplex.hu/admin/felhom-hub:0.162.0
|
||||
hub global floor (controller): v0.336.0 (DB override); min_agent 0.158.1
|
||||
hub Day-0 manifest: agent 0.161.0 (sha ff4e6dc5…), golden 0.331.0 (sha 3881de62…)
|
||||
installer served at felhom.eu/scripts: SCRIPT_VERSION=1.35.0
|
||||
ISO on the download page: felhom-installer-1.29.0-pve9.2-1.iso, published sha256 dceacae5da247d76cad065bf6c0d3bbefac8d8a5f8e571db2fd8449a97e94829
|
||||
fleet (hub /hosts, /configs): demo-felhom, demo-hp, tester-1 = agent 0.161.0, controller 0.336.0; Tester-2 = agent 0.142.0, controller 0.292.0 (DOWN, laptop off)
|
||||
Tester4: customer exists (domain tester4.test); NO host record (old Tester4-31fa0e is gone — precondition met);
|
||||
off-site: NOT enabled — absent from the hub's 'customers actually using an offsite tier' list (offsite_box.go:166-168),
|
||||
positive control: the other four customers are listed; no WireGuard peer for Tester4 on ep0's peer table.
|
||||
@@ -0,0 +1,19 @@
|
||||
Tue Oct 20 09:41:52 UTC 2026
|
||||
total used free shared buff/cache available
|
||||
Mem: 29994 14381 528 200 15724 15612
|
||||
Swap: 8191 792 7399
|
||||
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
|
||||
felhom-pbs pbs active 0 0 0 0.00%
|
||||
local dir active 40453376 28477900 9888360 70.40%
|
||||
local-lvm lvmthin active 56487936 33192311 23295624 58.76%
|
||||
nvme-scratch dir active 983379700 203081848 730271240 20.65%
|
||||
Filesystem 1B-blocks Used Available Use% Mounted on
|
||||
/dev/nvme0n1 1006980812800 207955812352 747797749760 22% /mnt/hdd_1
|
||||
/dev/mapper/pve-root 41424257024 29161369600 10125680640 75% /
|
||||
11:41:53 up 4 days, 6:49, 1 user, load average: 1.63, 1.42, 1.27
|
||||
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
|
||||
341 night1004-tester1 running 8192 200.00 1851
|
||||
VMID Status Lock Name
|
||||
9201 running demo-hp
|
||||
9202 running demo-hp-scratch
|
||||
9401 stopped upgrade-harness
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,51 @@
|
||||
# B6 — failure scenario order, drawn ONCE from seed 20261020 before round 1 (A6)
|
||||
|
||||
Method: `random.Random(20261020)`; shuffle the ★ list [1..10], then shuffle the rest [11..16]; this file is written before B1 bound the box (2026-10-20 ~10:00 UTC).
|
||||
|
||||
| run | scenario | what |
|
||||
|---|---|---|
|
||||
| 1 | ★2 | Power cut mid off-site restore (B8 run) |
|
||||
| 2 | ★3 | Data disk 100 % during nightly backup (via own upload); system disk 97 % |
|
||||
| 3 | ★5 | Slow lossy internet (1 Mbit/s up, 5 % loss) during first big off-site run |
|
||||
| 4 | ★9 | Two actions at once: Restore+Remove in two tabs; double click Visszaallitas |
|
||||
| 5 | ★7 | DST: clock 2026-10-25 01:30 CEST -> 04:00 CET, NTP off |
|
||||
| 6 | ★1 | Power cut mid app install (image pull) |
|
||||
| 7 | ★10 | Box off for 3 days (VM stopped, clock moved forward on start) |
|
||||
| 8 | ★4 | Data drive pulled during upload, put back (same slot; other slot/order) |
|
||||
| 9 | ★6 | Off-site path blocked for one night (ep0 + Storage Box dropped at the VM's outer side) |
|
||||
| 10 | ★8 | Clock wrong at boot: -2 years, then +2 years, NTP blocked |
|
||||
| 11 | 14 | App removed keeping data, installed again (use kept data) |
|
||||
| 12 | 16 | English browser: one full pass of the dashboard |
|
||||
| 13 | 11 | Hard reset mid app update |
|
||||
| 14 | 12 | DNS answers nothing for an hour |
|
||||
| 15 | 15 | Box restarted during the catch-up of scenario 10 |
|
||||
| 16 | 13 | Second drive added later: offered as backup target; accept; remove safely |
|
||||
|
||||
## Dependencies fixed NOW (before any result), so the order is never re-chosen after one
|
||||
|
||||
- **2** needs a running off-site restore — it runs INSIDE B8's A.1.5 run whatever its slot; its slot is then used for the next scenario.
|
||||
- **15** runs inside **10**'s catch-up (it is the restart during that catch-up) — it rides with 10.
|
||||
- **5** needs the FIRST big off-site run — it runs on night one (B5) if off-site is on by then; otherwise at its slot with a fresh large upload, and it is marked „not the first run".
|
||||
- **6** spans one night — it takes the second night if the run lasts; otherwise a shortened window (≥ 1 off-site schedule), marked.
|
||||
- **7** (DST) and **8** (wrong clock) change the VM's clock; they run after the off-site and restore work they could disturb, but in this drawn order relative to each other.
|
||||
- Anything that needs off-site (2, 5, 6) is SKIPPED, not reordered, if Tester4's off-site is still off when its slot comes.
|
||||
|
||||
|
||||
## Amendment written 2026-10-20 ~15:45 UTC — BEFORE scenario 5 runs (no result seen)
|
||||
|
||||
- **5** (slow, lossy internet during the first big off-site run): the first big run is started by the household's
|
||||
„Távoli mentés most" this evening, with `tc netem` 1 Mbit/s up + 5 % loss on the VM's tap for its FIRST 30 MINUTES;
|
||||
then the shaping is removed and the run must finish (or resume) by itself. Reason: at 1 Mbit/s the ~3.7 GB first copy
|
||||
needs ~8 h, and B7 (box died) needs a complete off-site copy by the morning. What is measured: progress, what the
|
||||
page and the hub say while slow, and whether the run finishes or resumes after.
|
||||
- The box froze twice under ordinary uploads (R-1192). Every heavy scenario now runs ALONE, with the health watch
|
||||
beside it; a freeze during a scenario is recorded as such, not as the scenario's own result.
|
||||
|
||||
## Amendment 2026-10-20 ~17:50 UTC — before any of these ran
|
||||
|
||||
Run tonight, before the night chain (B5 must stay „no hands"): #16 English pass, #9 two-at-once, #13 second drive later,
|
||||
#4 drive pulled, #3 disk full (the „nightly backup" stood in by „Mentés most" — the one real night is reserved for B5;
|
||||
the filler goes to a share with Felhőmentés OFF and is deleted before the night), #12 DNS dead, #6 off-site blocked
|
||||
(shortened to one manual run + the catch-up, not a night — same reason). After the night: #7 DST, #8 wrong clock,
|
||||
#10 three days off (+#15 inside it), #1 power cut mid install, #14 kept data, #11 if an update exists (none expected —
|
||||
the catalog is frozen). Reason: the clock scenarios move the box's time and would spoil the first night's record.
|
||||
@@ -0,0 +1,5 @@
|
||||
dceacae5da247d76cad065bf6c0d3bbefac8d8a5f8e571db2fd8449a97e94829 felhom-installer-1.29.0-pve9.2-1.iso
|
||||
|
||||
dceacae5da247d76cad065bf6c0d3bbefac8d8a5f8e571db2fd8449a97e94829 rehearsal1020-felhom-installer-1.29.0-pve9.2-1.iso
|
||||
1705324544
|
||||
secs=62
|
||||
@@ -0,0 +1,19 @@
|
||||
Formatting '/mnt/hdd_1/images/9320/vm-9320-disk-0.raw', fmt=raw size=274877906944 preallocation=off
|
||||
scsi0: successfully created disk 'nvme-scratch:9320/vm-9320-disk-0.raw,size=256G'
|
||||
Formatting '/mnt/hdd_1/images/9320/vm-9320-disk-1.raw', fmt=raw size=214748364800 preallocation=off
|
||||
scsi1: successfully created disk 'nvme-scratch:9320/vm-9320-disk-1.raw,size=200G'
|
||||
boot: order=scsi0;ide2
|
||||
cores: 4
|
||||
cpu: host
|
||||
ide2: local:iso/rehearsal1020-felhom-installer-1.29.0-pve9.2-1.iso,media=cdrom,size=1665356K
|
||||
memory: 8192
|
||||
meta: creation-qemu=11.0.3,ctime=1792489316
|
||||
name: rehearsal1020-box1
|
||||
net0: virtio=BC:24:11:3F:4F:3F,bridge=vmbr0
|
||||
ostype: l26
|
||||
scsi0: nvme-scratch:9320/vm-9320-disk-0.raw,size=256G
|
||||
scsi1: nvme-scratch:9320/vm-9320-disk-1.raw,size=200G
|
||||
scsihw: virtio-scsi-single
|
||||
smbios1: uuid=323698f4-6179-4f14-83c0-8109044c5cda
|
||||
vga: std
|
||||
vmgenid: 976cd28b-5675-42f2-bd65-fb0ec8ae6be7
|
||||
@@ -0,0 +1,23 @@
|
||||
# B1 — download and install (2026-10-20, 09:40–09:51 UTC)
|
||||
|
||||
Method: the guide's URL `felhom.eu/letoltes` (no public page links to it — by design, the guide carries it); the ISO
|
||||
downloaded onto demo-hp; VM 9320 (4 vCPU, 8 GiB, 256 GB system disk + 200 GB data disk, raw on `nvme-scratch` =
|
||||
`/mnt/hdd_1` root); the graphical installer driven with `qm sendkey` (`tools/vmctl.sh`), the **Hungarian keyboard
|
||||
kept**; screenshots by QEMU screendump (`screens/install/`).
|
||||
|
||||
| step | result | vs the guide (`runbooks/VOLUNTEER-first-hour.md`) |
|
||||
|---|---|---|
|
||||
| download | 1 705 324 544 B in 62 s; sha256 `dceacae5…4829` = the page = the `.sha256` file (`01-iso-download.txt`) | as written. **But the download link is not visible as a link** (white bold file name; no button, no colour, no underline — phone and computer, `screens/b1-site/`) |
|
||||
| boot menu | auto-boots „Felhom telepítés / Install Felhom" | as written; the boot text „válassza ki a lemezt" is FORMAL (Ön) — the guide and dashboard say „te" |
|
||||
| EULA → disk | `/dev/sda (256 GiB)` pre-selected = the right disk | as written. Proxmox's own text „Final configuration will be done on the graphical user interface, via a web browser" is not true for a household |
|
||||
| location | Hungary / Europe/Budapest / Hungarian, pre-filled | as written. **Enter in the Country field does not advance** — a keyboard user must use Next |
|
||||
| password + e-mail | ≥ 8 chars; the e-mail field pre-filled `mail@example.invalid` | as written |
|
||||
| network | DHCP address frozen as a STATIC one (192.168.0.134/24) | as written („hagyd, ahogy van"); the static address is the R-1166 shape (a later DHCP lease can clash) |
|
||||
| hostname | `felhom.tester4.test` | as written |
|
||||
| summary | auto-reboot ticked; Install pressed 09:46:50 | as written |
|
||||
| first boot | installer reboots by itself; the console: „Felhom otthoni szerver / Ezen a gépen most nincs dolgod, és bejelentkezni sem kell. A beállításhoz kövesd a Felhomtól kapott útmutatót." | **wording differs from the guide** (guide: „Ezen a képernyőn nincs teendőd…") — harmless |
|
||||
| pairing code | shown ≤ 09:51 (≤ 4 min after Install), HU + EN, code blacked out in the screenshot | as written |
|
||||
|
||||
Harness substitutions (not interventions): H3 the VM boots its disk first (= the stick pulled in time); H5 a nested
|
||||
VM with virtual disks; the public domain `tester4.test` never resolves (Tester4 is a test customer with no tunnel) —
|
||||
the dashboard is reached on its LAN address with the `Host:` header, which a household cannot do.
|
||||
@@ -0,0 +1,9 @@
|
||||
immich running deployed= True sub= photos gate= True
|
||||
nextcloud running deployed= True sub= cloud gate= False
|
||||
paperless-ngx running deployed= True sub= paperless gate= False
|
||||
vaultwarden running deployed= True sub= vault gate= False
|
||||
mealie running deployed= True sub= recipes gate= False
|
||||
actualbudget running deployed= True sub= budget gate= True
|
||||
radicale running deployed= True sub= calendar gate= False
|
||||
bookstack running deployed= True sub= wiki gate= False
|
||||
jellyfin not_deployed deployed= False sub= None gate= False
|
||||
@@ -0,0 +1,40 @@
|
||||
14:13:55 diskread=3GB health=200 0.026323
|
||||
14:14:27 diskread=3GB health=200 0.010294
|
||||
14:14:59 diskread=4GB health=200 0.011968
|
||||
14:15:30 diskread=4GB health=200 0.009450
|
||||
14:16:02 diskread=4GB health=200 0.010128
|
||||
14:16:33 diskread=4GB health=200 0.009558
|
||||
14:17:05 diskread=4GB health=200 0.010571
|
||||
14:17:36 diskread=4GB health=200 0.009807
|
||||
14:18:08 diskread=4GB health=200 0.010191
|
||||
14:18:40 diskread=4GB health=200 0.009850
|
||||
14:19:11 diskread=4GB health=200 0.010203
|
||||
14:19:43 diskread=4GB health=200 0.011617
|
||||
14:20:14 diskread=4GB health=200 0.014898
|
||||
14:20:46 diskread=4GB health=200 0.010567
|
||||
14:21:17 diskread=4GB health=200 0.011040
|
||||
14:21:49 diskread=4GB health=200 0.009551
|
||||
14:22:21 diskread=4GB health=200 0.009957
|
||||
14:22:52 diskread=4GB health=200 0.010580
|
||||
14:23:24 diskread=4GB health=200 0.010163
|
||||
14:23:56 diskread=4GB health=200 0.010898
|
||||
14:24:27 diskread=4GB health=200 0.010501
|
||||
14:24:58 diskread=4GB health=200 0.011854
|
||||
14:25:30 diskread=4GB health=200 0.011567
|
||||
14:26:01 diskread=4GB health=200 0.010254
|
||||
14:26:32 diskread=4GB health=200 0.010620
|
||||
14:27:03 diskread=4GB health=200 0.011295
|
||||
14:27:35 diskread=4GB health=200 0.010245
|
||||
14:28:06 diskread=4GB health=200 0.010855
|
||||
14:28:37 diskread=4GB health=200 0.010371
|
||||
14:29:09 diskread=4GB health=200 0.015567
|
||||
14:29:40 diskread=4GB health=200 0.010465
|
||||
14:30:11 diskread=4GB health=200 0.010190
|
||||
14:30:42 diskread=4GB health=200 0.011160
|
||||
14:31:14 diskread=4GB health=200 0.009954
|
||||
14:31:45 diskread=4GB health=200 0.011131
|
||||
14:32:16 diskread=4GB health=200 0.011337
|
||||
14:32:48 diskread=4GB health=200 0.010336
|
||||
14:33:19 diskread=4GB health=200 0.011064
|
||||
14:33:50 diskread=4GB health=200 0.010944
|
||||
14:34:22 diskread=4GB health=200 0.010249
|
||||
@@ -0,0 +1,2 @@
|
||||
run 200 {'ok': True, 'message': 'Mentés elindítva'}
|
||||
120 s {"db_dump": {"count": 4, "duration": "1m58.175464265s", "last_run": "2026-10-20T16:30:58.136294428Z", "success": true}, "enabled": true, "running": false}
|
||||
@@ -0,0 +1,16 @@
|
||||
UTC cpu% load1 mem_GB (the box's own /api/metrics/system history)
|
||||
12:00 1.4 1.6 0.9384765625
|
||||
12:07 6.1 0.9 0.93359375
|
||||
12:14 3.0 0.6 0.970703125
|
||||
12:21 14.0 1.6 1.5107421875
|
||||
12:28 1.0 0.3 1.6123046875
|
||||
12:36 1.5 0.3 1.607421875
|
||||
12:43 11.8 1.4 1.6298828125
|
||||
12:50 15.2 1.5 2.056640625
|
||||
12:57 61.3 8.3 2.318359375
|
||||
13:33 78.1 139.3 2.2900390625
|
||||
13:55 78.1 151.6 2.3486328125
|
||||
14:09 0.0 2.0 1.1875
|
||||
keys: ['cpu', 'labels', 'load1', 'memory', 'temp']
|
||||
{"ok": true, "data": {"sync_status": {"last_sync": "2026-10-20T14:13:31.264994519Z", "last_status": "ok", "syncing": false}, "system": {"total_mem_mb": 4096, "used_mem_mb": 3638, "avail_mem_mb": 458, "mem_percent": 88.818359375, "disk_total_gb": 49.03971862792969, "disk_used_gb": 12.562141418457031, "disk_avail_gb": 33.96195983886719, "disk_percent": 27.001362904306276, "disk_known": true, "hdd_total_gb": 195.8027114868164, "hdd_used_gb": 0.10123825073242188, "hdd_avail_gb": 185.68584823608398, "hdd_percent": 0.054491543328877075, "hdd_configured": true, "hdd_known": true, "cpu_percent": 55.314861460957175, "load_avg_1": 3.31, "load_avg_5": 1.24, "load_avg_15": 0.44, "temperature_celsius": 0}}}
|
||||
{"ok": true, "data": [{"name": "felhom-controller", "cpu_percent": 42.69, "mem_usage_mb": 92.52, "mem_limit_mb": 4096}, {"name": "actualbudget", "cpu_percent": 31.28, "mem_usage_mb": 128.6, "mem_limit_mb": 256}, {"name": "radicale", "cpu_percent": 28.31, "mem_usage_mb": 28.39, "mem_limit_mb": 128}, {"name": "bookstack-db", "cpu_percent": 27.86, "mem_usage_mb": 153.2, "mem_limit_mb": 256}, {"name": "immich-machine-learning", "cpu_percent": 27.57, "mem_usage_mb": 107.2, "mem_limit_mb": 2048}, {"name": "paperless-webserver", "cpu_percent": 27.41, "mem_usage_mb": 72.22, "mem_limit_mb": 1280}, {"name": "mealie", "cpu_percent": 26.57, "mem_usage_mb": 67.21, "mem_limit_mb": 1000}, {"name": "nextcloud-db", "cpu_percent": 26.17, "mem_usage_mb": 116.3, "mem_limit_mb": 512}, {"name": "immich-server", "cpu_percent": 22.66, "mem_usage_mb": 206, "mem_limit_mb": 3072}, {"name": "nextcloud", "cpu_percent": 19.4, "mem_usage_mb": 157.1, "mem_limit_mb": 1024}, {"name": "traefik", "cpu_percent": 10.78, "mem_usage_mb": 186.4, "mem_limit_mb": 4096}, {"name": "bookstack", "cpu_percent": 0.61, "mem_usage_mb": 19.94, "mem_limit_mb": 512}, {"name": "paperless-redis", "cpu_percent": 0.47, "mem_usage_mb": 26.13, "mem_limit_mb": 128}, {"name": "immich-redis", "cpu_percent": 0.43, "mem_usage_mb": 3.16, "mem_limit_mb": 128}, {"name": "nextcloud-redis", "cpu_percent": 0.42, "mem_usage_mb": 10.36, "mem_limit_mb": 128}, {"name": "paperless-postgres", "cpu_percent": 0.03, "mem_usage_mb": 58.27, "mem_limit_mb": 256}, {"name": "immich-postgres", "cpu_percent": 0.03, "mem_usage_mb": 133.1, "mem_limit_mb": 768}, {"name": "vaultwarden", "cpu_percent": 0, "mem_usage_mb": 72.27, "mem_limit_mb": 256}, {"name": "filebrowser", "cpu_percent": 0, "mem_usage_mb": 145, "mem_limit_mb": 256}]}
|
||||
@@ -0,0 +1,60 @@
|
||||
15:26:05 health=200 0.011492
|
||||
15:26:35 health=200 0.013093
|
||||
15:27:05 health=200 0.325949
|
||||
15:27:55 health=000 20.002381
|
||||
15:28:45 health=000 20.002064
|
||||
15:29:35 health=000 20.002874
|
||||
15:30:26 health=000 20.002573
|
||||
15:31:16 health=000 20.002897
|
||||
15:32:06 health=000 20.002446
|
||||
15:32:56 health=000 20.002815
|
||||
15:33:46 health=000 20.002487
|
||||
15:34:36 health=000 20.002728
|
||||
15:35:26 health=000 20.002934
|
||||
15:36:16 health=000 20.003039
|
||||
15:37:06 health=000 20.002029
|
||||
15:37:56 health=000 20.002896
|
||||
15:38:46 health=000 20.002072
|
||||
15:39:36 health=000 20.002908
|
||||
15:40:26 health=000 20.002099
|
||||
15:41:16 health=000 20.001909
|
||||
15:42:06 health=000 20.002225
|
||||
15:42:56 health=000 20.002048
|
||||
15:43:46 health=000 20.003050
|
||||
15:44:36 health=000 20.001694
|
||||
15:45:26 health=000 20.002376
|
||||
15:46:16 health=000 20.002820
|
||||
15:47:06 health=000 20.002634
|
||||
15:47:56 health=000 20.002259
|
||||
15:48:46 health=000 20.002093
|
||||
15:49:36 health=000 20.002486
|
||||
15:50:26 health=000 20.001753
|
||||
15:51:16 health=000 20.001476
|
||||
15:52:06 health=000 20.002140
|
||||
15:52:56 health=000 20.002111
|
||||
15:53:46 health=000 20.002107
|
||||
15:54:36 health=000 20.002294
|
||||
15:55:26 health=000 20.002923
|
||||
15:56:16 health=000 20.002691
|
||||
15:57:06 health=000 20.002300
|
||||
15:57:56 health=000 20.002884
|
||||
15:58:46 health=000 20.002233
|
||||
15:59:36 health=000 20.001755
|
||||
16:00:26 health=000 20.002217
|
||||
16:01:16 health=000 20.002917
|
||||
16:02:06 health=000 20.001900
|
||||
16:02:41 health=200 4.333646
|
||||
16:03:31 health=000 20.002229
|
||||
16:04:21 health=000 20.002154
|
||||
16:05:11 health=000 20.002066
|
||||
16:06:01 health=000 20.002208
|
||||
16:06:51 health=000 20.002962
|
||||
16:07:41 health=000 20.001192
|
||||
16:08:31 health=000 20.002116
|
||||
16:09:21 health=000 20.002092
|
||||
16:10:11 health=000 20.002032
|
||||
16:11:01 health=000 20.002964
|
||||
16:11:51 health=000 20.002792
|
||||
16:12:41 health=000 20.001994
|
||||
16:13:31 health=000 20.002290
|
||||
16:14:21 health=000 20.002875
|
||||
@@ -0,0 +1,118 @@
|
||||
2026-10-20T16:03:26 health=000 20.001467 diskread=250GB
|
||||
2026-10-20T16:04:47 health=000 20.002188 diskread=257GB
|
||||
2026-10-20T16:06:08 health=000 20.002982 diskread=263GB
|
||||
2026-10-20T16:07:30 health=000 20.001529 diskread=271GB
|
||||
2026-10-20T16:08:51 health=000 20.001679 diskread=278GB
|
||||
2026-10-20T16:10:12 health=000 20.002149 diskread=285GB
|
||||
2026-10-20T16:11:33 health=000 20.002090 diskread=294GB
|
||||
2026-10-20T16:12:55 health=000 20.002713 diskread=301GB
|
||||
2026-10-20T16:14:16 health=000 20.002244 diskread=0GB
|
||||
2026-10-20T16:15:21 health=000 3.052227 diskread=0GB
|
||||
2026-10-20T16:16:23 health=200 0.028660 diskread=3GB
|
||||
2026-10-20T16:17:24 health=200 0.011805 diskread=4GB
|
||||
2026-10-20T16:18:26 health=200 0.011279 diskread=4GB
|
||||
2026-10-20T16:19:28 health=200 0.011336 diskread=4GB
|
||||
2026-10-20T16:20:29 health=200 0.010931 diskread=4GB
|
||||
2026-10-20T16:21:31 health=200 0.010568 diskread=4GB
|
||||
2026-10-20T16:22:32 health=200 0.012190 diskread=4GB
|
||||
2026-10-20T16:23:34 health=200 0.009670 diskread=4GB
|
||||
2026-10-20T16:24:36 health=200 0.011174 diskread=4GB
|
||||
2026-10-20T16:25:37 health=200 0.010032 diskread=4GB
|
||||
2026-10-20T16:26:39 health=200 0.010610 diskread=4GB
|
||||
2026-10-20T16:27:40 health=200 0.010425 diskread=4GB
|
||||
2026-10-20T16:28:42 health=200 0.012126 diskread=4GB
|
||||
2026-10-20T16:29:44 health=200 0.010823 diskread=7GB
|
||||
2026-10-20T16:30:45 health=200 0.009510 diskread=9GB
|
||||
2026-10-20T16:31:47 health=200 0.259351 diskread=10GB
|
||||
2026-10-20T16:32:49 health=200 0.137947 diskread=13GB
|
||||
2026-10-20T16:33:51 health=200 0.048527 diskread=14GB
|
||||
2026-10-20T16:34:53 health=200 0.236325 diskread=15GB
|
||||
2026-10-20T16:35:55 health=200 0.729571 diskread=15GB
|
||||
2026-10-20T16:36:57 health=200 0.306518 diskread=15GB
|
||||
2026-10-20T16:37:59 health=200 0.328963 diskread=15GB
|
||||
2026-10-20T16:39:01 health=200 0.135941 diskread=15GB
|
||||
2026-10-20T16:40:03 health=200 0.342418 diskread=15GB
|
||||
2026-10-20T16:41:04 health=200 0.057255 diskread=15GB
|
||||
2026-10-20T16:42:06 health=200 0.055219 diskread=15GB
|
||||
2026-10-20T16:43:08 health=200 0.047625 diskread=15GB
|
||||
2026-10-20T16:44:09 health=200 0.048725 diskread=15GB
|
||||
2026-10-20T16:45:11 health=200 0.047987 diskread=15GB
|
||||
2026-10-20T16:46:13 health=200 0.046522 diskread=15GB
|
||||
2026-10-20T16:47:14 health=200 0.045488 diskread=15GB
|
||||
2026-10-20T16:48:16 health=200 0.060400 diskread=15GB
|
||||
2026-10-20T16:49:18 health=200 0.053359 diskread=16GB
|
||||
2026-10-20T16:50:19 health=200 0.045452 diskread=16GB
|
||||
2026-10-20T16:51:21 health=200 0.054048 diskread=16GB
|
||||
2026-10-20T16:52:23 health=200 0.046350 diskread=16GB
|
||||
2026-10-20T16:53:25 health=200 0.059690 diskread=16GB
|
||||
2026-10-20T16:54:27 health=200 0.289137 diskread=16GB
|
||||
2026-10-20T16:55:29 health=200 0.274621 diskread=16GB
|
||||
2026-10-20T16:56:31 health=200 0.150415 diskread=16GB
|
||||
2026-10-20T16:57:33 health=200 0.407832 diskread=16GB
|
||||
2026-10-20T16:58:34 health=200 0.109780 diskread=16GB
|
||||
2026-10-20T16:59:36 health=200 0.166653 diskread=16GB
|
||||
2026-10-20T17:00:38 health=200 0.157504 diskread=16GB
|
||||
2026-10-20T17:01:39 health=200 0.010882 diskread=17GB
|
||||
2026-10-20T17:02:41 health=200 0.010171 diskread=19GB
|
||||
2026-10-20T17:03:42 health=200 0.010227 diskread=19GB
|
||||
2026-10-20T17:04:44 health=200 0.010828 diskread=21GB
|
||||
2026-10-20T17:05:46 health=200 0.010940 diskread=23GB
|
||||
2026-10-20T17:06:48 health=200 0.014798 diskread=25GB
|
||||
2026-10-20T17:07:49 health=200 0.030366 diskread=27GB
|
||||
2026-10-20T17:08:51 health=200 0.012455 diskread=27GB
|
||||
2026-10-20T17:09:52 health=200 0.012290 diskread=27GB
|
||||
2026-10-20T17:10:54 health=200 0.010052 diskread=27GB
|
||||
2026-10-20T17:11:56 health=200 0.011465 diskread=27GB
|
||||
2026-10-20T17:12:57 health=200 0.015121 diskread=27GB
|
||||
2026-10-20T17:13:59 health=200 0.011157 diskread=27GB
|
||||
2026-10-20T17:15:01 health=200 0.012024 diskread=27GB
|
||||
2026-10-20T17:16:02 health=200 0.010357 diskread=28GB
|
||||
2026-10-20T17:17:04 health=200 0.017752 diskread=33GB
|
||||
2026-10-20T17:18:05 health=200 0.009962 diskread=33GB
|
||||
2026-10-20T17:19:07 health=200 0.011363 diskread=33GB
|
||||
2026-10-20T17:20:09 health=200 0.011524 diskread=34GB
|
||||
2026-10-20T17:21:10 health=200 0.009969 diskread=34GB
|
||||
2026-10-20T17:22:12 health=200 0.841755 diskread=35GB
|
||||
2026-10-20T17:23:14 health=200 0.010010 diskread=38GB
|
||||
2026-10-20T17:24:16 health=200 0.012834 diskread=42GB
|
||||
2026-10-20T17:25:18 health=200 0.012292 diskread=42GB
|
||||
2026-10-20T17:26:20 health=200 0.014307 diskread=42GB
|
||||
2026-10-20T17:27:21 health=200 0.016217 diskread=42GB
|
||||
2026-10-20T17:28:23 health=200 0.010172 diskread=42GB
|
||||
2026-10-20T17:29:25 health=000 0.331161 diskread=0GB
|
||||
2026-10-20T17:30:29 health=000 3.054892 diskread=0GB
|
||||
2026-10-20T17:31:31 health=200 0.011323 diskread=3GB
|
||||
2026-10-20T17:32:33 health=200 0.009743 diskread=4GB
|
||||
2026-10-20T17:33:35 health=200 0.012219 diskread=4GB
|
||||
2026-10-20T17:34:39 health=000 3.077297 diskread=0GB
|
||||
2026-10-20T17:35:44 health=000 3.069825 diskread=0GB
|
||||
2026-10-20T17:36:46 health=200 0.011239 diskread=3GB
|
||||
2026-10-20T17:37:47 health=200 0.010636 diskread=4GB
|
||||
2026-10-20T17:38:49 health=200 0.009873 diskread=4GB
|
||||
2026-10-20T17:39:50 health=200 0.011005 diskread=4GB
|
||||
2026-10-20T17:40:52 health=200 0.011215 diskread=5GB
|
||||
2026-10-20T17:41:53 health=200 0.011258 diskread=5GB
|
||||
2026-10-20T17:42:54 health=200 0.012194 diskread=5GB
|
||||
2026-10-20T17:43:56 health=200 0.010113 diskread=5GB
|
||||
2026-10-20T17:44:58 health=200 0.010286 diskread=5GB
|
||||
2026-10-20T17:45:59 health=200 0.010282 diskread=5GB
|
||||
2026-10-20T17:47:01 health=200 0.010560 diskread=5GB
|
||||
2026-10-20T17:48:02 health=200 0.009732 diskread=5GB
|
||||
2026-10-20T17:49:03 health=200 0.010065 diskread=5GB
|
||||
2026-10-20T17:50:05 health=200 0.010006 diskread=5GB
|
||||
2026-10-20T17:51:06 health=200 0.010605 diskread=5GB
|
||||
2026-10-20T17:52:08 health=200 0.012332 diskread=5GB
|
||||
2026-10-20T17:53:09 health=200 0.010066 diskread=5GB
|
||||
2026-10-20T17:54:11 health=200 0.011118 diskread=5GB
|
||||
2026-10-20T17:55:12 health=200 0.010705 diskread=5GB
|
||||
2026-10-20T17:56:13 health=200 0.011581 diskread=5GB
|
||||
2026-10-20T17:57:14 health=200 0.009849 diskread=6GB
|
||||
2026-10-20T17:58:16 health=200 0.010308 diskread=6GB
|
||||
2026-10-20T17:59:17 health=200 0.009395 diskread=6GB
|
||||
2026-10-20T18:00:19 health=200 0.010613 diskread=6GB
|
||||
2026-10-20T18:01:20 health=200 0.010129 diskread=7GB
|
||||
2026-10-20T18:02:22 health=200 0.010121 diskread=7GB
|
||||
2026-10-20T18:03:24 health=200 0.012405 diskread=7GB
|
||||
2026-10-20T18:04:25 health=200 0.012071 diskread=7GB
|
||||
2026-10-20T18:05:27 health=200 0.012165 diskread=7GB
|
||||
2026-10-20T18:06:29 health=200 0.021087 diskread=8GB
|
||||
@@ -0,0 +1,49 @@
|
||||
15:26:49
|
||||
Traceback (most recent call last):
|
||||
File "/root/work/apps.py", line 206, in <module>
|
||||
fn(*sys.argv[2:])
|
||||
~~^^^^^^^^^^^^^^^
|
||||
File "/root/work/apps.py", line 168, in immich
|
||||
st, out, _ = http("POST", base + "/api/assets", body, dict(tok, **{"Content-Type": ct}))
|
||||
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/root/work/apps.py", line 34, in http
|
||||
with urllib.request.urlopen(req, context=CTX, timeout=timeout) as r:
|
||||
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/lib/python3.13/urllib/request.py", line 189, in urlopen
|
||||
return opener.open(url, data, timeout)
|
||||
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/lib/python3.13/urllib/request.py", line 489, in open
|
||||
response = self._open(req, data)
|
||||
File "/usr/lib/python3.13/urllib/request.py", line 506, in _open
|
||||
result = self._call_chain(self.handle_open, protocol, protocol +
|
||||
'_open', req)
|
||||
File "/usr/lib/python3.13/urllib/request.py", line 466, in _call_chain
|
||||
result = func(*args)
|
||||
File "/usr/lib/python3.13/urllib/request.py", line 1367, in https_open
|
||||
return self.do_open(http.client.HTTPSConnection, req,
|
||||
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
context=self._context)
|
||||
^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/lib/python3.13/urllib/request.py", line 1323, in do_open
|
||||
r = h.getresponse()
|
||||
File "/usr/lib/python3.13/http/client.py", line 1459, in getresponse
|
||||
response.begin()
|
||||
~~~~~~~~~~~~~~^^
|
||||
File "/usr/lib/python3.13/http/client.py", line 336, in begin
|
||||
version, status, reason = self._read_status()
|
||||
~~~~~~~~~~~~~~~~~^^
|
||||
File "/usr/lib/python3.13/http/client.py", line 297, in _read_status
|
||||
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
|
||||
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
|
||||
File "/usr/lib/python3.13/socket.py", line 719, in readinto
|
||||
return self._sock.recv_into(b)
|
||||
~~~~~~~~~~~~~~~~~~~~^^^
|
||||
File "/usr/lib/python3.13/ssl.py", line 1304, in recv_into
|
||||
return self.read(nbytes, buffer)
|
||||
~~~~~~~~~^^^^^^^^^^^^^^^^
|
||||
File "/usr/lib/python3.13/ssl.py", li15:26:50Z immich admin-sign-up 400 b'{"message":"Admin setup is not available"}'
|
||||
ne 1138, in read
|
||||
return self._sslobj.read(len, buffer)
|
||||
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
|
||||
TimeoutError: The read operation timed out
|
||||
15:32:07
|
||||
@@ -0,0 +1,49 @@
|
||||
12:59:51
|
||||
Traceback (most recent call last):
|
||||
File "/root/work/apps.py", line 182, in <module>
|
||||
fn(*sys.argv[2:])
|
||||
~~^^^^^^^^^^^^^^^
|
||||
File "/root/work/apps.py", line 168, in immich
|
||||
st, out, _ = http("POST", base + "/api/assets", body, dict(tok, **{"Content-Type": ct}))
|
||||
~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/root/work/apps.py", line 34, in http
|
||||
with urllib.request.urlopen(req, context=CTX, timeout=timeout) as r:
|
||||
~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/lib/python3.13/urllib/request.py", line 189, in urlopen
|
||||
return opener.open(url, data, timeout)
|
||||
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/lib/python3.13/urllib/request.py", line 489, in open
|
||||
response = self._open(req, data)
|
||||
File "/usr/lib/python3.13/urllib/request.py", line 506, in _open
|
||||
result = self._call_chain(self.handle_open, protocol, protocol +
|
||||
'_open', req)
|
||||
File "/usr/lib/python3.13/urllib/request.py", line 466, in _call_chain
|
||||
result = func(*args)
|
||||
File "/usr/lib/python3.13/urllib/request.py", line 1367, in https_open
|
||||
return self.do_open(http.client.HTTPSConnection, req,
|
||||
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
context=self._context)
|
||||
^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/lib/python3.13/urllib/request.py", line 1323, in do_open
|
||||
r = h.getresponse()
|
||||
File "/usr/lib/python3.13/http/client.py", line 1459, in getresponse
|
||||
response.begin()
|
||||
~~~~~~~~~~~~~~^^
|
||||
File "/usr/lib/python3.13/http/client.py", line 336, in begin
|
||||
version, status, reason = self._read_status()
|
||||
~~~~~~~~~~~~~~~~~^^
|
||||
File "/usr/lib/python3.13/http/client.py", line 297, in _read_status
|
||||
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
|
||||
~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
|
||||
File "/usr/lib/python3.13/socket.py", line 719, in readinto
|
||||
return self._sock.recv_into(b)
|
||||
~~~~~~~~~~~~~~~~~~~~^^^
|
||||
File "/usr/lib/python3.13/ssl.py", line 1304, in recv_into
|
||||
return self.read(nbytes, buffer)
|
||||
~~~~~~~~~^^^^^^^^^^^^^^^^
|
||||
File "/usr/lib/python3.13/ssl.py", line 1138, in read
|
||||
return self._sslobj.read(len, buffer)
|
||||
~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
|
||||
TimeoutError: The read operation timed out
|
||||
12:59:52Z immich admin-sign-up 400 b'{"message":"Admin setup is not available"}'
|
||||
13:05:57
|
||||
@@ -0,0 +1,218 @@
|
||||
16:16:15Z used=3638MB avail=458MB load1=4.61 cpu=67% nextcloud-db:270/512MB 29% | immich-server:232/3072MB 25% | traefik:184/4096MB 26% | nextcloud:157/1024MB 0% | filebrowser:145/256MB 0% | immich-postgres:134/768MB 0% | actualbudget:130/256MB 27% | immich-machine-learning:106/2048MB 29%
|
||||
16:16:46Z used=3638MB avail=458MB load1=4.47 cpu=3% nextcloud-db:270/512MB 29% | immich-server:232/3072MB 25% | traefik:184/4096MB 26% | nextcloud:157/1024MB 0% | filebrowser:145/256MB 0% | immich-postgres:134/768MB 0% | actualbudget:130/256MB 27% | immich-machine-learning:106/2048MB 29%
|
||||
16:17:16Z used=3638MB avail=458MB load1=4.59 cpu=74% immich-server:1094/3072MB 0% | mealie:215/1000MB 0% | immich-machine-learning:173/2048MB 0% | paperless-webserver:148/1280MB 175% | nextcloud-db:118/512MB 0% | bookstack-db:104/256MB 0% | immich-postgres:78/768MB 0% | filebrowser:64/256MB 0%
|
||||
16:17:46Z used=3638MB avail=458MB load1=3.6 cpu=3% immich-server:1094/3072MB 0% | mealie:215/1000MB 0% | immich-machine-learning:173/2048MB 0% | paperless-webserver:148/1280MB 175% | nextcloud-db:118/512MB 0% | bookstack-db:104/256MB 0% | immich-postgres:78/768MB 0% | filebrowser:64/256MB 0%
|
||||
16:18:16Z used=3638MB avail=458MB load1=4.3 cpu=7% immich-server:1018/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:220/1000MB 0% | immich-machine-learning:134/2048MB 0% | nextcloud-db:101/512MB 0% | bookstack-db:98/256MB 0% | immich-postgres:90/768MB 0% | felhom-controller:89/4096MB 26%
|
||||
16:18:46Z used=3638MB avail=458MB load1=2.92 cpu=5% immich-server:1018/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:220/1000MB 0% | immich-machine-learning:134/2048MB 0% | nextcloud-db:101/512MB 0% | bookstack-db:98/256MB 0% | immich-postgres:90/768MB 0% | felhom-controller:89/4096MB 26%
|
||||
16:19:16Z used=3638MB avail=458MB load1=2.47 cpu=6% immich-server:778/3072MB 0% | paperless-webserver:545/1280MB 0% | mealie:218/1000MB 0% | immich-machine-learning:136/2048MB 0% | nextcloud-db:101/512MB 6% | bookstack-db:97/256MB 0% | felhom-controller:90/4096MB 20% | immich-postgres:90/768MB 0%
|
||||
16:19:47Z used=3638MB avail=458MB load1=1.65 cpu=5% immich-server:778/3072MB 0% | paperless-webserver:545/1280MB 0% | mealie:218/1000MB 0% | immich-machine-learning:136/2048MB 0% | nextcloud-db:101/512MB 6% | bookstack-db:97/256MB 0% | felhom-controller:90/4096MB 20% | immich-postgres:90/768MB 0%
|
||||
16:20:17Z used=3638MB avail=458MB load1=1.52 cpu=7% immich-server:778/3072MB 0% | paperless-webserver:545/1280MB 0% | mealie:218/1000MB 0% | immich-machine-learning:136/2048MB 0% | nextcloud-db:102/512MB 6% | bookstack-db:97/256MB 0% | felhom-controller:92/4096MB 26% | immich-postgres:90/768MB 0%
|
||||
16:20:47Z used=3638MB avail=458MB load1=1.06 cpu=8% immich-server:778/3072MB 0% | paperless-webserver:545/1280MB 0% | mealie:218/1000MB 0% | immich-machine-learning:136/2048MB 0% | nextcloud-db:102/512MB 6% | bookstack-db:97/256MB 0% | felhom-controller:92/4096MB 26% | immich-postgres:90/768MB 0%
|
||||
16:21:17Z used=3638MB avail=458MB load1=1.22 cpu=9% immich-server:780/3072MB 2% | paperless-webserver:550/1280MB 0% | mealie:218/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:101/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:96/4096MB 149% | immich-postgres:90/768MB 0%
|
||||
16:21:47Z used=3638MB avail=458MB load1=1.04 cpu=6% immich-server:780/3072MB 2% | paperless-webserver:550/1280MB 0% | mealie:218/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:101/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:96/4096MB 149% | immich-postgres:90/768MB 0%
|
||||
16:22:17Z used=3638MB avail=458MB load1=0.96 cpu=9% immich-server:781/3072MB 1% | paperless-webserver:551/1280MB 0% | mealie:254/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:101/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:96/4096MB 23% | immich-postgres:90/768MB 0%
|
||||
16:22:47Z used=3638MB avail=458MB load1=0.8 cpu=6% immich-server:781/3072MB 1% | paperless-webserver:551/1280MB 0% | mealie:254/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:101/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:96/4096MB 23% | immich-postgres:90/768MB 0%
|
||||
16:23:17Z used=3638MB avail=458MB load1=0.71 cpu=9% immich-server:781/3072MB 1% | paperless-webserver:551/1280MB 0% | mealie:254/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:102/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:95/4096MB 6% | immich-postgres:90/768MB 0%
|
||||
16:23:47Z used=3638MB avail=458MB load1=0.65 cpu=7% immich-server:781/3072MB 1% | paperless-webserver:551/1280MB 0% | mealie:254/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:102/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:95/4096MB 6% | immich-postgres:90/768MB 0%
|
||||
16:24:18Z used=3638MB avail=458MB load1=0.58 cpu=9% immich-server:783/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:254/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:101/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:96/4096MB 21% | immich-postgres:90/768MB 0%
|
||||
16:24:48Z used=3638MB avail=458MB load1=0.35 cpu=7% immich-server:783/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:254/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:101/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:96/4096MB 21% | immich-postgres:90/768MB 0%
|
||||
16:25:18Z used=3638MB avail=458MB load1=0.29 cpu=8% immich-server:782/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:254/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:101/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:95/4096MB 6% | immich-postgres:90/768MB 0%
|
||||
16:25:48Z used=3638MB avail=458MB load1=0.4 cpu=6% immich-server:782/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:254/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:101/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:95/4096MB 6% | immich-postgres:90/768MB 0%
|
||||
16:26:18Z used=3638MB avail=458MB load1=0.78 cpu=8% immich-server:782/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:254/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:101/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:96/4096MB 145% | immich-postgres:90/768MB 0%
|
||||
16:26:48Z used=3638MB avail=458MB load1=0.55 cpu=5% immich-server:782/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:254/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:101/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:96/4096MB 145% | immich-postgres:90/768MB 0%
|
||||
16:27:18Z used=3638MB avail=458MB load1=0.47 cpu=8% immich-server:783/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:254/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:102/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:96/4096MB 12% | immich-postgres:90/768MB 0%
|
||||
16:27:48Z used=3638MB avail=458MB load1=0.48 cpu=6% immich-server:783/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:254/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:102/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:96/4096MB 12% | immich-postgres:90/768MB 0%
|
||||
16:28:18Z used=3638MB avail=458MB load1=0.43 cpu=9% immich-server:785/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:254/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:101/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:96/4096MB 23% | immich-postgres:90/768MB 0%
|
||||
16:28:49Z used=3638MB avail=458MB load1=0.26 cpu=6% immich-server:785/3072MB 0% | paperless-webserver:550/1280MB 0% | mealie:254/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:101/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:96/4096MB 23% | immich-postgres:90/768MB 0%
|
||||
16:29:19Z used=3638MB avail=458MB load1=0.99 cpu=32% immich-server:784/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:254/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:101/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:97/4096MB 6% | immich-postgres:90/768MB 0%
|
||||
16:29:49Z used=3638MB avail=458MB load1=2.24 cpu=21% immich-server:784/3072MB 0% | paperless-webserver:551/1280MB 0% | mealie:254/1000MB 0% | immich-machine-learning:137/2048MB 0% | nextcloud-db:101/512MB 0% | bookstack-db:97/256MB 0% | felhom-controller:97/4096MB 6% | immich-postgres:90/768MB 0%
|
||||
16:30:19Z used=3288MB avail=808MB load1=3.46 cpu=46% paperless-webserver:533/1280MB 0% | immich-server:502/3072MB 12% | immich-machine-learning:431/2048MB 1% | immich-postgres:132/768MB 3% | bookstack:131/512MB 0% | felhom-controller:116/4096MB 46% | mealie:103/1000MB 87% | bookstack-db:97/256MB 0%
|
||||
16:30:50Z used=3638MB avail=458MB load1=2.35 cpu=23% paperless-webserver:533/1280MB 0% | immich-server:502/3072MB 12% | immich-machine-learning:431/2048MB 1% | immich-postgres:132/768MB 3% | bookstack:131/512MB 0% | felhom-controller:116/4096MB 46% | mealie:103/1000MB 87% | bookstack-db:97/256MB 0%
|
||||
16:31:20Z used=3638MB avail=458MB load1=4.47 cpu=95% immich-server:908/3072MB 0% | mealie:244/1000MB 0% | immich-machine-learning:168/2048MB 0% | felhom-controller:166/4096MB 126% | nextcloud:162/1024MB 0% | nextcloud-db:131/512MB 0% | paperless-webserver:114/1280MB 66% | actualbudget:85/256MB 0%
|
||||
16:31:50Z used=3438MB avail=658MB load1=3.84 cpu=44% immich-server:908/3072MB 0% | mealie:244/1000MB 0% | immich-machine-learning:168/2048MB 0% | felhom-controller:166/4096MB 126% | nextcloud:162/1024MB 0% | nextcloud-db:131/512MB 0% | paperless-webserver:114/1280MB 66% | actualbudget:85/256MB 0%
|
||||
16:32:21Z used=3638MB avail=458MB load1=3.85 cpu=61% paperless-webserver:526/1280MB 0% | mealie:250/1000MB 0% | nextcloud-db:123/512MB 0% | bookstack-db:116/256MB 0% | felhom-controller:114/4096MB 84% | traefik:84/4096MB 2% | actualbudget:62/256MB 0% | nextcloud:50/1024MB 0%
|
||||
16:32:51Z used=3288MB avail=808MB load1=5.41 cpu=48% paperless-webserver:526/1280MB 0% | mealie:250/1000MB 0% | nextcloud-db:123/512MB 0% | bookstack-db:116/256MB 0% | felhom-controller:114/4096MB 84% | traefik:84/4096MB 2% | actualbudget:62/256MB 0% | nextcloud:50/1024MB 0%
|
||||
16:33:23Z used=3038MB avail=1058MB load1=4.16 cpu=48% immich-server:842/3072MB 5% | paperless-webserver:422/1280MB 0% | mealie:208/1000MB 0% | felhom-controller:96/4096MB 0% | immich-machine-learning:89/2048MB 0% | immich-postgres:79/768MB 0% | bookstack-db:74/256MB 0% | traefik:56/4096MB 26%
|
||||
16:33:53Z used=3638MB avail=458MB load1=3.01 cpu=11% immich-server:842/3072MB 5% | paperless-webserver:422/1280MB 0% | mealie:208/1000MB 0% | felhom-controller:96/4096MB 0% | immich-machine-learning:89/2048MB 0% | immich-postgres:79/768MB 0% | bookstack-db:74/256MB 0% | traefik:56/4096MB 26%
|
||||
16:34:23Z used=3638MB avail=458MB load1=2.18 cpu=33% immich-server:882/3072MB 4% | felhom-controller:227/4096MB 38% | mealie:208/1000MB 0% | nextcloud:170/1024MB 0% | nextcloud-db:134/512MB 0% | immich-machine-learning:97/2048MB 0% | paperless-webserver:91/1280MB 0% | actualbudget:87/256MB 0%
|
||||
16:34:54Z used=3638MB avail=458MB load1=2.12 cpu=8% immich-server:882/3072MB 4% | felhom-controller:227/4096MB 38% | mealie:208/1000MB 0% | nextcloud:170/1024MB 0% | nextcloud-db:134/512MB 0% | immich-machine-learning:97/2048MB 0% | paperless-webserver:91/1280MB 0% | actualbudget:87/256MB 0%
|
||||
16:35:25Z used=3638MB avail=458MB load1=1.42 cpu=7% immich-server:840/3072MB 7% | paperless-webserver:714/1280MB 0% | felhom-controller:273/4096MB 17% | mealie:217/1000MB 0% | nextcloud:177/1024MB 0% | nextcloud-db:123/512MB 7% | immich-machine-learning:102/2048MB 0% | actualbudget:96/256MB 0%
|
||||
16:35:55Z used=3638MB avail=458MB load1=1 cpu=11% immich-server:840/3072MB 7% | paperless-webserver:714/1280MB 0% | felhom-controller:273/4096MB 17% | mealie:217/1000MB 0% | nextcloud:177/1024MB 0% | nextcloud-db:123/512MB 7% | immich-machine-learning:102/2048MB 0% | actualbudget:96/256MB 0%
|
||||
16:36:26Z used=3638MB avail=458MB load1=1.33 cpu=7% immich-server:735/3072MB 6% | paperless-webserver:561/1280MB 0% | felhom-controller:281/4096MB 168% | mealie:218/1000MB 0% | nextcloud-db:110/512MB 6% | immich-machine-learning:103/2048MB 0% | actualbudget:96/256MB 0% | immich-postgres:94/768MB 0%
|
||||
16:36:56Z used=3638MB avail=458MB load1=0.8 cpu=8% immich-server:735/3072MB 6% | paperless-webserver:561/1280MB 0% | felhom-controller:281/4096MB 168% | mealie:218/1000MB 0% | nextcloud-db:110/512MB 6% | immich-machine-learning:103/2048MB 0% | actualbudget:96/256MB 0% | immich-postgres:94/768MB 0%
|
||||
16:37:26Z used=3638MB avail=458MB load1=0.55 cpu=8% immich-server:736/3072MB 5% | paperless-webserver:561/1280MB 0% | felhom-controller:250/4096MB 26% | mealie:218/1000MB 0% | nextcloud-db:110/512MB 8% | immich-machine-learning:103/2048MB 0% | actualbudget:96/256MB 0% | immich-postgres:94/768MB 0%
|
||||
16:37:58Z used=3638MB avail=458MB load1=0.62 cpu=7% immich-server:739/3072MB 0% | paperless-webserver:562/1280MB 0% | felhom-controller:251/4096MB 36% | mealie:237/1000MB 0% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:97/256MB 0% | immich-postgres:94/768MB 0%
|
||||
16:38:28Z used=3638MB avail=458MB load1=0.57 cpu=3% immich-server:739/3072MB 0% | paperless-webserver:562/1280MB 0% | felhom-controller:251/4096MB 36% | mealie:237/1000MB 0% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:97/256MB 0% | immich-postgres:94/768MB 0%
|
||||
16:38:59Z used=3638MB avail=458MB load1=0.35 cpu=5% immich-server:739/3072MB 0% | paperless-webserver:561/1280MB 0% | mealie:237/1000MB 0% | felhom-controller:224/4096MB 18% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 31% | actualbudget:97/256MB 0% | immich-postgres:94/768MB 0%
|
||||
16:39:29Z used=3638MB avail=458MB load1=0.36 cpu=3% immich-server:739/3072MB 0% | paperless-webserver:561/1280MB 0% | mealie:237/1000MB 0% | felhom-controller:224/4096MB 18% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 31% | actualbudget:97/256MB 0% | immich-postgres:94/768MB 0%
|
||||
16:40:00Z used=3638MB avail=458MB load1=0.38 cpu=4% immich-server:739/3072MB 0% | paperless-webserver:562/1280MB 0% | mealie:237/1000MB 26% | felhom-controller:225/4096MB 38% | immich-machine-learning:118/2048MB 27% | nextcloud-db:110/512MB 0% | actualbudget:97/256MB 0% | immich-postgres:94/768MB 0%
|
||||
16:40:30Z used=3638MB avail=458MB load1=0.76 cpu=10% immich-server:739/3072MB 0% | paperless-webserver:562/1280MB 0% | mealie:237/1000MB 26% | felhom-controller:225/4096MB 38% | immich-machine-learning:118/2048MB 27% | nextcloud-db:110/512MB 0% | actualbudget:97/256MB 0% | immich-postgres:94/768MB 0%
|
||||
16:41:02Z used=3638MB avail=458MB load1=1.27 cpu=19% immich-server:741/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:237/1000MB 25% | felhom-controller:224/4096MB 137% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:97/256MB 0% | immich-postgres:94/768MB 10%
|
||||
16:41:32Z used=3638MB avail=458MB load1=1.57 cpu=10% immich-server:741/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:237/1000MB 25% | felhom-controller:224/4096MB 137% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:97/256MB 0% | immich-postgres:94/768MB 10%
|
||||
16:42:03Z used=3638MB avail=458MB load1=1.02 cpu=15% immich-server:741/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:237/1000MB 0% | felhom-controller:224/4096MB 37% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:98/256MB 0% | immich-postgres:95/768MB 9%
|
||||
16:42:34Z used=3638MB avail=458MB load1=0.8 cpu=9% immich-server:741/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:237/1000MB 0% | felhom-controller:224/4096MB 37% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:98/256MB 0% | immich-postgres:95/768MB 9%
|
||||
16:43:04Z used=3638MB avail=458MB load1=0.64 cpu=14% immich-server:739/3072MB 3% | paperless-webserver:559/1280MB 0% | mealie:237/1000MB 0% | felhom-controller:224/4096MB 21% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:98/256MB 0% | immich-postgres:94/768MB 0%
|
||||
16:43:34Z used=3638MB avail=458MB load1=0.45 cpu=10% immich-server:739/3072MB 3% | paperless-webserver:559/1280MB 0% | mealie:237/1000MB 0% | felhom-controller:224/4096MB 21% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:98/256MB 0% | immich-postgres:94/768MB 0%
|
||||
16:44:04Z used=3638MB avail=458MB load1=0.35 cpu=5% immich-server:740/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:237/1000MB 0% | felhom-controller:224/4096MB 1% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:98/256MB 0% | immich-postgres:94/768MB 0%
|
||||
16:44:35Z used=3638MB avail=458MB load1=0.4 cpu=5% immich-server:740/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:237/1000MB 0% | felhom-controller:224/4096MB 1% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:98/256MB 0% | immich-postgres:94/768MB 0%
|
||||
16:45:06Z used=3638MB avail=458MB load1=0.24 cpu=5% immich-server:750/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:236/1000MB 0% | felhom-controller:224/4096MB 21% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:98/256MB 0% | immich-postgres:97/768MB 0%
|
||||
16:45:36Z used=3638MB avail=458MB load1=0.27 cpu=5% immich-server:750/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:236/1000MB 0% | felhom-controller:224/4096MB 21% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:98/256MB 0% | immich-postgres:97/768MB 0%
|
||||
16:46:06Z used=3638MB avail=458MB load1=1.22 cpu=14% immich-server:747/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:234/1000MB 0% | felhom-controller:233/4096MB 154% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:100/256MB 0% | immich-postgres:98/768MB 0%
|
||||
16:46:36Z used=3638MB avail=458MB load1=1.26 cpu=5% immich-server:747/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:234/1000MB 0% | felhom-controller:233/4096MB 154% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:100/256MB 0% | immich-postgres:98/768MB 0%
|
||||
16:47:06Z used=3638MB avail=458MB load1=0.94 cpu=5% immich-server:747/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:234/1000MB 0% | felhom-controller:234/4096MB 21% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:100/256MB 0% | immich-postgres:98/768MB 0%
|
||||
16:47:37Z used=3638MB avail=458MB load1=0.84 cpu=6% immich-server:747/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:234/1000MB 0% | felhom-controller:234/4096MB 21% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:100/256MB 0% | immich-postgres:98/768MB 0%
|
||||
16:48:09Z used=3638MB avail=458MB load1=0.64 cpu=8% immich-server:748/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:234/1000MB 0% | felhom-controller:175/4096MB 36% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:100/256MB 0% | immich-postgres:98/768MB 0%
|
||||
16:48:39Z used=3638MB avail=458MB load1=1.1 cpu=14% immich-server:748/3072MB 0% | paperless-webserver:559/1280MB 0% | mealie:234/1000MB 0% | felhom-controller:175/4096MB 36% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | actualbudget:100/256MB 0% | immich-postgres:98/768MB 0%
|
||||
16:49:10Z used=3638MB avail=458MB load1=0.78 cpu=6% immich-server:675/3072MB 0% | paperless-webserver:529/1280MB 0% | felhom-controller:270/4096MB 19% | mealie:220/1000MB 0% | actualbudget:119/256MB 0% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:49:40Z used=3638MB avail=458MB load1=0.65 cpu=4% immich-server:675/3072MB 0% | paperless-webserver:529/1280MB 0% | felhom-controller:270/4096MB 19% | mealie:220/1000MB 0% | actualbudget:119/256MB 0% | nextcloud-db:110/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:50:10Z used=3638MB avail=458MB load1=0.66 cpu=4% immich-server:694/3072MB 0% | paperless-webserver:538/1280MB 0% | felhom-controller:268/4096MB 51% | mealie:219/1000MB 0% | actualbudget:119/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:50:40Z used=3638MB avail=458MB load1=0.53 cpu=4% immich-server:694/3072MB 0% | paperless-webserver:538/1280MB 0% | felhom-controller:268/4096MB 51% | mealie:219/1000MB 0% | actualbudget:119/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:51:10Z used=3638MB avail=458MB load1=0.78 cpu=4% immich-server:700/3072MB 0% | paperless-webserver:537/1280MB 1% | felhom-controller:241/4096MB 160% | mealie:218/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:51:41Z used=3638MB avail=458MB load1=0.77 cpu=5% immich-server:700/3072MB 0% | paperless-webserver:537/1280MB 1% | felhom-controller:241/4096MB 160% | mealie:218/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:52:11Z used=3638MB avail=458MB load1=0.63 cpu=6% immich-server:701/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:242/4096MB 34% | mealie:218/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:52:42Z used=3638MB avail=458MB load1=0.41 cpu=5% immich-server:701/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:242/4096MB 34% | mealie:218/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:53:13Z used=3638MB avail=458MB load1=0.44 cpu=4% immich-server:703/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:235/4096MB 20% | mealie:218/1000MB 0% | actualbudget:119/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:53:43Z used=3638MB avail=458MB load1=0.47 cpu=6% immich-server:703/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:235/4096MB 20% | mealie:218/1000MB 0% | actualbudget:119/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:54:13Z used=3638MB avail=458MB load1=0.34 cpu=5% immich-server:704/3072MB 2% | paperless-webserver:537/1280MB 0% | felhom-controller:236/4096MB 35% | mealie:218/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:54:44Z used=3638MB avail=458MB load1=0.55 cpu=5% immich-server:704/3072MB 2% | paperless-webserver:537/1280MB 0% | felhom-controller:236/4096MB 35% | mealie:218/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:55:15Z used=3638MB avail=458MB load1=0.46 cpu=4% immich-server:705/3072MB 1% | paperless-webserver:537/1280MB 0% | felhom-controller:234/4096MB 19% | mealie:218/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:55:45Z used=3638MB avail=458MB load1=0.55 cpu=11% immich-server:705/3072MB 1% | paperless-webserver:537/1280MB 0% | felhom-controller:234/4096MB 19% | mealie:218/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:111/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:56:16Z used=3638MB avail=458MB load1=0.7 cpu=13% immich-server:710/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:232/4096MB 164% | mealie:217/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:112/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:56:47Z used=3638MB avail=458MB load1=0.42 cpu=7% immich-server:710/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:232/4096MB 164% | mealie:217/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:112/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:57:17Z used=3638MB avail=458MB load1=0.59 cpu=12% immich-server:710/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:231/4096MB 17% | mealie:217/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:112/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:57:47Z used=3638MB avail=458MB load1=0.48 cpu=6% immich-server:710/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:231/4096MB 17% | mealie:217/1000MB 0% | actualbudget:118/256MB 0% | nextcloud-db:112/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
16:58:18Z used=3638MB avail=458MB load1=0.42 cpu=12% immich-server:712/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:232/4096MB 38% | mealie:217/1000MB 0% | actualbudget:118/256MB 1% | nextcloud-db:112/512MB 6% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 9%
|
||||
16:58:50Z used=3638MB avail=458MB load1=0.72 cpu=7% immich-server:712/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:232/4096MB 38% | mealie:217/1000MB 0% | actualbudget:118/256MB 1% | nextcloud-db:112/512MB 6% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 9%
|
||||
16:59:20Z used=3638MB avail=458MB load1=0.63 cpu=45% immich-server:713/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:233/4096MB 24% | mealie:217/1000MB 0% | actualbudget:118/256MB 12% | nextcloud-db:113/512MB 5% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 8%
|
||||
16:59:50Z used=3638MB avail=458MB load1=0.56 cpu=10% immich-server:713/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:233/4096MB 24% | mealie:217/1000MB 0% | actualbudget:118/256MB 12% | nextcloud-db:113/512MB 5% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 8%
|
||||
17:00:21Z used=3638MB avail=458MB load1=0.61 cpu=30% immich-server:713/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:232/4096MB 37% | mealie:217/1000MB 0% | actualbudget:118/256MB 11% | nextcloud-db:112/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
17:00:51Z used=3638MB avail=458MB load1=0.72 cpu=17% immich-server:713/3072MB 0% | paperless-webserver:537/1280MB 0% | felhom-controller:232/4096MB 37% | mealie:217/1000MB 0% | actualbudget:118/256MB 11% | nextcloud-db:112/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
17:01:22Z used=3638MB avail=458MB load1=1.81 cpu=52% immich-server:720/3072MB 0% | paperless-webserver:538/1280MB 0% | felhom-controller:238/4096MB 138% | mealie:216/1000MB 0% | actualbudget:117/256MB 10% | nextcloud-db:112/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
17:01:52Z used=3638MB avail=458MB load1=1.42 cpu=25% immich-server:720/3072MB 0% | paperless-webserver:538/1280MB 0% | felhom-controller:238/4096MB 138% | mealie:216/1000MB 0% | actualbudget:117/256MB 10% | nextcloud-db:112/512MB 0% | immich-machine-learning:103/2048MB 0% | immich-postgres:102/768MB 0%
|
||||
17:02:22Z used=3638MB avail=458MB load1=0.93 cpu=29% immich-server:548/3072MB 0% | paperless-webserver:496/1280MB 0% | mealie:174/1000MB 0% | felhom-controller:170/4096MB 0% | nextcloud-db:99/512MB 0% | immich-machine-learning:88/2048MB 0% | nextcloud:70/1024MB 0% | immich-postgres:67/768MB 0%
|
||||
17:02:52Z used=3638MB avail=458MB load1=0.56 cpu=10% immich-server:548/3072MB 0% | paperless-webserver:496/1280MB 0% | mealie:174/1000MB 0% | felhom-controller:170/4096MB 0% | nextcloud-db:99/512MB 0% | immich-machine-learning:88/2048MB 0% | nextcloud:70/1024MB 0% | immich-postgres:67/768MB 0%
|
||||
17:03:22Z used=3638MB avail=458MB load1=0.41 cpu=27% immich-server:555/3072MB 0% | paperless-webserver:496/1280MB 0% | mealie:177/1000MB 0% | felhom-controller:169/4096MB 17% | nextcloud-db:99/512MB 0% | immich-machine-learning:88/2048MB 0% | nextcloud:71/1024MB 0% | immich-postgres:67/768MB 0%
|
||||
17:03:52Z used=3638MB avail=458MB load1=0.42 cpu=10% immich-server:555/3072MB 0% | paperless-webserver:496/1280MB 0% | mealie:177/1000MB 0% | felhom-controller:169/4096MB 17% | nextcloud-db:99/512MB 0% | immich-machine-learning:88/2048MB 0% | nextcloud:71/1024MB 0% | immich-postgres:67/768MB 0%
|
||||
17:04:22Z used=3638MB avail=458MB load1=0.8 cpu=65% immich-server:556/3072MB 1% | paperless-webserver:496/1280MB 0% | mealie:177/1000MB 0% | felhom-controller:169/4096MB 48% | nextcloud-db:99/512MB 0% | immich-machine-learning:88/2048MB 0% | nextcloud:71/1024MB 0% | immich-postgres:67/768MB 0%
|
||||
17:04:53Z used=1590MB avail=2506MB load1=3.55 cpu=20% immich-server:556/3072MB 1% | paperless-webserver:496/1280MB 0% | mealie:177/1000MB 0% | felhom-controller:169/4096MB 48% | nextcloud-db:99/512MB 0% | immich-machine-learning:88/2048MB 0% | nextcloud:71/1024MB 0% | immich-postgres:67/768MB 0%
|
||||
17:05:23Z used=3288MB avail=808MB load1=4.55 cpu=76% paperless-webserver:498/1280MB 0% | immich-machine-learning:431/2048MB 0% | felhom-controller:190/4096MB 23% | mealie:167/1000MB 0% | bookstack:132/512MB 6% | immich-postgres:129/768MB 0% | traefik:127/4096MB 2% | bookstack-db:118/256MB 0%
|
||||
17:05:53Z used=3038MB avail=1058MB load1=3.51 cpu=28% paperless-webserver:498/1280MB 0% | immich-machine-learning:431/2048MB 0% | felhom-controller:190/4096MB 23% | mealie:167/1000MB 0% | bookstack:132/512MB 6% | immich-postgres:129/768MB 0% | traefik:127/4096MB 2% | bookstack-db:118/256MB 0%
|
||||
17:06:23Z used=3638MB avail=458MB load1=4.16 cpu=76% immich-server:891/3072MB 0% | mealie:237/1000MB 0% | nextcloud:161/1024MB 4% | nextcloud-db:125/512MB 0% | immich-machine-learning:118/2048MB 23% | actualbudget:112/256MB 0% | felhom-controller:92/4096MB 147% | immich-postgres:81/768MB 8%
|
||||
17:06:53Z used=3638MB avail=458MB load1=4.32 cpu=50% immich-server:891/3072MB 0% | mealie:237/1000MB 0% | nextcloud:161/1024MB 4% | nextcloud-db:125/512MB 0% | immich-machine-learning:118/2048MB 23% | actualbudget:112/256MB 0% | felhom-controller:92/4096MB 147% | immich-postgres:81/768MB 8%
|
||||
17:07:23Z used=3638MB avail=458MB load1=3.32 cpu=43% immich-server:867/3072MB 0% | paperless-webserver:540/1280MB 1% | mealie:232/1000MB 0% | felhom-controller:185/4096MB 193% | nextcloud-db:112/512MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:102/768MB 0% | bookstack-db:66/256MB 0%
|
||||
17:07:53Z used=3638MB avail=458MB load1=2.39 cpu=16% immich-server:867/3072MB 0% | paperless-webserver:540/1280MB 1% | mealie:232/1000MB 0% | felhom-controller:185/4096MB 193% | nextcloud-db:112/512MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:102/768MB 0% | bookstack-db:66/256MB 0%
|
||||
17:08:24Z used=3638MB avail=458MB load1=1.63 cpu=27% immich-server:682/3072MB 2% | paperless-webserver:497/1280MB 0% | mealie:212/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:102/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:84/256MB 0% | felhom-controller:79/4096MB 38%
|
||||
17:08:54Z used=3638MB avail=458MB load1=0.99 cpu=11% immich-server:682/3072MB 2% | paperless-webserver:497/1280MB 0% | mealie:212/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:102/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:84/256MB 0% | felhom-controller:79/4096MB 38%
|
||||
17:09:24Z used=3638MB avail=458MB load1=0.67 cpu=28% immich-server:663/3072MB 2% | paperless-webserver:497/1280MB 0% | mealie:212/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:102/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:85/256MB 0% | felhom-controller:78/4096MB 16%
|
||||
17:09:54Z used=3638MB avail=458MB load1=0.56 cpu=9% immich-server:663/3072MB 2% | paperless-webserver:497/1280MB 0% | mealie:212/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:102/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:85/256MB 0% | felhom-controller:78/4096MB 16%
|
||||
17:10:24Z used=3638MB avail=458MB load1=0.67 cpu=28% immich-server:663/3072MB 0% | paperless-webserver:496/1280MB 0% | mealie:212/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:102/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:85/256MB 0% | felhom-controller:79/4096MB 32%
|
||||
17:10:54Z used=3638MB avail=458MB load1=0.48 cpu=16% immich-server:663/3072MB 0% | paperless-webserver:496/1280MB 0% | mealie:212/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:102/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:85/256MB 0% | felhom-controller:79/4096MB 32%
|
||||
17:11:24Z used=3638MB avail=458MB load1=0.55 cpu=28% immich-server:663/3072MB 0% | paperless-webserver:494/1280MB 0% | mealie:241/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:101/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:86/256MB 0% | felhom-controller:77/4096MB 138%
|
||||
17:11:54Z used=3638MB avail=458MB load1=0.63 cpu=9% immich-server:663/3072MB 0% | paperless-webserver:494/1280MB 0% | mealie:241/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:101/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:86/256MB 0% | felhom-controller:77/4096MB 138%
|
||||
17:12:25Z used=3638MB avail=458MB load1=0.66 cpu=27% immich-server:663/3072MB 0% | paperless-webserver:495/1280MB 0% | mealie:241/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:101/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:86/256MB 7% | felhom-controller:77/4096MB 32%
|
||||
17:12:55Z used=3638MB avail=458MB load1=0.4 cpu=8% immich-server:663/3072MB 0% | paperless-webserver:495/1280MB 0% | mealie:241/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:101/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:86/256MB 7% | felhom-controller:77/4096MB 32%
|
||||
17:13:25Z used=3638MB avail=458MB load1=0.38 cpu=27% immich-server:664/3072MB 0% | paperless-webserver:494/1280MB 0% | mealie:241/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:101/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:86/256MB 6% | felhom-controller:77/4096MB 16%
|
||||
17:13:55Z used=3638MB avail=458MB load1=0.84 cpu=9% immich-server:664/3072MB 0% | paperless-webserver:494/1280MB 0% | mealie:241/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:101/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:86/256MB 6% | felhom-controller:77/4096MB 16%
|
||||
17:14:25Z used=3638MB avail=458MB load1=0.88 cpu=47% immich-server:665/3072MB 0% | paperless-webserver:494/1280MB 0% | mealie:241/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:101/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:86/256MB 7% | felhom-controller:77/4096MB 40%
|
||||
17:14:55Z used=3638MB avail=458MB load1=0.68 cpu=13% immich-server:665/3072MB 0% | paperless-webserver:494/1280MB 0% | mealie:241/1000MB 0% | immich-machine-learning:108/2048MB 0% | immich-postgres:101/768MB 0% | nextcloud-db:100/512MB 0% | bookstack-db:86/256MB 7% | felhom-controller:77/4096MB 40%
|
||||
17:15:25Z used=3638MB avail=458MB load1=0.6 cpu=4% immich-server:660/3072MB 0% | paperless-webserver:493/1280MB 0% | mealie:110/1000MB 94% | immich-machine-learning:109/2048MB 0% | immich-postgres:100/768MB 0% | nextcloud-db:99/512MB 6% | bookstack-db:85/256MB 0% | felhom-controller:78/4096MB 40%
|
||||
17:15:55Z used=3638MB avail=458MB load1=0.68 cpu=12% immich-server:660/3072MB 0% | paperless-webserver:493/1280MB 0% | mealie:110/1000MB 94% | immich-machine-learning:109/2048MB 0% | immich-postgres:100/768MB 0% | nextcloud-db:99/512MB 6% | bookstack-db:85/256MB 0% | felhom-controller:78/4096MB 40%
|
||||
17:16:25Z used=3638MB avail=458MB load1=2.61 cpu=26% immich-server:658/3072MB 0% | paperless-webserver:492/1280MB 0% | mealie:293/1000MB 1% | immich-machine-learning:109/2048MB 0% | immich-postgres:100/768MB 0% | nextcloud-db:98/512MB 7% | felhom-controller:88/4096MB 158% | bookstack-db:84/256MB 0%
|
||||
17:16:56Z used=3638MB avail=458MB load1=3.83 cpu=14% immich-server:658/3072MB 0% | paperless-webserver:492/1280MB 0% | mealie:293/1000MB 1% | immich-machine-learning:109/2048MB 0% | immich-postgres:100/768MB 0% | nextcloud-db:98/512MB 7% | felhom-controller:88/4096MB 158% | bookstack-db:84/256MB 0%
|
||||
17:17:27Z used=3638MB avail=458MB load1=2.51 cpu=6% felhom-controller:952/4096MB 22% | immich-server:550/3072MB 0% | paperless-webserver:436/1280MB 0% | mealie:214/1000MB 0% | immich-machine-learning:108/2048MB 0% | nextcloud-db:78/512MB 0% | immich-postgres:62/768MB 0% | nextcloud:61/1024MB 0%
|
||||
17:17:57Z used=3638MB avail=458MB load1=1.67 cpu=8% felhom-controller:951/4096MB 39% | immich-server:596/3072MB 1% | paperless-webserver:436/1280MB 0% | mealie:215/1000MB 0% | immich-machine-learning:117/2048MB 0% | traefik:100/4096MB 0% | nextcloud-db:77/512MB 0% | immich-postgres:73/768MB 0%
|
||||
17:18:27Z used=3638MB avail=458MB load1=1.21 cpu=6% felhom-controller:951/4096MB 39% | immich-server:596/3072MB 1% | paperless-webserver:436/1280MB 0% | mealie:215/1000MB 0% | immich-machine-learning:117/2048MB 0% | traefik:100/4096MB 0% | nextcloud-db:77/512MB 0% | immich-postgres:73/768MB 0%
|
||||
17:18:57Z used=3638MB avail=458MB load1=0.86 cpu=9% felhom-controller:950/4096MB 17% | immich-server:597/3072MB 1% | paperless-webserver:437/1280MB 0% | mealie:215/1000MB 0% | immich-machine-learning:117/2048MB 0% | traefik:100/4096MB 0% | nextcloud-db:77/512MB 0% | immich-postgres:73/768MB 0%
|
||||
17:19:27Z used=3638MB avail=458MB load1=0.72 cpu=6% felhom-controller:950/4096MB 17% | immich-server:597/3072MB 1% | paperless-webserver:437/1280MB 0% | mealie:215/1000MB 0% | immich-machine-learning:117/2048MB 0% | traefik:100/4096MB 0% | nextcloud-db:77/512MB 0% | immich-postgres:73/768MB 0%
|
||||
17:19:58Z used=3638MB avail=458MB load1=0.79 cpu=29% felhom-controller:967/4096MB 41% | immich-server:600/3072MB 0% | paperless-webserver:438/1280MB 0% | mealie:215/1000MB 0% | traefik:142/4096MB 0% | nextcloud-db:123/512MB 6% | immich-machine-learning:117/2048MB 0% | immich-postgres:73/768MB 0%
|
||||
17:20:28Z used=3638MB avail=458MB load1=0.74 cpu=17% felhom-controller:967/4096MB 41% | immich-server:600/3072MB 0% | paperless-webserver:438/1280MB 0% | mealie:215/1000MB 0% | traefik:142/4096MB 0% | nextcloud-db:123/512MB 6% | immich-machine-learning:117/2048MB 0% | immich-postgres:73/768MB 0%
|
||||
17:20:58Z used=3638MB avail=458MB load1=0.83 cpu=10% felhom-controller:946/4096MB 146% | immich-server:607/3072MB 1% | paperless-webserver:459/1280MB 0% | mealie:247/1000MB 0% | nextcloud:152/1024MB 0% | nextcloud-db:121/512MB 0% | immich-machine-learning:120/2048MB 0% | immich-postgres:71/768MB 9%
|
||||
17:21:28Z used=3638MB avail=458MB load1=0.64 cpu=4% felhom-controller:946/4096MB 146% | immich-server:607/3072MB 1% | paperless-webserver:459/1280MB 0% | mealie:247/1000MB 0% | nextcloud:152/1024MB 0% | nextcloud-db:121/512MB 0% | immich-machine-learning:120/2048MB 0% | immich-postgres:71/768MB 9%
|
||||
17:21:58Z used=3438MB avail=658MB load1=1.36 cpu=31% felhom-controller:946/4096MB 146% | immich-server:607/3072MB 1% | paperless-webserver:459/1280MB 0% | mealie:247/1000MB 0% | nextcloud:152/1024MB 0% | nextcloud-db:121/512MB 0% | immich-machine-learning:120/2048MB 0% | immich-postgres:71/768MB 9%
|
||||
17:22:28Z used=3638MB avail=458MB load1=1.95 cpu=20% immich-server:532/3072MB 0% | paperless-webserver:449/1280MB 0% | mealie:219/1000MB 0% | nextcloud-db:132/512MB 4% | immich-postgres:129/768MB 10% | bookstack-db:113/256MB 101% | felhom-controller:105/4096MB 57% | immich-machine-learning:91/2048MB 0%
|
||||
17:22:58Z used=3288MB avail=808MB load1=2.14 cpu=25% immich-server:903/3072MB 0% | paperless-webserver:317/1280MB 0% | mealie:232/1000MB 22% | immich-machine-learning:168/2048MB 0% | felhom-controller:99/4096MB 17% | immich-postgres:81/768MB 0% | actualbudget:79/256MB 0% | traefik:70/4096MB 0%
|
||||
17:23:28Z used=3638MB avail=458MB load1=1.82 cpu=6% immich-server:903/3072MB 0% | paperless-webserver:317/1280MB 0% | mealie:232/1000MB 22% | immich-machine-learning:168/2048MB 0% | felhom-controller:99/4096MB 17% | immich-postgres:81/768MB 0% | actualbudget:79/256MB 0% | traefik:70/4096MB 0%
|
||||
17:23:59Z used=3638MB avail=458MB load1=2.37 cpu=46% immich-server:979/3072MB 0% | paperless-webserver:249/1280MB 180% | mealie:249/1000MB 12% | immich-machine-learning:168/2048MB 30% | felhom-controller:126/4096MB 63% | nextcloud-db:118/512MB 0% | actualbudget:115/256MB 0% | immich-postgres:103/768MB 0%
|
||||
17:24:29Z used=3638MB avail=458MB load1=2.35 cpu=9% immich-server:979/3072MB 0% | paperless-webserver:249/1280MB 180% | mealie:249/1000MB 12% | immich-machine-learning:168/2048MB 30% | felhom-controller:126/4096MB 63% | nextcloud-db:118/512MB 0% | actualbudget:115/256MB 0% | immich-postgres:103/768MB 0%
|
||||
17:24:59Z used=3638MB avail=458MB load1=3.64 cpu=25% immich-server:806/3072MB 0% | paperless-webserver:590/1280MB 0% | mealie:181/1000MB 0% | felhom-controller:173/4096MB 158% | immich-machine-learning:156/2048MB 21% | nextcloud-db:95/512MB 0% | immich-postgres:70/768MB 0% | bookstack:68/512MB 0%
|
||||
17:25:29Z used=3638MB avail=458MB load1=2.34 cpu=3% immich-server:806/3072MB 0% | paperless-webserver:590/1280MB 0% | mealie:181/1000MB 0% | felhom-controller:173/4096MB 158% | immich-machine-learning:156/2048MB 21% | nextcloud-db:95/512MB 0% | immich-postgres:70/768MB 0% | bookstack:68/512MB 0%
|
||||
17:25:59Z used=3638MB avail=458MB load1=1.94 cpu=11% immich-server:808/3072MB 1% | paperless-webserver:590/1280MB 0% | mealie:187/1000MB 0% | immich-machine-learning:142/2048MB 0% | felhom-controller:135/4096MB 0% | nextcloud-db:95/512MB 0% | immich-postgres:79/768MB 0% | nextcloud:71/1024MB 0%
|
||||
17:26:29Z used=3638MB avail=458MB load1=1.44 cpu=11% immich-server:808/3072MB 1% | paperless-webserver:590/1280MB 0% | mealie:187/1000MB 0% | immich-machine-learning:142/2048MB 0% | felhom-controller:135/4096MB 0% | nextcloud-db:95/512MB 0% | immich-postgres:79/768MB 0% | nextcloud:71/1024MB 0%
|
||||
17:26:59Z used=3638MB avail=458MB load1=1.75 cpu=5% immich-server:732/3072MB 0% | paperless-webserver:590/1280MB 0% | felhom-controller:216/4096MB 17% | mealie:188/1000MB 0% | immich-machine-learning:142/2048MB 0% | nextcloud-db:95/512MB 0% | immich-postgres:79/768MB 0% | nextcloud:72/1024MB 0%
|
||||
17:27:30Z used=3638MB avail=458MB load1=1.28 cpu=2% immich-server:732/3072MB 0% | paperless-webserver:590/1280MB 0% | felhom-controller:216/4096MB 17% | mealie:188/1000MB 0% | immich-machine-learning:142/2048MB 0% | nextcloud-db:95/512MB 0% | immich-postgres:79/768MB 0% | nextcloud:72/1024MB 0%
|
||||
17:28:00Z used=3638MB avail=458MB load1=2.24 cpu=17% immich-server:731/3072MB 1% | paperless-webserver:590/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:173/4096MB 76% | immich-machine-learning:142/2048MB 0% | nextcloud-db:95/512MB 0% | bookstack-db:81/256MB 0% | bookstack:81/512MB 0%
|
||||
17:28:30Z NO ANSWER TimeoutExpired
|
||||
17:29:25Z used=NoneMB avail=NoneMB load1=None cpu=0%
|
||||
17:30:00Z used=NoneMB avail=NoneMB load1=None cpu=0%
|
||||
17:30:36Z used=NoneMB avail=NoneMB load1=None cpu=0%
|
||||
17:31:13Z used=1590MB avail=2506MB load1=4.44 cpu=0% mealie:222/1000MB 57% | traefik:186/4096MB 23% | nextcloud:170/1024MB 0% | felhom-controller:167/4096MB 48% | nextcloud-db:166/512MB 0% | actualbudget:159/256MB 0% | paperless-webserver:156/1280MB 63% | filebrowser:145/256MB 0%
|
||||
17:31:43Z used=3638MB avail=458MB load1=5.72 cpu=76% mealie:222/1000MB 57% | traefik:186/4096MB 23% | nextcloud:170/1024MB 0% | felhom-controller:167/4096MB 48% | nextcloud-db:166/512MB 0% | actualbudget:159/256MB 0% | paperless-webserver:156/1280MB 63% | filebrowser:145/256MB 0%
|
||||
17:32:13Z used=3638MB avail=458MB load1=3.96 cpu=46% immich-server:896/3072MB 1% | mealie:216/1000MB 0% | immich-machine-learning:163/2048MB 0% | paperless-webserver:154/1280MB 195% | nextcloud-db:121/512MB 0% | felhom-controller:92/4096MB 0% | bookstack-db:91/256MB 0% | immich-postgres:79/768MB 0%
|
||||
17:32:43Z used=3638MB avail=458MB load1=3.33 cpu=62% immich-server:896/3072MB 1% | mealie:216/1000MB 0% | immich-machine-learning:163/2048MB 0% | paperless-webserver:154/1280MB 195% | nextcloud-db:121/512MB 0% | felhom-controller:92/4096MB 0% | bookstack-db:91/256MB 0% | immich-postgres:79/768MB 0%
|
||||
17:33:13Z used=3638MB avail=458MB load1=2.15 cpu=8% immich-server:887/3072MB 1% | paperless-webserver:628/1280MB 5% | mealie:216/1000MB 0% | immich-machine-learning:158/2048MB 0% | nextcloud-db:113/512MB 0% | felhom-controller:95/4096MB 1% | bookstack-db:88/256MB 0% | immich-postgres:79/768MB 0%
|
||||
17:33:43Z NO ANSWER TimeoutExpired
|
||||
17:34:38Z used=NoneMB avail=NoneMB load1=None cpu=0%
|
||||
17:35:15Z used=NoneMB avail=NoneMB load1=None cpu=0%
|
||||
17:35:50Z used=NoneMB avail=NoneMB load1=None cpu=0%
|
||||
17:36:24Z used=3638MB avail=458MB load1=3.94 cpu=78% traefik:187/4096MB 23% | nextcloud:170/1024MB 0% | actualbudget:160/256MB 15% | mealie:159/1000MB 53% | nextcloud-db:153/512MB 0% | filebrowser:145/256MB 0% | immich-postgres:128/768MB 1% | paperless-webserver:122/1280MB 53%
|
||||
17:36:54Z used=3638MB avail=458MB load1=3.48 cpu=7% traefik:187/4096MB 23% | nextcloud:170/1024MB 0% | actualbudget:160/256MB 15% | mealie:159/1000MB 53% | nextcloud-db:153/512MB 0% | filebrowser:145/256MB 0% | immich-postgres:128/768MB 1% | paperless-webserver:122/1280MB 53%
|
||||
17:37:25Z used=3638MB avail=458MB load1=3.33 cpu=60% immich-server:1048/3072MB 0% | mealie:332/1000MB 0% | immich-machine-learning:167/2048MB 0% | paperless-webserver:150/1280MB 100% | nextcloud-db:137/512MB 0% | bookstack:133/512MB 0% | felhom-controller:128/4096MB 5% | nextcloud:116/1024MB 0%
|
||||
17:37:55Z used=3638MB avail=458MB load1=2.46 cpu=6% immich-server:1048/3072MB 0% | mealie:332/1000MB 0% | immich-machine-learning:167/2048MB 0% | paperless-webserver:150/1280MB 100% | nextcloud-db:137/512MB 0% | bookstack:133/512MB 0% | felhom-controller:128/4096MB 5% | nextcloud:116/1024MB 0%
|
||||
17:38:25Z used=3638MB avail=458MB load1=2.9 cpu=14% immich-server:914/3072MB 3% | paperless-webserver:541/1280MB 0% | mealie:216/1000MB 0% | immich-machine-learning:154/2048MB 0% | nextcloud-db:97/512MB 0% | bookstack-db:96/256MB 0% | felhom-controller:82/4096MB 24% | immich-postgres:65/768MB 9%
|
||||
17:38:55Z used=3638MB avail=458MB load1=2.13 cpu=11% immich-server:914/3072MB 3% | paperless-webserver:541/1280MB 0% | mealie:216/1000MB 0% | immich-machine-learning:154/2048MB 0% | nextcloud-db:97/512MB 0% | bookstack-db:96/256MB 0% | felhom-controller:82/4096MB 24% | immich-postgres:65/768MB 9%
|
||||
17:39:25Z used=3638MB avail=458MB load1=1.37 cpu=30% immich-server:717/3072MB 1% | paperless-webserver:518/1280MB 0% | mealie:216/1000MB 0% | immich-machine-learning:152/2048MB 0% | felhom-controller:135/4096MB 6% | bookstack-db:112/256MB 0% | nextcloud-db:94/512MB 0% | bookstack:91/512MB 0%
|
||||
17:39:56Z used=3638MB avail=458MB load1=1.41 cpu=24% immich-server:717/3072MB 1% | paperless-webserver:518/1280MB 0% | mealie:216/1000MB 0% | immich-machine-learning:152/2048MB 0% | felhom-controller:135/4096MB 6% | bookstack-db:112/256MB 0% | nextcloud-db:94/512MB 0% | bookstack:91/512MB 0%
|
||||
17:40:26Z used=3638MB avail=458MB load1=1.82 cpu=33% immich-server:584/3072MB 1% | paperless-webserver:426/1280MB 0% | mealie:172/1000MB 0% | felhom-controller:149/4096MB 5% | immich-machine-learning:128/2048MB 0% | nextcloud:104/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:87/512MB 0%
|
||||
17:40:56Z used=3638MB avail=458MB load1=1.37 cpu=22% immich-server:584/3072MB 1% | paperless-webserver:426/1280MB 0% | mealie:172/1000MB 0% | felhom-controller:149/4096MB 5% | immich-machine-learning:128/2048MB 0% | nextcloud:104/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:87/512MB 0%
|
||||
17:41:26Z used=3638MB avail=458MB load1=1.09 cpu=32% immich-server:584/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:250/1000MB 0% | felhom-controller:152/4096MB 17% | immich-machine-learning:129/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0%
|
||||
17:41:56Z used=3638MB avail=458MB load1=0.8 cpu=29% immich-server:584/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:250/1000MB 0% | felhom-controller:152/4096MB 17% | immich-machine-learning:129/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0%
|
||||
17:42:26Z used=3638MB avail=458MB load1=0.84 cpu=29% immich-server:585/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:206/1000MB 0% | felhom-controller:153/4096MB 20% | immich-machine-learning:129/2048MB 0% | nextcloud:119/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0%
|
||||
17:42:56Z used=3638MB avail=458MB load1=0.58 cpu=9% immich-server:585/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:206/1000MB 0% | felhom-controller:153/4096MB 20% | immich-machine-learning:129/2048MB 0% | nextcloud:119/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0%
|
||||
17:43:27Z used=3638MB avail=458MB load1=0.43 cpu=30% immich-server:585/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:154/4096MB 1% | immich-machine-learning:129/2048MB 0% | nextcloud:117/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 5%
|
||||
17:43:57Z used=3638MB avail=458MB load1=0.48 cpu=9% immich-server:585/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:154/4096MB 1% | immich-machine-learning:129/2048MB 0% | nextcloud:117/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 5%
|
||||
17:44:27Z used=3638MB avail=458MB load1=0.65 cpu=31% immich-server:586/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:152/4096MB 1% | immich-machine-learning:129/2048MB 0% | nextcloud:117/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 5%
|
||||
17:44:57Z used=3638MB avail=458MB load1=0.45 cpu=9% immich-server:586/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:152/4096MB 1% | immich-machine-learning:129/2048MB 0% | nextcloud:117/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 5%
|
||||
17:45:27Z used=3638MB avail=458MB load1=0.35 cpu=30% immich-server:586/3072MB 0% | paperless-webserver:428/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:152/4096MB 0% | immich-machine-learning:129/2048MB 0% | nextcloud:117/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0%
|
||||
17:45:57Z used=3638MB avail=458MB load1=0.29 cpu=20% immich-server:586/3072MB 0% | paperless-webserver:428/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:152/4096MB 0% | immich-machine-learning:129/2048MB 0% | nextcloud:117/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0%
|
||||
17:46:27Z used=3638MB avail=458MB load1=0.5 cpu=30% immich-server:592/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:154/4096MB 3% | immich-machine-learning:129/2048MB 0% | nextcloud:119/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0%
|
||||
17:46:57Z used=3638MB avail=458MB load1=0.43 cpu=10% immich-server:592/3072MB 0% | paperless-webserver:427/1280MB 0% | mealie:220/1000MB 0% | felhom-controller:154/4096MB 3% | immich-machine-learning:129/2048MB 0% | nextcloud:119/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0%
|
||||
17:47:28Z used=3638MB avail=458MB load1=0.58 cpu=30% immich-server:598/3072MB 0% | paperless-webserver:428/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:154/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 6% | nextcloud-db:89/512MB 0%
|
||||
17:47:58Z used=3638MB avail=458MB load1=0.48 cpu=15% immich-server:598/3072MB 0% | paperless-webserver:428/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:154/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 6% | nextcloud-db:89/512MB 0%
|
||||
17:48:28Z used=3638MB avail=458MB load1=1.9 cpu=33% immich-server:599/3072MB 0% | paperless-webserver:427/1280MB 0% | felhom-controller:316/4096MB 154% | mealie:225/1000MB 0% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 6% | nextcloud-db:89/512MB 0%
|
||||
17:48:58Z used=3638MB avail=458MB load1=1.44 cpu=9% immich-server:599/3072MB 0% | paperless-webserver:427/1280MB 0% | felhom-controller:316/4096MB 154% | mealie:225/1000MB 0% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 6% | nextcloud-db:89/512MB 0%
|
||||
17:49:28Z used=3638MB avail=458MB load1=1.4 cpu=45% immich-server:599/3072MB 0% | paperless-webserver:428/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:156/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 5% | nextcloud-db:89/512MB 0%
|
||||
17:49:58Z used=3638MB avail=458MB load1=1.13 cpu=9% immich-server:599/3072MB 0% | paperless-webserver:428/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:156/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 5% | nextcloud-db:89/512MB 0%
|
||||
17:50:28Z used=3638MB avail=458MB load1=0.93 cpu=33% immich-server:600/3072MB 1% | paperless-webserver:427/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:196/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0%
|
||||
17:50:58Z used=3638MB avail=458MB load1=0.7 cpu=20% immich-server:600/3072MB 1% | paperless-webserver:427/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:196/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0%
|
||||
17:51:28Z used=3638MB avail=458MB load1=0.56 cpu=32% immich-server:673/3072MB 63% | paperless-webserver:428/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:192/4096MB 77% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0%
|
||||
17:51:59Z used=3638MB avail=458MB load1=0.48 cpu=10% immich-server:673/3072MB 63% | paperless-webserver:428/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:192/4096MB 77% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:91/256MB 0% | nextcloud-db:89/512MB 0%
|
||||
17:52:29Z used=3638MB avail=458MB load1=0.48 cpu=34% immich-server:641/3072MB 0% | paperless-webserver:428/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:89/512MB 0%
|
||||
17:52:59Z used=3638MB avail=458MB load1=0.55 cpu=9% immich-server:641/3072MB 0% | paperless-webserver:428/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:89/512MB 0%
|
||||
17:53:29Z used=3638MB avail=458MB load1=0.47 cpu=31% immich-server:641/3072MB 1% | paperless-webserver:429/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 0% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:89/512MB 0%
|
||||
17:53:59Z used=3638MB avail=458MB load1=0.67 cpu=20% immich-server:641/3072MB 1% | paperless-webserver:429/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 0% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:89/512MB 0%
|
||||
17:54:29Z used=3638MB avail=458MB load1=0.92 cpu=32% immich-server:641/3072MB 0% | paperless-webserver:429/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:90/512MB 0%
|
||||
17:54:59Z used=3638MB avail=458MB load1=0.67 cpu=10% immich-server:641/3072MB 0% | paperless-webserver:429/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:90/512MB 0%
|
||||
17:55:29Z used=3638MB avail=458MB load1=0.49 cpu=49% immich-server:641/3072MB 0% | paperless-webserver:429/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:90/512MB 0%
|
||||
17:55:59Z used=3638MB avail=458MB load1=0.5 cpu=22% immich-server:641/3072MB 0% | paperless-webserver:429/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 1% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:90/512MB 0%
|
||||
17:56:30Z used=3638MB avail=458MB load1=0.77 cpu=37% immich-server:641/3072MB 0% | paperless-webserver:429/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 22% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:90/512MB 0%
|
||||
17:57:00Z used=1240MB avail=2856MB load1=0.65 cpu=11% immich-server:641/3072MB 0% | paperless-webserver:429/1280MB 0% | mealie:225/1000MB 0% | felhom-controller:168/4096MB 22% | immich-machine-learning:130/2048MB 0% | nextcloud:121/1024MB 0% | bookstack-db:92/256MB 0% | nextcloud-db:90/512MB 0%
|
||||
17:57:30Z used=640MB avail=3456MB load1=0.67 cpu=29% mealie:225/1000MB 0% | felhom-controller:160/4096MB 1% | bookstack-db:93/256MB 0% | traefik:71/4096MB 0% | actualbudget:65/256MB 9% | bookstack:50/512MB 0% | felhom-samba:36/4096MB 0% | filebrowser:35/256MB 0%
|
||||
17:58:00Z used=640MB avail=3456MB load1=0.59 cpu=8% mealie:225/1000MB 0% | felhom-controller:160/4096MB 1% | bookstack-db:93/256MB 0% | traefik:71/4096MB 0% | actualbudget:65/256MB 9% | bookstack:50/512MB 0% | felhom-samba:36/4096MB 0% | filebrowser:35/256MB 0%
|
||||
17:58:30Z used=640MB avail=3456MB load1=0.5 cpu=29% mealie:225/1000MB 0% | felhom-controller:161/4096MB 1% | bookstack-db:93/256MB 0% | traefik:72/4096MB 0% | actualbudget:65/256MB 8% | bookstack:50/512MB 0% | felhom-samba:36/4096MB 0% | filebrowser:36/256MB 0%
|
||||
17:59:00Z used=640MB avail=3456MB load1=0.36 cpu=8% mealie:225/1000MB 0% | felhom-controller:161/4096MB 1% | bookstack-db:93/256MB 0% | traefik:72/4096MB 0% | actualbudget:65/256MB 8% | bookstack:50/512MB 0% | felhom-samba:36/4096MB 0% | filebrowser:36/256MB 0%
|
||||
17:59:30Z used=640MB avail=3456MB load1=0.35 cpu=30% mealie:225/1000MB 0% | felhom-controller:161/4096MB 0% | bookstack-db:93/256MB 0% | traefik:71/4096MB 1% | actualbudget:70/256MB 8% | bookstack:50/512MB 0% | filebrowser:36/256MB 0% | felhom-samba:36/4096MB 0%
|
||||
18:00:00Z used=640MB avail=3456MB load1=0.52 cpu=8% mealie:225/1000MB 0% | felhom-controller:161/4096MB 0% | bookstack-db:93/256MB 0% | traefik:71/4096MB 1% | actualbudget:70/256MB 8% | bookstack:50/512MB 0% | filebrowser:36/256MB 0% | felhom-samba:36/4096MB 0%
|
||||
18:00:30Z used=3638MB avail=458MB load1=1.51 cpu=65% immich-machine-learning:407/2048MB 0% | mealie:225/1000MB 0% | felhom-controller:183/4096MB 16% | immich-server:174/3072MB 97% | nextcloud-db:111/512MB 18% | bookstack-db:93/256MB 0% | actualbudget:80/256MB 8% | traefik:71/4096MB 0%
|
||||
18:01:01Z used=3638MB avail=458MB load1=2.16 cpu=79% immich-machine-learning:407/2048MB 0% | mealie:225/1000MB 0% | felhom-controller:183/4096MB 16% | immich-server:174/3072MB 97% | nextcloud-db:111/512MB 18% | bookstack-db:93/256MB 0% | actualbudget:80/256MB 8% | traefik:71/4096MB 0%
|
||||
18:01:31Z used=3638MB avail=458MB load1=1.66 cpu=9% immich-server:936/3072MB 0% | paperless-webserver:532/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:172/2048MB 0% | nextcloud-db:106/512MB 0% | actualbudget:103/256MB 0% | immich-postgres:82/768MB 0% | traefik:79/4096MB 3%
|
||||
18:02:01Z used=3638MB avail=458MB load1=1.3 cpu=7% immich-server:936/3072MB 0% | paperless-webserver:532/1280MB 0% | mealie:200/1000MB 0% | immich-machine-learning:172/2048MB 0% | nextcloud-db:106/512MB 0% | actualbudget:103/256MB 0% | immich-postgres:82/768MB 0% | traefik:79/4096MB 3%
|
||||
18:02:31Z used=3638MB avail=458MB load1=0.93 cpu=10% immich-server:750/3072MB 0% | paperless-webserver:536/1280MB 0% | mealie:211/1000MB 0% | immich-machine-learning:172/2048MB 0% | nextcloud-db:104/512MB 0% | felhom-controller:86/4096MB 0% | immich-postgres:84/768MB 0% | traefik:79/4096MB 0%
|
||||
18:03:01Z used=3638MB avail=458MB load1=0.76 cpu=22% immich-server:750/3072MB 0% | paperless-webserver:536/1280MB 0% | mealie:211/1000MB 0% | immich-machine-learning:172/2048MB 0% | nextcloud-db:104/512MB 0% | felhom-controller:86/4096MB 0% | immich-postgres:84/768MB 0% | traefik:79/4096MB 0%
|
||||
18:03:31Z used=3638MB avail=458MB load1=0.74 cpu=12% immich-server:752/3072MB 0% | paperless-webserver:536/1280MB 0% | mealie:211/1000MB 0% | immich-machine-learning:172/2048MB 0% | nextcloud:113/1024MB 2% | nextcloud-db:110/512MB 0% | felhom-controller:89/4096MB 0% | immich-postgres:84/768MB 0%
|
||||
18:04:01Z used=1240MB avail=2856MB load1=0.74 cpu=24% immich-server:752/3072MB 0% | paperless-webserver:536/1280MB 0% | mealie:211/1000MB 0% | immich-machine-learning:172/2048MB 0% | nextcloud:113/1024MB 2% | nextcloud-db:110/512MB 0% | felhom-controller:89/4096MB 0% | immich-postgres:84/768MB 0%
|
||||
18:04:31Z used=640MB avail=3456MB load1=0.66 cpu=13% mealie:212/1000MB 0% | traefik:96/4096MB 0% | felhom-controller:83/4096MB 1% | bookstack-db:63/256MB 0% | bookstack:53/512MB 0% | vaultwarden:44/256MB 0% | filebrowser:44/256MB 0% | actualbudget:41/256MB 0%
|
||||
18:05:02Z used=640MB avail=3456MB load1=0.45 cpu=4% mealie:212/1000MB 0% | traefik:96/4096MB 0% | felhom-controller:83/4096MB 1% | bookstack-db:63/256MB 0% | bookstack:53/512MB 0% | vaultwarden:44/256MB 0% | filebrowser:44/256MB 0% | actualbudget:41/256MB 0%
|
||||
18:05:32Z used=2688MB avail=1408MB load1=0.64 cpu=19% mealie:212/1000MB 0% | traefik:95/4096MB 0% | felhom-controller:83/4096MB 0% | bookstack-db:63/256MB 0% | bookstack:53/512MB 0% | vaultwarden:44/256MB 0% | filebrowser:44/256MB 0% | actualbudget:41/256MB 0%
|
||||
18:06:02Z used=3638MB avail=458MB load1=1.53 cpu=31% mealie:212/1000MB 0% | traefik:95/4096MB 0% | felhom-controller:83/4096MB 0% | bookstack-db:63/256MB 0% | bookstack:53/512MB 0% | vaultwarden:44/256MB 0% | filebrowser:44/256MB 0% | actualbudget:41/256MB 0%
|
||||
18:06:32Z used=3638MB avail=458MB load1=2.27 cpu=82% immich-server:1089/3072MB 0% | immich-machine-learning:271/2048MB 0% | mealie:212/1000MB 0% | paperless-webserver:143/1280MB 102% | nextcloud-db:119/512MB 0% | felhom-controller:112/4096MB 85% | immich-postgres:97/768MB 0% | traefik:95/4096MB 3%
|
||||
18:07:02Z used=3638MB avail=458MB load1=1.75 cpu=9% immich-server:1089/3072MB 0% | immich-machine-learning:271/2048MB 0% | mealie:212/1000MB 0% | paperless-webserver:143/1280MB 102% | nextcloud-db:119/512MB 0% | felhom-controller:112/4096MB 85% | imm
|
||||
@@ -0,0 +1,16 @@
|
||||
14:23:57
|
||||
15:20:50Z nextcloud rclone rc=1
|
||||
2026/10/20 14:23:59 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
|
||||
2026/10/20 14:23:59 NOTICE: Csalad/Kulonleges_nevek/link_a_nagy_fajlra: Can't follow symlink without -L/--copy-links
|
||||
2026/10/20 14:24:00 NOTICE: Csalad/Kulonleges_nevek/Ödön születésnapja – fénykép lista.txt: Duplicate object found in source - ignoring
|
||||
2026/10/20 14:24:04 ERROR : Csalad/nagy_fajl_1_5GB.bin: Failed to copy: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
|
||||
2026/10/20 15:20:25 ERROR : Attempt 1/3 failed with 1 errors and: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
|
||||
2026/10/20 15:20:26 NOTICE: Csalad/Kulonleges_nevek/link_a_nagy_fajlra: Can't follow symlink without -L/--copy-links
|
||||
2026/10/20 15:20:27 NOTICE: Csalad/Kulonleges_nevek/Ödön születésnapja – fénykép lista.txt: Duplicate object found in source - ignoring
|
||||
2026/10/20 15:20:34 ERROR : Csalad/nagy_fajl_1_5GB.bin: Failed to copy: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
|
||||
2026/10/20 15:20:38 ERROR : Attempt 2/3 failed with 1 errors and: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
|
||||
2026/10/20 15:20:39 NOTICE: Csalad/Kulonleges_nevek/link_a_nagy_fajlra: Can't follow symlink without -L/--copy-links
|
||||
2026/10/20 15:20:40 NOTICE: Csalad/Kulonleges_nevek/Ödön születésnapja – fénykép lista.txt: Duplicate object found in source - ignoring
|
||||
2026/10/20 15:20:45 ERROR : Csalad/nagy_fajl_1_5GB.bin: Failed to copy: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
|
||||
2026/10/20 15:20:50 ERROR : Attempt 3/3 failed with 1 errors and: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
|
||||
15:20:50
|
||||
@@ -0,0 +1,7 @@
|
||||
12:57:15
|
||||
13:12:06Z nextcloud rclone rc=-15
|
||||
2026/10/20 12:57:17 NOTICE: Config file "/root/.config/rclone/rclone.conf" not found - using defaults
|
||||
2026/10/20 12:57:18 NOTICE: Csalad/Kulonleges_nevek/link_a_nagy_fajlra: Can't follow symlink without -L/--copy-links
|
||||
2026/10/20 12:57:18 NOTICE: Csalad/Kulonleges_nevek/Ödön születésnapja – fénykép lista.txt: Duplicate object found in source - ignoring
|
||||
2026/10/20 12:57:27 ERROR : Csalad/nagy_fajl_1_5GB.bin: Failed to copy: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
|
||||
13:12:06
|
||||
@@ -0,0 +1,120 @@
|
||||
14:23:58 diskread=4GB health=200 0.010208
|
||||
14:24:30 diskread=4GB health=200 0.010427
|
||||
14:25:01 diskread=4GB health=200 0.012785
|
||||
14:25:32 diskread=4GB health=200 0.010109
|
||||
14:26:04 diskread=4GB health=200 0.009754
|
||||
14:26:35 diskread=4GB health=200 0.010561
|
||||
14:27:06 diskread=4GB health=200 0.013033
|
||||
14:27:37 diskread=4GB health=200 0.010545
|
||||
14:28:09 diskread=4GB health=200 0.010882
|
||||
14:28:40 diskread=4GB health=200 0.010998
|
||||
14:29:11 diskread=4GB health=200 0.009735
|
||||
14:29:42 diskread=4GB health=200 0.010706
|
||||
14:30:14 diskread=4GB health=200 0.012557
|
||||
14:30:45 diskread=4GB health=200 0.011448
|
||||
14:31:16 diskread=4GB health=200 0.010879
|
||||
14:31:47 diskread=4GB health=200 0.012360
|
||||
14:32:19 diskread=4GB health=200 0.009276
|
||||
14:32:50 diskread=4GB health=200 0.010233
|
||||
14:33:22 diskread=4GB health=200 0.013962
|
||||
14:33:53 diskread=4GB health=200 0.011101
|
||||
14:34:24 diskread=4GB health=200 0.012738
|
||||
14:34:55 diskread=4GB health=200 0.011132
|
||||
14:35:27 diskread=4GB health=200 0.017582
|
||||
14:35:58 diskread=4GB health=200 0.009786
|
||||
14:36:30 diskread=4GB health=200 0.011178
|
||||
14:37:01 diskread=4GB health=200 0.011017
|
||||
14:37:32 diskread=4GB health=200 0.009835
|
||||
14:38:03 diskread=4GB health=200 0.011396
|
||||
14:38:35 diskread=4GB health=200 0.011475
|
||||
14:39:06 diskread=4GB health=200 0.010567
|
||||
14:39:37 diskread=4GB health=200 0.010764
|
||||
14:40:09 diskread=4GB health=200 0.011841
|
||||
14:40:40 diskread=4GB health=200 0.012188
|
||||
14:41:11 diskread=4GB health=200 0.010514
|
||||
14:41:42 diskread=4GB health=200 0.010198
|
||||
14:42:14 diskread=4GB health=200 0.008837
|
||||
14:42:45 diskread=4GB health=200 0.013116
|
||||
14:43:16 diskread=4GB health=200 0.015796
|
||||
14:43:48 diskread=4GB health=200 0.011376
|
||||
14:44:19 diskread=4GB health=200 0.013047
|
||||
14:44:51 diskread=4GB health=200 0.011553
|
||||
14:45:22 diskread=4GB health=200 0.014702
|
||||
14:45:53 diskread=4GB health=200 0.011081
|
||||
14:46:24 diskread=4GB health=200 0.010798
|
||||
14:46:56 diskread=5GB health=200 0.016907
|
||||
14:47:27 diskread=5GB health=200 0.009167
|
||||
14:47:59 diskread=5GB health=200 0.011809
|
||||
14:48:30 diskread=5GB health=200 0.021601
|
||||
14:49:01 diskread=5GB health=200 0.012060
|
||||
14:49:33 diskread=5GB health=200 0.010723
|
||||
14:50:04 diskread=5GB health=200 0.010155
|
||||
14:50:35 diskread=5GB health=200 0.015786
|
||||
14:51:07 diskread=5GB health=200 0.010058
|
||||
14:51:38 diskread=5GB health=200 0.009848
|
||||
14:52:09 diskread=5GB health=200 0.009929
|
||||
14:52:41 diskread=5GB health=200 0.010351
|
||||
14:53:12 diskread=5GB health=200 0.009107
|
||||
14:53:43 diskread=5GB health=200 0.011299
|
||||
14:54:14 diskread=5GB health=200 0.009412
|
||||
14:54:46 diskread=5GB health=200 0.012795
|
||||
14:55:17 diskread=5GB health=200 0.010044
|
||||
14:55:48 diskread=5GB health=200 0.011059
|
||||
14:56:20 diskread=5GB health=200 0.013508
|
||||
14:56:51 diskread=5GB health=200 0.013017
|
||||
14:57:23 diskread=5GB health=200 0.011061
|
||||
14:57:54 diskread=5GB health=200 0.010914
|
||||
14:58:25 diskread=5GB health=200 0.010145
|
||||
14:58:57 diskread=5GB health=200 0.010722
|
||||
14:59:28 diskread=5GB health=200 0.015195
|
||||
14:59:59 diskread=5GB health=200 0.011767
|
||||
15:00:31 diskread=5GB health=200 0.011621
|
||||
15:01:02 diskread=5GB health=200 0.010509
|
||||
15:01:33 diskread=5GB health=200 0.010722
|
||||
15:02:04 diskread=5GB health=200 0.009856
|
||||
15:02:36 diskread=5GB health=200 0.010521
|
||||
15:03:07 diskread=5GB health=200 0.013668
|
||||
15:03:38 diskread=5GB health=200 0.012826
|
||||
15:04:10 diskread=5GB health=200 0.011463
|
||||
15:04:41 diskread=5GB health=200 0.011157
|
||||
15:05:12 diskread=5GB health=200 0.010431
|
||||
15:05:43 diskread=5GB health=200 0.010361
|
||||
15:06:15 diskread=5GB health=200 0.011914
|
||||
15:06:46 diskread=5GB health=200 0.021984
|
||||
15:07:17 diskread=5GB health=200 0.013204
|
||||
15:07:49 diskread=5GB health=200 0.010710
|
||||
15:08:20 diskread=5GB health=200 0.009754
|
||||
15:08:51 diskread=5GB health=200 0.011999
|
||||
15:09:23 diskread=5GB health=200 0.010382
|
||||
15:09:54 diskread=5GB health=200 0.011156
|
||||
15:10:25 diskread=5GB health=200 0.010280
|
||||
15:10:57 diskread=5GB health=200 0.011247
|
||||
15:11:28 diskread=5GB health=200 0.010959
|
||||
15:11:59 diskread=5GB health=200 0.009834
|
||||
15:12:31 diskread=5GB health=200 0.013080
|
||||
15:13:02 diskread=5GB health=200 0.010975
|
||||
15:13:33 diskread=5GB health=200 0.009572
|
||||
15:14:04 diskread=5GB health=200 0.010632
|
||||
15:14:36 diskread=5GB health=200 0.011168
|
||||
15:15:07 diskread=5GB health=200 0.011665
|
||||
15:15:38 diskread=5GB health=200 0.009601
|
||||
15:16:09 diskread=5GB health=200 0.010077
|
||||
15:16:41 diskread=5GB health=200 0.011031
|
||||
15:17:12 diskread=5GB health=200 0.009288
|
||||
15:17:43 diskread=5GB health=200 0.017476
|
||||
15:18:15 diskread=5GB health=200 0.013927
|
||||
15:18:46 diskread=5GB health=200 0.011434
|
||||
15:19:17 diskread=5GB health=200 0.009776
|
||||
15:19:48 diskread=5GB health=200 0.010952
|
||||
15:20:20 diskread=5GB health=200 0.011278
|
||||
15:20:51 diskread=5GB health=200 0.010643
|
||||
15:21:23 diskread=5GB health=200 0.011549
|
||||
15:21:54 diskread=5GB health=200 0.009613
|
||||
15:22:26 diskread=5GB health=200 0.011846
|
||||
15:22:58 diskread=5GB health=200 0.011471
|
||||
15:23:29 diskread=5GB health=200 0.010358
|
||||
15:24:01 diskread=5GB health=200 0.011769
|
||||
15:24:32 diskread=5GB health=200 0.011816
|
||||
15:25:03 diskread=5GB health=200 0.010058
|
||||
15:25:34 diskread=5GB health=200 0.011328
|
||||
15:26:06 diskread=5GB health=200 0.010774
|
||||
@@ -0,0 +1,6 @@
|
||||
diskread: 483315769272
|
||||
diskwrite: 47257374720
|
||||
mem: 7075991552
|
||||
memhost: 8218480640
|
||||
netin: 4738753971
|
||||
netout: 916386350
|
||||
@@ -0,0 +1,16 @@
|
||||
# DST — what each scheduler does on Sunday 2026-10-25 (03:00 CEST → 02:00 CET), read from code and replayed
|
||||
|
||||
No architecture document mentions DST (`07` has no hit for DST, daylight or timezone — checked again 2026-10-20 with
|
||||
ASCII fragments `DST`, `daylight`, `time.?zone`, `ora.?allit`; positive control: `Europe/Budapest` is found in 07).
|
||||
|
||||
| scheduler | 2026-10-25 | 2027-03-28 (spring) | how known |
|
||||
|---|---|---|---|
|
||||
| controller `sched.Daily` (db-dump W, second copy W+60, off-site W+105, fill-watch, metrics, asset-sync, abandon-sweep, proof, integrity) | **once each**; a 02:xx time runs at the SECOND (CET) occurrence | a 02:xx time runs at 03:xx CEST — **the default 02:30 dump lands on 03:30, the same instant as the 03:30 second-drive copy** | PRODUCTION `nextDailyRunAt` replayed (`replay.txt`) |
|
||||
| hub `scheduleDaily` (db-snapshot 02:00, prune 04:30, deadline-check 05:00, key-audit 07:10) | **TWICE** — e.g. the alarm check at 04:00 CET and 05:00 CET | once each, one hour late | the hub's arithmetic replayed (`../../../dataloss-hunt-2026-10-20/C3-T9-hub-dst/replay.txt`) → **R-1183** |
|
||||
| agent (whole-guest due, restore-test, OS leg, cap) | unaffected (UTC durations) | unaffected | code read (C1 reviewer 4) |
|
||||
| controller quiesce gate [W+2h, W+6h) by wall minute | 04:30–08:30 CET for W = 02:30 (4 h real) | 04:30–08:30 CEST | code read |
|
||||
| host timers on a household box | monotonic only | monotonic only | code read |
|
||||
| DooPlex timers | Oct 25 is a Sunday: both weekly restore tests run on the DST day; the 02:30 hub-db-backup timer is inside the repeated hour — not verified | — | code read |
|
||||
|
||||
The live check of the box itself (B6 #7: the rehearsal VM's clock set to 2026-10-25 01:30 CEST and run to 04:00 CET)
|
||||
is recorded separately when it runs.
|
||||
@@ -0,0 +1,25 @@
|
||||
# controller 7ca59e2 (v0.336.0) scheduler.nextDailyRunAt replayed; go version go1.27.2 linux/amd64; 2026-10-20T11:06:36Z
|
||||
=== RUN TestRehearsalDST_ControllerDailyTimes
|
||||
zz_rehearsal_dst_test.go:24: 02:00: 10-25 02:00 CET(01:00Z) 10-26 02:00 CET(01:00Z) 10-27 02:00 CET(01:00Z)
|
||||
zz_rehearsal_dst_test.go:24: 02:15: 10-25 02:15 CET(01:15Z) 10-26 02:15 CET(01:15Z) 10-27 02:15 CET(01:15Z)
|
||||
zz_rehearsal_dst_test.go:24: 02:30: 10-25 02:30 CET(01:30Z) 10-26 02:30 CET(01:30Z) 10-27 02:30 CET(01:30Z)
|
||||
zz_rehearsal_dst_test.go:24: 02:59: 10-25 02:59 CET(01:59Z) 10-26 02:59 CET(01:59Z) 10-27 02:59 CET(01:59Z)
|
||||
zz_rehearsal_dst_test.go:24: 03:00: 10-25 03:00 CET(02:00Z) 10-26 03:00 CET(02:00Z) 10-27 03:00 CET(02:00Z)
|
||||
zz_rehearsal_dst_test.go:24: 03:30: 10-25 03:30 CET(02:30Z) 10-26 03:30 CET(02:30Z) 10-27 03:30 CET(02:30Z)
|
||||
zz_rehearsal_dst_test.go:24: 04:15: 10-25 04:15 CET(03:15Z) 10-26 04:15 CET(03:15Z) 10-27 04:15 CET(03:15Z)
|
||||
zz_rehearsal_dst_test.go:24: 05:30: 10-25 05:30 CET(04:30Z) 10-26 05:30 CET(04:30Z) 10-27 05:30 CET(04:30Z)
|
||||
zz_rehearsal_dst_test.go:24: 06:00: 10-25 06:00 CET(05:00Z) 10-26 06:00 CET(05:00Z) 10-27 06:00 CET(05:00Z)
|
||||
zz_rehearsal_dst_test.go:24: 22:00: 10-24 22:00 CEST(20:00Z) 10-25 22:00 CET(21:00Z) 10-26 22:00 CET(21:00Z)
|
||||
zz_rehearsal_dst_test.go:24: 02:00: 03-28 03:00 CEST(01:00Z) 03-29 02:00 CEST(00:00Z) 03-30 02:00 CEST(00:00Z)
|
||||
zz_rehearsal_dst_test.go:24: 02:15: 03-28 03:15 CEST(01:15Z) 03-29 02:15 CEST(00:15Z) 03-30 02:15 CEST(00:15Z)
|
||||
zz_rehearsal_dst_test.go:24: 02:30: 03-28 03:30 CEST(01:30Z) 03-29 02:30 CEST(00:30Z) 03-30 02:30 CEST(00:30Z)
|
||||
zz_rehearsal_dst_test.go:24: 02:59: 03-28 03:59 CEST(01:59Z) 03-29 02:59 CEST(00:59Z) 03-30 02:59 CEST(00:59Z)
|
||||
zz_rehearsal_dst_test.go:24: 03:00: 03-28 03:00 CEST(01:00Z) 03-29 03:00 CEST(01:00Z) 03-30 03:00 CEST(01:00Z)
|
||||
zz_rehearsal_dst_test.go:24: 03:30: 03-28 03:30 CEST(01:30Z) 03-29 03:30 CEST(01:30Z) 03-30 03:30 CEST(01:30Z)
|
||||
zz_rehearsal_dst_test.go:24: 04:15: 03-28 04:15 CEST(02:15Z) 03-29 04:15 CEST(02:15Z) 03-30 04:15 CEST(02:15Z)
|
||||
zz_rehearsal_dst_test.go:24: 05:30: 03-28 05:30 CEST(03:30Z) 03-29 05:30 CEST(03:30Z) 03-30 05:30 CEST(03:30Z)
|
||||
zz_rehearsal_dst_test.go:24: 06:00: 03-28 06:00 CEST(04:00Z) 03-29 06:00 CEST(04:00Z) 03-30 06:00 CEST(04:00Z)
|
||||
zz_rehearsal_dst_test.go:24: 22:00: 03-27 22:00 CET(21:00Z) 03-28 22:00 CEST(20:00Z) 03-29 22:00 CEST(20:00Z)
|
||||
--- PASS: TestRehearsalDST_ControllerDailyTimes (0.00s)
|
||||
PASS
|
||||
ok gitea.dooplex.hu/admin/felhom-controller/internal/scheduler 0.004s
|
||||
+27
@@ -0,0 +1,27 @@
|
||||
package scheduler
|
||||
|
||||
// THROWAWAY replay for the 2026-10-20 rehearsal (B6 #7, DST) — never committed. Replays the PRODUCTION
|
||||
// nextDailyRunAt over the two DST changes for every controller daily time; the loop re-reads the clock at each
|
||||
// fire instant exactly as the scheduler does.
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestRehearsalDST_ControllerDailyTimes(t *testing.T) {
|
||||
loc := getBudapestLocation()
|
||||
for _, win := range []struct{ from, to time.Time }{
|
||||
{time.Date(2026, 10, 24, 12, 0, 0, 0, loc), time.Date(2026, 10, 26, 12, 0, 0, 0, loc)},
|
||||
{time.Date(2027, 3, 27, 12, 0, 0, 0, loc), time.Date(2027, 3, 29, 12, 0, 0, 0, loc)},
|
||||
} {
|
||||
for _, hm := range []string{"02:00", "02:15", "02:30", "02:59", "03:00", "03:30", "04:15", "05:30", "06:00", "22:00"} {
|
||||
now, line := win.from, ""
|
||||
for now.Before(win.to) {
|
||||
next := nextDailyRunAt(hm, now)
|
||||
line += " " + next.In(loc).Format("01-02 15:04 MST") + "(" + next.UTC().Format("15:04Z") + ")"
|
||||
now = next
|
||||
}
|
||||
t.Logf("%s:%s", hm, line)
|
||||
}
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user