Compare commits
80 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 1b442bcc2e | |||
| 82ce9d72d6 | |||
| ecfc09c3ef | |||
| 25dae2f568 | |||
| 5ae5984cf9 | |||
| a92837410a | |||
| 115f15c89e | |||
| 6d5a7f1b92 | |||
| 6b391a639a | |||
| 0b7a5ab3bc | |||
| 26dbc987d9 | |||
| 97f088ec02 | |||
| dee26da3fa | |||
| c2c130ecbb | |||
| f576d7f7ba | |||
| 307d25b174 | |||
| 57c06ba99d | |||
| 9d3bf76def | |||
| 2293b252d5 | |||
| 6b0d23c29c | |||
| 2fa5b8f3d8 | |||
| f90e22293a | |||
| 3c620f8d2e | |||
| dbfe5dac06 | |||
| 47fa1e621e | |||
| ef4dbcf984 | |||
| cd3d05981a | |||
| 9688e76efd | |||
| e07b077ac2 | |||
| 7ad17be6bd | |||
| eaff4d890c | |||
| 7a287e71fc | |||
| 6559aa6d2e | |||
| a2d2569719 | |||
| ebf142d921 | |||
| 6d73110cd0 | |||
| ff06ab52da | |||
| a3be7525cd | |||
| 3decdf94a8 | |||
| 7a773207f8 | |||
| 19064ef1a7 | |||
| 31d975c997 | |||
| 0d3f72e463 | |||
| c4262a6f6c | |||
| a600d68d50 | |||
| 830ed9714e | |||
| 16188df64c | |||
| 0d4bfe7ca6 | |||
| 938afb7416 | |||
| c0b70b15ac | |||
| ce8cf2c477 | |||
| 4a557f54a5 | |||
| 92de2d324e | |||
| 12090df2d9 | |||
| 434fc7cc02 | |||
| 53d962d95b | |||
| 836b47f1ec | |||
| bb187f3503 | |||
| 9db8aacdad | |||
| 78278502ac | |||
| 2228b9f6b4 | |||
| ee743477b4 | |||
| ad6397308c | |||
| d750a90d0e | |||
| a942ae90d6 | |||
| 20a77d367b | |||
| a57a8363df | |||
| bffe2d9dea | |||
| 242584af4e | |||
| 43e5a5496d | |||
| c701b2530b | |||
| 54d1ee8210 | |||
| 1d7071e8fd | |||
| 9ec5f76a88 | |||
| f84f94d987 | |||
| 2954b0be3a | |||
| 5c487313fa | |||
| bcaa9576b2 | |||
| 301f4e9e92 | |||
| b0ae2c574a |
@@ -138,7 +138,14 @@ something, not only sessions that touch `documentation/` — which is why it is
|
|||||||
and the eleven names head `OPEN-ITEMS.md`, and `register_shape_gate.py` refuses anything else.
|
and the eleven names head `OPEN-ITEMS.md`, and `register_shape_gate.py` refuses anything else.
|
||||||
- **Root `STATUS.md`** — at the end of every session in which something shipped, broke or was
|
- **Root `STATUS.md`** — at the end of every session in which something shipped, broke or was
|
||||||
decided. It is a **view** of `OPEN-ITEMS.md`; nothing may exist only there. One screen, written for
|
decided. It is a **view** of `OPEN-ITEMS.md`; nothing may exist only there. One screen, written for
|
||||||
the operator in plain language, and deliberately **not** `CONTEXT.md`.
|
the operator in plain language (STE English), and deliberately **not** `CONTEXT.md`. How to update it:
|
||||||
|
1. **Replace** the „Updated" paragraph (at most 3 sentences, the newest change only) — never prepend; the old one
|
||||||
|
moves to `documentation/backlog/STATUS-HISTORY.md` under a dated heading.
|
||||||
|
2. A morning list goes under a dated sub-heading in „Waiting on you — do".
|
||||||
|
3. Remove what you finished: a done „do" item, a fixed „broken" line, a decided row (its ruling is in its row first).
|
||||||
|
4. A decide row: the question in one answerable sentence, at most 2 options, CC's pick first with the reason, the
|
||||||
|
other option's cost, what happens if the operator does nothing, and who is blocked.
|
||||||
|
5. `scripts/status_view_gate.py` checks it on every push (warn-first until 2026-10-27, then it refuses; R-1174).
|
||||||
- **The golden, weekly** (operator ruling 2026-10-15 11:26, `09` §3 decision 250; decision 7 stands): **a controller
|
- **The golden, weekly** (operator ruling 2026-10-15 11:26, `09` §3 decision 250; decision 7 stands): **a controller
|
||||||
release needs no golden** — the floor carries it, MinAgent declared, after the three test boxes (`09` §3 decision 250
|
release needs no golden** — the floor carries it, MinAgent declared, after the three test boxes (`09` §3 decision 250
|
||||||
item 2). The golden is baked **every Sunday night** when the floor is above it (`runbooks/nightly-rotation.md`) and
|
item 2). The golden is baked **every Sunday night** when the floor is above it (`runbooks/nightly-rotation.md`) and
|
||||||
|
|||||||
@@ -16,6 +16,85 @@
|
|||||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||||
|
|
||||||
|
|
||||||
|
> **2026-10-22 ~14:50 — operator acts, checked by CC (read only).** Host `Tester-2-be8404` deleted (12:54:47 UTC);
|
||||||
|
> customer `Tester-2` language set to Magyar; customer `Tester-2` RESET (12:55:56–12:56:04 UTC, journal #25, every leg
|
||||||
|
> ok). The first live run of R-32's purge-first leg: logged in with the sub-account's own password, found no repository
|
||||||
|
> (0 paths — the box never backed up off-site), deleted sub-account 322460; ep0 namespace + its 2026-10-04 copy + prune
|
||||||
|
> job + token gone. No other customer touched: Storage Box counts after = the hub's 05:10 listing (demo-hp 303,
|
||||||
|
> tester-1 62, demo-felhom 43; Tester4 has no box — compared by the hub on 10-23 05:10), ep0 after = the 03:30 prune
|
||||||
|
> keep-lists (2/2/2/1, operator 27). No false alarm. `audits/tester2-reset-2026-10-22/`.
|
||||||
|
|
||||||
|
> **2026-10-22 14:11 — operator ruling: Tester 2 is a NEW INSTALL, not an upgrade.** The friend reinstalls the laptop
|
||||||
|
> from the public download (installer 1.30.0 or newer) with the new drive in before the install; the old box's data is
|
||||||
|
> not needed („he barely used the laptop, no useful data on it"). Read on the hub the same day (read only): no sealed
|
||||||
|
> key package; off-site provisioned (shared) but its key never deposited; one ep0 whole-box copy (2026-10-04); the mail
|
||||||
|
> language is English. CC's pick for the operator: host delete, then customer RESET (it empties the Storage Box folder
|
||||||
|
> and the ep0 namespace — the operator's press). R-862 and R-1139 close at the reinstall; Tester 2's RESET is R-32's
|
||||||
|
> first live purge. List: `documentation/pilot/TESTER2-handover.md`.
|
||||||
|
|
||||||
|
> **2026-10-22 day — operator rulings (11:42) and what was built.** Rulings: **D28 = A** (loosen the host-swap check);
|
||||||
|
> agent 0.162.0 **vouched** (read back: `artifact_agent_version` 0.162.0 at 07:57 UTC); Tester 2's laptop was installed
|
||||||
|
> with **our installer**; Tester 2 is off, no date. Built: **R-1199 reopened** — controller 0.338.0's start-up default
|
||||||
|
> push overwrote demo-felhom's own saved mail list (13 → 17 events, 07:54:55 UTC); hub **0.165.0** marks a list the
|
||||||
|
> household saved (`household_saved`, backfilled once: Tester-2, Tester4 false; the other four true) and never lets a
|
||||||
|
> default overwrite it; it answers a box's self-scoped read; controller **0.339.0** reads the hub's list when it has
|
||||||
|
> none, adopts a household's, pushes its default once only when no household saved one. demo-felhom's list restored
|
||||||
|
> from the hub's 00:00 UTC snapshot through the hub's own save route; all six rows equal the snapshot after. Found and
|
||||||
|
> fixed with it: an empty saved list read back as the 17 defaults (`omitempty`; Tester 1 and demo-hp hold that shape),
|
||||||
|
> and the add-only seeds ran on a new box's second start. **R-1216 (D28 = A):** short host swap = none, or below 90 %
|
||||||
|
> of min(guests' swap, 8192 MB) — the installer's maximum (`Install.pm compute_swapsize`); one case table in hub and
|
||||||
|
> controller, `scripts/host_swap_rule_gate.py`. Tester 2 hand-over: `documentation/pilot/TESTER2-handover.md`.
|
||||||
|
|
||||||
|
> **2026-10-22 morning — the operator's five yeses, carried out.** Controller **0.338.0** released (image
|
||||||
|
> `sha256:e8a529f3…`, MinAgent 0.158.1): Tester 1, demo-hp, demo-felhom through per-customer floors, checks green
|
||||||
|
> (https-only 28/28, 0 unhealthy, 0 errors), then the global floor 0.338.0 (read back with the WAL). Agent **0.162.0**
|
||||||
|
> released (`5fd7a23b…` = the soak's bytes) and delivered by signed job to the same three; **the vouch is the
|
||||||
|
> operator's**. Hub **0.164.0** deployed (only `Deployment/hub` synced). ISO **1.30.0** (W1, W7) built, gated (G1–G16,
|
||||||
|
> G12 not re-read) and proof-installed from both menu entries on demo-hp, published, round trip `590d7b9b…`; both
|
||||||
|
> download pages point at it and carry the W2 button. Operator answer: Tester 2's laptop was installed with our
|
||||||
|
> installer (R-1207). **Found:** the host-swap check (agent WARN, hub `host_swap_short`, the controller's install
|
||||||
|
> sentence) compares host swap with the guests' WHOLE swap allowance, so every box with a guest above ~8 GB trips it
|
||||||
|
> (demo-hp, demo-felhom mailed) though the soak burst used 2.7 GB — R-1216, decision D28. The runbook's G16 command
|
||||||
|
> was stale (now `scripts/iso_bootstrap_gate.py`). Record: `REPORT-morning-2026-10-22.md`.
|
||||||
|
|
||||||
|
> **2026-10-21 night (unattended; nothing released, deployed, published or vouched).** Operator rulings recorded:
|
||||||
|
> **D22 = yes (20:05)** — D20's put-back extended to the local unit restore, the second-drive restores and a restore cut by
|
||||||
|
> a power loss (`07` §6.3, the 2026-10-21 `[DESIGN]` paragraph; controller `d582feb`, proven on 9202 with 0.338.0-rc.1:
|
||||||
|
> a damaged unit → the live data back and running; a controller killed after the volume step → put back on start);
|
||||||
|
> **D26 = A (20:03)** — release and vouch the agent with the swap fix in the morning; a host-swap warning follows (agent
|
||||||
|
> `0da792e` reports `host_swap_mb`/`guest_swap_mb`, one WARN; hub 0.164.0 built, not deployed: operator-only
|
||||||
|
> `host_swap_short`; controller install-page sentence). **Decided by CC unattended — operator may reverse (R-1199):** the
|
||||||
|
> box's notification page is the source the hub mails the household by: a box whose household never saved a choice now
|
||||||
|
> pushes the default list its page shows at start (no address; the hub's no-clobber guard keeps the stored one) —
|
||||||
|
> `08` (the household decides; „the box's page is the source"). Measured: Tester4's hub list was saved 2026-10-18 and
|
||||||
|
> lacked disk warning, drive disconnected and second-drive failure, so the household got 1 alarm of 9. Built tonight:
|
||||||
|
> controller `0.338.0-rc.2` (`sha256:ae52c93e…`, all of tonight's controller work, smoke-tested on 9202, 9202 back on
|
||||||
|
> 0.337.0), hub `0.164.0` (`sha256:931594e3…`, manifest still 0.163.0), agent candidate `0.162.0` from `0da792e`
|
||||||
|
> (sha256 `5fd7a23b…`, not published). The 8 GB soak with the swap fix (`audits/night-2026-10-21/P1-soak/`): 11 h, no
|
||||||
|
> freeze, 0 failed health probes (largest interval 16 s), every night job ran; memory „full" above 20 % for ~3 min in the
|
||||||
|
> 03:30 tier2 job (R-1214); 1 photo of ~26 000 not stored (R-1215). With `swapoff -a` on the host the freeze came back
|
||||||
|
> within 20 s (dead up to 289 s; `P2-noswap/`) — the fix depends on host swap (R-1207).
|
||||||
|
|
||||||
|
> **2026-10-21 (day brief, before Tester 2) — hub 0.163.0 DEPLOYED (operator yes, 11:46 UTC); controller 0.337.0 released
|
||||||
|
> (three test boxes 14:06 UTC, global floor after the freeze proof); agent `286a7e3` committed, NOT released.** Operator rulings
|
||||||
|
> today: **D24 = yes** (the hub's DST double run fixed: `nextDailyRun`, calendar day+1, `05` §16.5); **D25** (a second-drive
|
||||||
|
> copy whose SOURCE is gone is kept until the household releases it — `07` §8.2a; controller `tier2_kept.go`, the restore
|
||||||
|
> page's „Megőrzött másolatok"); the **R-1176 shape** = the hub CARRIES the household's retained sealed package to the
|
||||||
|
> rebuilt host (`store.ReattachRetainedEscrow`: one live host, no package held, the newest escrow-ACKED deletion; moved in
|
||||||
|
> one tx at the agent's first report and the controller's report; never opened/logged; two live hosts → refused + operator
|
||||||
|
> mail) — `03` §8a. Proven live on demo-hp (Tester4 from the public ISO): no fresh key minted, recovery offered, wrong code
|
||||||
|
> refused, right code opened the history, Nextcloud 3 031/3 033 files byte-identical, off-site appended 63 → 64
|
||||||
|
> (`audits/fixes-2026-10-21/P1-live/`). R-1182: retained rows carry `repo_tag` (NULL legacy / 'live' / the set-aside path
|
||||||
|
> tagged at the hub's move-aside); the abandonment purges only rows tagged with a path the hub DELETED; the ACK says
|
||||||
|
> `abandon_purge_done`. R-1203: the new-link page tells the holder of a real spent link that the old box is still linked;
|
||||||
|
> operator mail `bind_old_host_linked`. **R-1192 diagnosed with numbers:** Proxmox's LXC default swap cap (512 MB) — the
|
||||||
|
> agent never set `swap` — turned Immich's model load (+1.15 GB) into a code-page refault thrash at the 4096 MB guest limit
|
||||||
|
> (150 000–600 000 refaults and ~1 GB of reads per 10 s, memory „full" 92–95 %, load 155, dashboard dead 6½ min, 0 OOM
|
||||||
|
> kills); `pct set --swap 4096` live ended it within a minute. Agent fix (swap = RAM at bring-up, at RAM resize, once at
|
||||||
|
> start) committed; its release is the operator's. The controller's „Normál használat mellett ez nem okoz problémát" is
|
||||||
|
> gone (0.337.0). Observed and filed: R-1205 (PBS re-issue after an un-acked second install), R-1206 (future-dated
|
||||||
|
> snapshots). Report: `REPORT-fixes-2026-10-21.md`.
|
||||||
|
|
||||||
> **2026-10-19 (night) — controller 0.335.0 is the floor (MinAgent 0.158.1, 18:39 UTC); hub 0.162.0 BUILT, not deployed;
|
> **2026-10-19 (night) — controller 0.335.0 is the floor (MinAgent 0.158.1, 18:39 UTC); hub 0.162.0 BUILT, not deployed;
|
||||||
> agent `56aabc7` and installer 1.35.0 committed, not released/published.** R-921 (operator answer „add one field"): the
|
> agent `56aabc7` and installer 1.35.0 committed, not released/published.** R-921 (operator answer „add one field"): the
|
||||||
> agent serves its host-wide heavy-operation gate as `host_busy` on `GET /backup/status`; the controller reads it before
|
> agent serves its host-wide heavy-operation gate as `host_busy` on `GET /backup/status`; the controller reads it before
|
||||||
|
|||||||
@@ -0,0 +1,70 @@
|
|||||||
|
# REPORT — the day of 2026-10-22: the family's mail list, the swap check (D28 = A), the Tester 2 hand-over
|
||||||
|
|
||||||
|
**Operator rulings (11:42):** D28 = A; agent 0.162.0 vouched (read back: hub `artifact_agent_version` 0.162.0, 07:57
|
||||||
|
UTC); Tester 2's laptop used our installer; Tester 2 off, no date. Design read: `08-alarm-ladder.md`,
|
||||||
|
`05-hub-architecture.md`, `03-host-agent.md`. Baselines: controller `08fa440d`, agent `2d5840ba`, felhom.eu `a9283741`.
|
||||||
|
|
||||||
|
## Four lines
|
||||||
|
|
||||||
|
1. **Mail list fix live** on the hub (0.165.0) and on Tester 1, demo-hp and demo-felhom (controller 0.339.0, global
|
||||||
|
floor). A box with no list of its own now adopts the family's list from the hub; a default never overwrites it.
|
||||||
|
2. **One family list restored: demo-felhom** (13 events, from the hub's 00:00 UTC snapshot; this morning's release had
|
||||||
|
made it 17). After the release demo-felhom adopted it; all six hub rows equal the snapshot.
|
||||||
|
3. **Swap warning rule:** none, or below 90 % of min(the guests' swap, 8 GB). The two morning alarms cleared at the next
|
||||||
|
report (demo-hp 11:04, demo-felhom 11:09 UTC), no mail. No agent change.
|
||||||
|
4. **Hand-over list:** `documentation/pilot/TESTER2-handover.md`, linked from STATUS.
|
||||||
|
|
||||||
|
## Versions, digests, CI
|
||||||
|
|
||||||
|
| | version | digest | CI (commit → job) |
|
||||||
|
|---|---|---|---|
|
||||||
|
| hub | 0.165.0, DEPLOYED (operator yes; only `Deployment/hub` synced, manifest `25dae2f5`) | `sha256:c6180fb79a2ff7d5546a704c68b634b9207afb0a0ecd524c74b0310434532537` | `5ae5984c` → 2207 success; `25dae2f5` → 2208 success |
|
||||||
|
| controller | 0.339.0-rc.1 (9202) | `sha256:7d4c88528b50adeb891e409cf50343f44fb168a84e890d42864b8124f3051fdd` | `add310c` → 2206 success |
|
||||||
|
| controller | 0.339.0, released, global floor (MinAgent 0.158.1) | `sha256:fee71aafceb4a7d5f24ae4a6a273a8d51eb26e0550fe9d12d4d86955f6a343a8` | `fc5a20f` → 2209 success |
|
||||||
|
|
||||||
|
Agent: no change. Evidence: `documentation/audits/day-2026-10-22/` (rows before, the restore, rows after, the release
|
||||||
|
on the three boxes, https-only).
|
||||||
|
|
||||||
|
## What was built
|
||||||
|
|
||||||
|
- **Hub:** `customer_notifications.household_saved` — set only by the household's own save (`source: household`),
|
||||||
|
never cleared; any other push over a marked list → `{"status":"kept"}` (an older controller's push with neither
|
||||||
|
address nor clear counts as not the household's — that is 0.338.0's default push); `GET /api/v1/preferences`
|
||||||
|
(customer-scoped); one-time backfill (written after creation → a box wrote it): Tester-2 false, Tester4 false,
|
||||||
|
demo-felhom / demo-hp / tester-1 / `_resend-rotation-test` true. `host_swap_short` by the new rule.
|
||||||
|
- **Controller:** start-up — a saved list as before (R-922); none saved → read the hub, adopt a household's, else push
|
||||||
|
the default once and save it; unreadable hub → nothing. The swap sentence by the same rule.
|
||||||
|
- **Contract:** one case table in both repos, byte-identical, each run by a Go test; `scripts/host_swap_rule_gate.py`
|
||||||
|
with 4 decoys (`test_host_swap_rule_gate.py`).
|
||||||
|
- **Red-proofs seen failing:** hub A / A-legacy / C (refusal removed), the swap table and the clear (morning rule), the
|
||||||
|
backfill (marking off); controller A / B / unreadable hub (0.338.0's rule), the empty-list test (getter restored),
|
||||||
|
the swap table (morning rule).
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
|
||||||
|
Before 178, after 178. Opened 2: R-1199 (reopened), R-1217 (demo-hp's hub list has 3 app events its page does not
|
||||||
|
show — P4). Closed 2: R-1199, R-1216.
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- An empty saved list read back as the 17 defaults (Tester 1, demo-hp) — **NOT-A-FINDING: fixed in this session**
|
||||||
|
(size rule; controller CHANGELOG „Fixed without a row").
|
||||||
|
- The add-only seeds re-added a just-switched-off mail on a new box's second start — **NOT-A-FINDING: fixed in this
|
||||||
|
session** (same entry).
|
||||||
|
- demo-hp: hub mails 3 app events the page does not show — **FILED: R-1217**.
|
||||||
|
- The cleared swap alarms sent no mail (`host_swap_ok` is info) — **NOT-A-FINDING: the ladder sends no operator mail
|
||||||
|
for an info clear; the event row is the record.**
|
||||||
|
- A docstring test refused my first push (`9c.` is not a gate-list number) — **NOT-A-FINDING: my numbering; fixed
|
||||||
|
before the push.**
|
||||||
|
- The safety check blocked one `sh -c` with a temp-file removal; rewritten with no file to remove — **NOT-A-FINDING:
|
||||||
|
nothing was run.**
|
||||||
|
- 9202 has no hub link, so the rc there proved start-up only — **NOT-A-FINDING: the mail path was proven on the three
|
||||||
|
test boxes through the floor.**
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
- **Machine:** nothing created. 9202 now runs 0.339.0-rc.1 (the scratch box; the floor moves it at its next report if
|
||||||
|
it reports — it has no hub link, so set it by hand next time it is needed).
|
||||||
|
- **Host:** the three test boxes on controller 0.339.0 — intended.
|
||||||
|
- **Hub:** one row written (demo-felhom's restored list, through `POST /api/v1/preferences`); global floor 0.339.0;
|
||||||
|
no per-customer floor left.
|
||||||
@@ -0,0 +1,131 @@
|
|||||||
|
# REPORT — the day brief of 2026-10-21: fix the four serious problems from the rehearsal, and the Sunday clock change
|
||||||
|
|
||||||
|
Separate from the shared `REPORT.md` on purpose: another session works in this clone (repo rule: a second session writes
|
||||||
|
`REPORT-<topic>.md`). Evidence: `documentation/audits/fixes-2026-10-21/`. Design read first: `03` §8a, `07` §7.0/§6/§8.2,
|
||||||
|
`05` (scheduler, escrow store), `04` (self-scoped reads), `09` §3 decisions D19, 78, R-241.
|
||||||
|
|
||||||
|
## 1. Five lines
|
||||||
|
|
||||||
|
1. **R-1176 (a rebuilt box gets its off-site backups back) — FIXED AND LIVE.** Hub 0.163.0 (operator yes, 11:46 UTC);
|
||||||
|
proven end to end on demo-hp: no fresh key, recovery offered, wrong code refused, right code opened the history,
|
||||||
|
Nextcloud 3 031 of 3 033 files byte-identical, the next off-site runs appended (63 → 64).
|
||||||
|
2. **R-1175 (shares on two drives) — FIXED AND LIVE.** Controller 0.337.0 (global floor 15:54 UTC), D25: a gone source
|
||||||
|
keeps its copy until the household removes it.
|
||||||
|
3. **R-1191 (an imported app keeps its family lock) — FIXED AND LIVE.** Controller 0.337.0; proven on 9202: a stranger
|
||||||
|
gets 401 after export → remove → import (it was 200).
|
||||||
|
4. **R-1192 (the 8 GB freeze) — CAUSE MEASURED, FIX COMMITTED, NOT RELEASED.** The guest's swap was capped at Proxmox's
|
||||||
|
512 MB default; the agent fix (swap = RAM) is committed (`286a7e3`); its release and vouch are the operator's (D26).
|
||||||
|
Proven by applying the same setting by hand for 2 hours of the same uploads.
|
||||||
|
5. **Ready for Tester 2: NO — one step left:** the agent release with the swap fix. Everything else on the P1 list is live.
|
||||||
|
|
||||||
|
## 2. R-1192 — the mechanism, the numbers, the fix
|
||||||
|
|
||||||
|
**One sentence:** the guest's swap is capped at 512 MB (Proxmox's LXC default — the agent never set `swap`), so when
|
||||||
|
Immich loaded its analysis models (+1.15 GB) on top of 8 apps, the 4096 MB guest could evict only program code, and
|
||||||
|
~25 containers re-read their code from disk without end.
|
||||||
|
|
||||||
|
| | the freeze (13:26–13:34 UTC, cap 512 MB) | the proof (13:43–15:43 UTC, cap = RAM by hand) |
|
||||||
|
|---|---|---|
|
||||||
|
| guest private memory (anon), max | 3 585 MB | 2 782 MB |
|
||||||
|
| guest swap used, max | 511 MB of 512 (full) | 2 555 MB |
|
||||||
|
| refaults per 10 s, max | 2 741 774 | 187 462 |
|
||||||
|
| disk reads per 10 s, max | 12 GB | 0.9 GB |
|
||||||
|
| memory „full" pressure (avg10), max | 96 % | 12 % |
|
||||||
|
| IO pressure, max | 97 % | 53 % |
|
||||||
|
| load, max | 162 | 33 |
|
||||||
|
| guest „limit hit" events per 10 s, max | 3 889 | 0 |
|
||||||
|
| OOM kills | 0 | 0 |
|
||||||
|
| dashboard (DooPlex probe every 10 s) | dead 13:28:18–13:34:48 | every probe answered, slowest 1.26 s |
|
||||||
|
|
||||||
|
The experiment between them: `pct set 9201 --swap 4096` at 13:34:06, uploads still running → memory „full" 96 % → 1.7 %
|
||||||
|
within 2 minutes, the dashboard back 42 s after the change. The host had 2.2 GB RAM and 7.4 GB swap free throughout.
|
||||||
|
Load for the proof: the same uploads as fresh work, repeated — Paperless 110 cycles × 53 PDFs (emptied each time), Immich
|
||||||
|
111 new users × 306 photos (305 accepted each time; one photo is refused every cycle), two Nextcloud trees (WebDAV small
|
||||||
|
files, ~20 files/min under this load — slow, nothing lost). Method: a recorder on the box (`tools/rec.py`, 10 s, copied to
|
||||||
|
DooPlex every minute) and the second channel on DooPlex (`tools/outside.sh`: the dashboard's `/api/health` and QEMU's
|
||||||
|
disk counters from demo-hp).
|
||||||
|
|
||||||
|
**Fix:** agent `286a7e3` (commit only): `swap` = memory at bring-up, at a RAM resize, and once at agent start for every
|
||||||
|
pool guest whose swap is below its RAM (live, no restart, never lowers). Tests red-proved. The controller half (0.337.0):
|
||||||
|
„Normál használat mellett ez nem okoz problémát." is gone; the install page now says **„Az alkalmazások csúcsterhelése
|
||||||
|
meghaladhatja a rendelkezésre álló memóriát. Ilyenkor a doboz nagy feltöltések alatt lelassulhat, és akár percekig sem
|
||||||
|
válaszol. Ha ez gyakran előfordul, távolíts el egy alkalmazást."** No catalog change and no admission-ceiling change:
|
||||||
|
the measured cause is the swap cap, and the order of preference (catalog → controller → agent) gave way to the proof.
|
||||||
|
R-1193 (an earlier, plainer alarm): a design note in its row; build it only if a freeze is seen after the agent release.
|
||||||
|
|
||||||
|
## 3. R-1176 — the shape, the order check, the live proof
|
||||||
|
|
||||||
|
**Shape (operator ruling):** `store.ReattachRetainedEscrow` moves the newest retained row of the customer's newest
|
||||||
|
escrow-ACKED host deletion into `host_escrow` for its one live host, in one transaction (no other live host, no live
|
||||||
|
package, the deleted host not live again, an identity blob present). Called on the agent's host report and before the
|
||||||
|
controller ACK's escrow status. Never opened, re-sealed or logged; audit `escrow_carried` (host ids + hash); two live
|
||||||
|
hosts → `escrow_carry_refused` mailed once. Self-scope unchanged: the new host reads only rows attached to it (its
|
||||||
|
retained GET for the old host is 403).
|
||||||
|
|
||||||
|
**Order check (measured, not assumed):** the hub carried at the agent's FIRST report (12:04:21 UTC); the controller's
|
||||||
|
settle gate went GO at 12:06:25 after its first ACK; at 12:06:28 it logged „NOT minting a repository password: the hub
|
||||||
|
holds a sealed recovery package". So no controller change was needed; 0.337.0 still records the escrow before
|
||||||
|
`SetFloor` (a pinned belt for a hub that carries later).
|
||||||
|
|
||||||
|
**Live proof (demo-hp VM 9321, Tester4, public ISO 1.29.0, no operator press before the install):** bind (one OCR misread
|
||||||
|
of the pairing code cost one of five tries — harness), claim, the dashboard offered „A korábbi, házon kívüli mentéseid
|
||||||
|
visszaszerezhetők"; a wrong code: „A megadott helyreállítási kódot nem fogadtuk el … Semmi nem változott"; the right code
|
||||||
|
unlocked 9 groups; Nextcloud reinstalled („előbb telepítsd újra" is the page's own order), „Teljes visszaállítás" with
|
||||||
|
the placement confirm → „3113 fájl és 2 adatkötet és az adatbázis visszaállítva"; **sha256 through Nextcloud's own WebDAV
|
||||||
|
against the rehearsal's A5 manifest: 3 031 of 3 033 byte-identical** — the 2 missing never reached the old box (the 1.5 GB
|
||||||
|
file Nextcloud refused, a symlink rclone skips); 4 extra files are the rehearsal's later uploads. Off-site: two runs, 63 → 64
|
||||||
|
snapshots in the same 1.4 GB store, `orphaned: false`; the hub's full log has no set-aside and no key change.
|
||||||
|
|
||||||
|
## 4. Versions, commits, tests
|
||||||
|
|
||||||
|
| Component | Version | State | Commits |
|
||||||
|
|---|---|---|---|
|
||||||
|
| hub | 0.163.0 | **deployed** 11:46 UTC (operator yes), Synced/Healthy | `9688e76e`, manifest `cd3d0598` |
|
||||||
|
| controller | 0.337.0 (`sha256:264af313…`) | **released**: rc.1 on 9202; per-box floors 13:59 → three test boxes on 0.337.0 by 14:06; box walk green; **global floor 0.337.0 / MinAgent 0.158.1 at 15:54:29 UTC**, read back; per-box floors cleared | `8f5bea0`, `5929b8e`, `7848d4a` |
|
||||||
|
| agent | (0.162.0 when released) | **committed, not released** | `286a7e3` |
|
||||||
|
| catalog, installer | — | unchanged | — |
|
||||||
|
|
||||||
|
Tests (all pass; each red-proved — seen failing first): hub `TestR1183_*` (3), `TestR1176_*` (store 7, api 3),
|
||||||
|
`TestR1182_*` (store 2, api 1), `TestR1203_*`; controller `TestR1175_*` (3), `TestR1181_*` (2), `TestD25_*` (3),
|
||||||
|
`TestR1179_*`, `TestR1188c_*`, `TestR1191_*` (2), `TestR1176_EscrowIsRecordedBeforeTheFloorIsKnown`, `TestR1182_*` (2);
|
||||||
|
agent `TestR1192_*` (2) + the resize test's swap assertion. CI: green by job id for every push (see §5).
|
||||||
|
|
||||||
|
## 5. Morning list, teardown, rows, observations
|
||||||
|
|
||||||
|
**Morning list (operator):** D26 — release + vouch the agent with the swap fix; delete host **Tester4-f6b67c** (asked
|
||||||
|
today); R-1205 (the whole-box tier waits for „Re-issue PBS credentials" after a failed second install); the
|
||||||
|
`REHEARSAL_2026_10_20_*` / `FIXES_2026_10_21_*` names in sec.
|
||||||
|
|
||||||
|
**Operator acts today:** the hub-deploy yes; the root credential reveal for the freeze test (~2 min); the host delete
|
||||||
|
(done 16:49 UTC — the delete appears 6 minutes after a box goes off, R-30's rule).
|
||||||
|
|
||||||
|
**Teardown (three layers):**
|
||||||
|
- **Machine (demo-hp, Tier 0):** VM 9321 `rh1021-box2` and helper CT 9331 `rh1021-loader` destroyed with `--purge` at
|
||||||
|
16:27:40 UTC (names checked first); no image left on `nvme-scratch`; 341, 9201, 9202, 9401 untouched. 9202 moved from
|
||||||
|
`0.337.0-rc.1` to `0.337.0` by hand (it does not take the hub's floor). The public ISO 1.29.0 was only read.
|
||||||
|
- **Hub:** the operator deleted host Tester4-f6b67c (16:49:37 UTC, escrow acknowledged): no Tester4 host, no current
|
||||||
|
package, the household's package retained under Tester4-f6b67c with the same hash, `repo_tag` 'live' — a future
|
||||||
|
rebuild carries it again; the WireGuard peer gone (`P6-release/05-teardown-hub.txt`).
|
||||||
|
- **ep0 / Storage Box (read only, nothing removed):** Tester4's off-site store gained 2 snapshots (63 → 64, normal
|
||||||
|
traffic); the hub's registrar pinned the rebuilt box's key and removed the dead box's line (normal enrollment). No PBS
|
||||||
|
descriptor was made (R-1205), so nothing new on ep0's PBS.
|
||||||
|
|
||||||
|
**Rows:** before 182 (181 after a parallel session closed R-1201); after 176; opened 2 (R-1205, R-1206); closed 7 (R-1176,
|
||||||
|
R-1203, R-1175, R-1179, R-1181, R-1182, R-1191). R-1183 → VERIFY (due 2026-10-26); R-1192 → WAITING-ON-OPERATOR;
|
||||||
|
R-1188 (c) fixed. Capability map A12, A13 → PROVEN-LIVE. `unproven.py --summary`: unchanged, 35 of 55 not walked.
|
||||||
|
|
||||||
|
**Observations:**
|
||||||
|
- PBS auto re-issue refused for the rebuilt Tester4 (latest deletion un-acked) — **FILED: R-1205**.
|
||||||
|
- Future-dated snapshots from the clock test show as „Legutóbbi mentés: 2026. 10. 28." — **FILED: R-1206**.
|
||||||
|
- The pairing-code OCR misread one character — **NOT-A-FINDING:** harness (a human reads the screen); corrected against the hub's record.
|
||||||
|
- Immich's first admin needs the household's dashboard session (the setup gate) — **NOT-A-FINDING:** designed.
|
||||||
|
- The full restore asks to confirm a new place (backup on `hdd_1`, now the internal disk) — **NOT-A-FINDING:** designed (R-351).
|
||||||
|
- Nextcloud's small-file WebDAV upload ~20 files/min under heavy concurrent load — **NOT-A-FINDING:** slow, not lost; the new install-page sentence says big uploads can be slow.
|
||||||
|
- `kubectl logs -l` reads only 10 lines — **NOT-A-FINDING** (a tooling trap; saved to memory, two checks redone over the full log).
|
||||||
|
- Tester 2 last reported 2026-10-04 on 0.292.0; the floor reaches it only when it reports — **NOT-A-FINDING** (R-1139 covers the box).
|
||||||
|
|
||||||
|
**Secrets:** names only — `FIXES_2026_10_21_PAIRING_CODE`, `FIXES_2026_10_21_BIND_TOKEN`, `FIXES_2026_10_21_SETUP_CODE`
|
||||||
|
(one-time, spent), `FIXES_2026_10_21_ROOTPW` (the operator's reveal; the box is torn down). The bind link and the setup code
|
||||||
|
were read from the mailbox by the mail connector, so both one-time values entered this session's record (spent within
|
||||||
|
minutes, as in the rehearsal). A scan of all 45 evidence files for every secret used today: 0 hits. Nothing secret in any
|
||||||
|
commit.
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
# REPORT — the morning of 2026-10-22: the operator's five yeses
|
||||||
|
|
||||||
|
**Operator answers:** release controller 0.338.0 — yes; release and vouch agent 0.162.0 — yes; deploy hub 0.164.0 — yes
|
||||||
|
(Tester4 host deleted); Tester 2's laptop — installed with our installer; publish items — yes, all. Design read:
|
||||||
|
`07` §6.3 (restore put-back), `03` §9 (guest sizing), the felhom-build-deploy runbooks, `runbooks/iso-release-gate.md`.
|
||||||
|
|
||||||
|
## Decisions taken by CC
|
||||||
|
|
||||||
|
None new. The night's R-1199 decision (a box with no saved notification choice pushes its page's default list) now
|
||||||
|
ships in controller 0.338.0 — *operator may reverse*.
|
||||||
|
|
||||||
|
## What was done
|
||||||
|
|
||||||
|
| | what | result |
|
||||||
|
|---|---|---|
|
||||||
|
| agent | `release-agent.sh 0.162.0` from a clean clone at `b310773` | sha256 `5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928` (= the soak's bytes), bundle `ac772967…60b5a5`, tag `v0.162.0`, round trip OK |
|
||||||
|
| agent | one signed `agent_update` per box: demo-hp, Tester 1, demo-felhom (07:41 UTC) | all three `update committed`, `ok=76 total=76`, `guest-swap: raised` for 9201 (512 MB → RAM); `pct config` swap = memory on all three |
|
||||||
|
| hub | manifest `0.163.0 → 0.164.0` (`0b7a5ab3`), only `Deployment/hub` synced | running `sha256:931594e3…` (= the night's build), log `felhom-hub 0.164.0 starting` |
|
||||||
|
| controller | `build.sh 0.338.0 --push` from `7900a32` | `sha256:e8a529f395bfe8c0e7b87d5d98929a6f8e9834aa13c0ab77bd214ee9e7e53433` |
|
||||||
|
| controller | per-customer floors (tester-1, demo-hp, demo-felhom), MinAgent 0.158.1 | all three on 0.338.0 within 10 s (agent `controller-swap: new controller healthy`); https-only 28/28 + control; 10/24/9 containers, 0 unhealthy, 0 restarting; 0 `level=ERROR` |
|
||||||
|
| controller | global floor 0.338.0 / MinAgent 0.158.1 with CC's hub token | `303 → flash=floor_set`; hub DB (with WAL): `min_controller_version=0.338.0`, `0.338.0=0.158.1`; the three per-customer floors cleared |
|
||||||
|
| website | the W2 patch (`6b391a63`), site gates OK | live on felhom.eu/letoltes.html and /en/download |
|
||||||
|
| ISO | 1.30.0 `--release` from `115f15c8` | `590d7b9b3ccbc7644536a1971a44deb2d1bf7136f69eae2b4dc42961e74d5cc1`, 1 705 324 544 B; gate G1–G16 PASS (G12 not re-read); proof installs from BOTH entries (VMs 9341, 9342), first boot + one reboot; W1 seen on screen; published, round trip identical; both download pages point at 1.30.0 |
|
||||||
|
|
||||||
|
Evidence: `documentation/audits/morning-2026-10-22/release/`, `documentation/tests/iso-release-1.30.0-2026-10-22/`.
|
||||||
|
|
||||||
|
## What broke
|
||||||
|
|
||||||
|
- **The host-swap check is too strict (R-1216, D28).** Within minutes of agent 0.162.0 demo-hp (host swap 8191 MB vs
|
||||||
|
guests 25898 MB) and demo-felhom (8191 vs 12288) warned and the hub mailed `host_swap_short`. The check compares host
|
||||||
|
swap with the guests' whole swap allowance; the installer gives at most 8 GB, the soak's burst used 2.7 GB. The same
|
||||||
|
comparison drives the controller's install-page sentence. Not fixed: the threshold is the operator's pick.
|
||||||
|
- **A stale runbook command (fixed).** `iso-release-gate.md` G16 mounted `scripts/iso` as `/work`; the harness needs
|
||||||
|
`golden/` and `postinst` staged, so 10 checks failed on a correct tree. Now names `scripts/iso_bootstrap_gate.py`
|
||||||
|
(73 checks green). My first run with the old line left harness scratch files in `scripts/iso/` (root-owned); removed
|
||||||
|
by name before any commit.
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
|
||||||
|
Before 184, after 178. Opened 1 (R-1216). Closed 7: R-1192, R-1177, R-1178, R-1180, R-1195, R-1199, R-1210. Updated:
|
||||||
|
R-1194 (released; WATCHING for the next fresh install; owner CC), R-1207 (Tester 2 = our installer). Section headers
|
||||||
|
recounted (Backup & restore read 44, held 42).
|
||||||
|
|
||||||
|
## Waiting on the operator
|
||||||
|
|
||||||
|
1. **Vouch agent 0.162.0** (Configs → Day-0 artifacts). If not: new installs get 0.161.0, without the swap fix.
|
||||||
|
2. **Discard appliances 47 and 48** (the proof installs). If not: two „unclaimed" rows stay.
|
||||||
|
3. **D28** — loosen the host-swap check (A, CC's pick) or keep it (B).
|
||||||
|
4. **Tester 2** is off; when it is back, CC sends it agent 0.162.0.
|
||||||
|
5. The secret names `REHEARSAL_2026_10_20_*`, `FIXES_2026_10_21_*`, `NIGHT_2026_10_21_*` are KEPT (the proof installs
|
||||||
|
used the night's root password); say „remove" to drop them.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
- **Machine:** VMs 9341, 9342 destroyed (names checked); ISO 1.30.0 kept in demo-hp `local:iso`. No thin-pool write.
|
||||||
|
- **Host:** demo-hp, Tester 1, demo-felhom now on agent 0.162.0 and controller 0.338.0 — intended, not torn down.
|
||||||
|
- **Hub:** appliances 47, 48 for the operator to discard; floors: global 0.338.0, no per-customer floor left.
|
||||||
|
|
||||||
|
## Instruction files / poster
|
||||||
|
|
||||||
|
No instruction file changed. No system-poster fact changed (versions are not poster facts).
|
||||||
@@ -1,4 +1,129 @@
|
|||||||
# REPORT — evening/night brief 2026-10-19 (one stop less per night, no false tunnel error, R-893 measured, hub and agent fixes ready, a scanner on every push, ep0's first weekly clean-up)
|
# REPORT — the night briefs of 2026-10-19 and 2026-10-20 (part 2 run by day): one stop less per night, the safe undo for a failed off-site restore, hub 0.162.0 / agent 0.161.0 / installer 1.35.0 delivered, downloads out of the backups, a scanner on every push
|
||||||
|
|
||||||
|
# PART 2 — the night brief part 2 (run by day, 2026-10-20): your four answers, the safe undo, downloads out of the backups, Immich, the update test
|
||||||
|
|
||||||
|
**Safest readings taken alone (written first, as the brief asks):**
|
||||||
|
1. **R-1083 is not in agent 0.161.0.** An agent key would make 9202's controller book the agent's whole-guest backups,
|
||||||
|
one of which writes to ep0 — a change on ep0 the brief forbids; 9202 is also outside the `felhom` pool.
|
||||||
|
2. **All five downloaders' folders are `excluded`, not only the two named** (qBittorrent, Transmission; also NZBGet,
|
||||||
|
SABnzbd, MeTube): the brief's gate rule — a download folder is never optional or mandatory — covers all five.
|
||||||
|
3. **No update-test edge was run:** the bench (guest 9401) and the drill catalog are outside the brief's machines and
|
||||||
|
repos. The list done and the list left are written; the ask is on the morning list.
|
||||||
|
4. The brief was written for a night run and ran by day; its „stop by 05:00" was read as a time box, not a clock.
|
||||||
|
|
||||||
|
## Part 2 — in plain words
|
||||||
|
|
||||||
|
- **Morning list:** vouch agent 0.161.0 · decide D21 (Immich) · MeTube's videos out of the second-drive copy — keep? ·
|
||||||
|
the bench and the drill catalog for the update test — yes?
|
||||||
|
- **Hub 0.162.0 is live.** Its own deploy was the real restart test: Tester-2 (down since 4 October) got no repeat mail
|
||||||
|
— read in the hub and, separately, in your mailbox.
|
||||||
|
- **Agent 0.161.0 runs on demo-hp, Tester 1 and demo-felhom.** On demo-hp the agent said „busy" during a backup and
|
||||||
|
„free" before and after. **Not vouched — that is yours.**
|
||||||
|
- **Installer 1.35.0 is published.** The served file equals the tag's file.
|
||||||
|
- **A failed off-site restore now puts the app back exactly as it was** (your D20 = A, controller 0.336.0, the floor).
|
||||||
|
Proven on the scratch box: after a forced failure the database matched its fingerprints from before, the app ran,
|
||||||
|
nothing was held; a normal restore then worked.
|
||||||
|
- **Downloads are out of the backups** (torrent, usenet, MeTube): the controller's own check copies 0 of them now.
|
||||||
|
- **Immich's off-site copy:** 1.33 GB, of which 102 MB must survive. Two options; pick B (leave out the AI model files).
|
||||||
|
- **ep0's first weekly clean-up:** done this morning (Part 10 of part 1 below).
|
||||||
|
|
||||||
|
## Part 2 — each part
|
||||||
|
|
||||||
|
| Part | Done? | Why / how |
|
||||||
|
|---|---|---|
|
||||||
|
| 1a hub deploy | **done** | `manifests/hub.yaml` → 0.162.0 (`e1d349f1`), only Deployment/hub synced (the only OutOfSync); image `sha256:69285e64…` = the build; Synced/Healthy; reports from all three boxes; floor read 0.335.0; Day-0 agent list has no `-step1`. R-1001 + R-1011 closed. |
|
||||||
|
| 1b agent release | **done** | R-1083 left out (above). `release-agent.sh 0.161.0` from a clean clone; signed `agent_update` per box: demo-hp 06:38, Tester 1 06:54, demo-felhom 06:55 UTC, each COMPLETED, 76/76, committed. `host_busy` live; tunnel `absent` = the fixture row (9202's real Docker 29 answer); R-1010 by its tests. R-1070 + R-1010 closed. |
|
||||||
|
| 1b R-921 live | **not done** | The no-stop case happens only on the night schedule (a press backs up the local tier only). Row R-1173 with a dated check after demo-hp's next off-site night. |
|
||||||
|
| 1c installer | **done** | tag `installer-v1.35.0`, both webpage pins (`b0ae2c57`), webpage Deployment synced alone; served sha256 `f7a196fd…` = tag. R-1167 closed. |
|
||||||
|
| 2 safe undo | **done** | below. R-893 + R-1169 closed. |
|
||||||
|
| 3 downloads | **done, one read left** | catalog `c50adfc`, gate `download-binds`; classifier proof. demo-felhom (qBittorrent) runs its next second-drive copy 2026-10-21 03:30 — dated check on R-1019 (NARROWED). |
|
||||||
|
| 4 Immich | **done (plan only)** | below. R-49 → READY, D21. |
|
||||||
|
| 5 update test | **not done** | fixture side complete for every installed app; the edges need the bench / drill catalog (above). R-462 NARROWED. |
|
||||||
|
| 6 ep0 clean-up | **done** | part 1, Part 10. |
|
||||||
|
| 7 controller release | **done** | 0.336.0 below. |
|
||||||
|
|
||||||
|
## Part 2 — the safe undo, the 9202 proof (`07` §6.3; `audits/night2-2026-10-20/P2/`)
|
||||||
|
|
||||||
|
- Code `93fb251`: the fit check before the stop, the copy right after it (`backups/restore-undo/<app>/`), the put-back on
|
||||||
|
every failure branch, the hold only when the put-back fails, the copy removed on success. 11 tests; nine mutations red.
|
||||||
|
- **9202, rc.1** (a helper with the fences in full; Rallly + postgres 18; the same forced failure as on 19 October):
|
||||||
|
|
||||||
|
| | Live before | After the forced failure |
|
||||||
|
|---|---|---|
|
||||||
|
| public tables | 36 | 36 |
|
||||||
|
| markers | A, B, C, D | A, B, C, D |
|
||||||
|
| `r893_only_old` (backup only) | absent | **absent** (on 19 October: present) |
|
||||||
|
| schema / data / row-count md5 | `05828c77` / `e5c1a72f` / `d8bcbc4b` | **equal** |
|
||||||
|
| app | running | **running**, no hold |
|
||||||
|
| the copy | — | 69 MB in 0.9 s, removed after |
|
||||||
|
|
||||||
|
The sentence: „…visszaállítása sikertelen. Nem változott semmi: az alkalmazás adatai visszakerültek a visszaállítás
|
||||||
|
előtti állapotba, és az alkalmazás újra fut". Then a normal restore of the same snapshot: the snapshot's content
|
||||||
|
(A–C, the backup-only table present), the app running, the copy removed. 9202 back on 0.331.0 as found.
|
||||||
|
- Not measured live (each pinned by a red-proved test): the version-change branch, the `_files` sentence, the refusals
|
||||||
|
before the stop, a failing put-back, the operator event (9202 has no hub).
|
||||||
|
|
||||||
|
## Part 2 — download binds (`15` §7; `audits/night2-2026-10-20/P3/`)
|
||||||
|
|
||||||
|
| App | Bind (userdata) | Class now | Copied by a tier |
|
||||||
|
|---|---|---|---|
|
||||||
|
| qBittorrent | media/torrents | excluded (was optional) | none (was the second drive) |
|
||||||
|
| Transmission | media/transmission, media/transmission/watch | excluded, excluded | none |
|
||||||
|
| NZBGet | media/nzbget | excluded (was optional) | none |
|
||||||
|
| SABnzbd | media/usenet | excluded (was optional) | none |
|
||||||
|
| MeTube | media/metube | excluded (was optional) | none |
|
||||||
|
| Sonarr, Radarr | media (rw) | excluded | none |
|
||||||
|
| Jellyfin, Plex, Emby | media (:ro) | excluded | none |
|
||||||
|
| Prowlarr | no userdata bind | — | — |
|
||||||
|
|
||||||
|
## Part 2 — Immich (`audits/night2-2026-10-20/P4/immich-offsite-plan.md`)
|
||||||
|
|
||||||
|
Model cache 824 MB (62 %) · raw database folder 283 MB (21 %) · Immich's own backups 115 MB (9 %) · logical dump 54 MB
|
||||||
|
· originals 49 MB · thumbnails 2 MB. **A** (leave out all three re-fetchable parts, −92 %) needs a never-run restore
|
||||||
|
from the dump alone on the vectorchord image; **B** (only the model cache, −62 %) needs one restore with an empty cache.
|
||||||
|
**Pick B now**, A's other parts after their proof.
|
||||||
|
|
||||||
|
## Part 2 — update-test edges
|
||||||
|
|
||||||
|
None run (above). Installed apps: 16; with a fixture: 15; without: bentopdf (keeps no data). Upstream within a major,
|
||||||
|
waiting for a tested step: PrivateBin 2.0.6→2.1.0, Home Assistant 2026.9.4→2026.10.2, RomM 5.3.1→5.4.0, Plex
|
||||||
|
1.41.4→1.43.4, Kimai 2.67.0→2.70.0 (`audits/night2-2026-10-20/P5/r462-list.txt`).
|
||||||
|
|
||||||
|
## Part 2 — versions, commits, rows
|
||||||
|
|
||||||
|
- Hub 0.162.0 deployed (`e1d349f1`). Agent 0.161.0 released (`5152123`, tag `v0.161.0`; notes `4e1621c`). Installer
|
||||||
|
1.35.0 published (`b0ae2c57`). Controller 0.336.0 the floor (`93fb251`; notes `aa9e09e`, `7ca59e2`), image
|
||||||
|
`sha256:93cd67d8…8d73`. Catalog `c50adfc`. Evidence `301f4e9e`, `bcaa9576`.
|
||||||
|
- i18n: no page template changed; seven Go-side keys (HU + EN), one rewritten (R-1169); no fixture changed.
|
||||||
|
- **System poster:** no listed fact changed — none of the three.
|
||||||
|
- **Register 160 before, 156 after. Closed 7:** R-1001, R-1011, R-1167, R-1070, R-1010, R-893, R-1169. **Opened 3:**
|
||||||
|
R-1171 (demo-hp's local copy never restore-tested), R-1172 (the put-back logs at ERROR; an empty folder stays),
|
||||||
|
R-1173 (R-921's no-stop not yet seen live). Re-stated: R-1019 (NARROWED, dated 2026-10-21), R-1083, R-462 (NARROWED),
|
||||||
|
R-49 (READY, D21). **Both briefs together: 161 before, 156 after; closed 11, opened 6.**
|
||||||
|
|
||||||
|
## Observations — part 2
|
||||||
|
|
||||||
|
1. demo-hp's local whole-guest copy is never restore-tested: one copy kept, a 24 h settle, replaced at 24 h. FILED: R-1171
|
||||||
|
2. A successful put-back is logged at ERROR, and an empty `backups/restore-undo/` folder stays after a restore. FILED: R-1172
|
||||||
|
3. R-921's no-stop branch has no live observation yet (it needs a night schedule). FILED: R-1173
|
||||||
|
4. R-462's „only three apps are tested" was out of date: 58 apps have fixtures on both venues. NOT-A-FINDING: the row is re-stated with today's list.
|
||||||
|
5. The hub deploy at 06:03 UTC removed the old pod's log, which held the 07:10 Budapest off-site audit lines of today (R-895). NOT-A-FINDING: R-895's dated check is 2026-10-21; that run happens in the new pod.
|
||||||
|
6. In step 4 of the 9202 drill the data-only md5 differed from a read taken before snapshot 2, while the schema and every row count were equal. NOT-A-FINDING: the two reads were at different moments (the read, then the snapshot's own dump); the claim under test — before and after the forced failure — matched on all three fingerprints.
|
||||||
|
7. The helper's brief expected 37 tables after the normal restore; snapshot 2 has 36. NOT-A-FINDING: last night's 37 was the mixed state, not the snapshot (my arithmetic, corrected).
|
||||||
|
8. The first background wait loop for demo-hp's agent update printed a time and stopped before the box had updated. NOT-A-FINDING: the version was read directly and a foreground loop used instead.
|
||||||
|
9. A Python edit of the controller CHANGELOG failed on a quote character, and the commit carried only REPORT.md. NOT-A-FINDING: fixed by the next commit (`7ca59e2`).
|
||||||
|
|
||||||
|
## Part 2 — teardown
|
||||||
|
|
||||||
|
- Machine: 9202 back on 0.331.0 as found (Rallly removed through the product, the SFTP target, images and files removed
|
||||||
|
by name, settings and the off-box folder put back); demo-hp 9201's probe script removed.
|
||||||
|
- Host: demo-hp `/tmp` copies removed; nothing on `pve/data`; no whole-guest backup of 9202.
|
||||||
|
- Hub: per-box floors set and cleared; the global floor raised; nothing else.
|
||||||
|
- ep0: read only.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# PART 1 — the evening/night brief of 2026-10-19
|
||||||
|
|
||||||
## In plain words
|
## In plain words
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,209 @@
|
|||||||
|
# REPORT — the night of 2026-10-21 → 22: make tomorrow's release safe
|
||||||
|
|
||||||
|
Unattended, 20:15 CEST → 08:00. Nothing released, deployed, published or vouched. Evidence:
|
||||||
|
`documentation/audits/night-2026-10-21/`. Design read first: `07` §6.3/§8 (the restore ladder, R-893 option A),
|
||||||
|
`03` §9 (guest sizing), `08` (notification groups), `10-localisation.md`, the controller CHANGELOG v0.336.0.
|
||||||
|
|
||||||
|
**Safest readings taken at the start (unattended):** (1) Tester4 is the soak customer — a new customer would provision on
|
||||||
|
ep0 and the Storage Box, which are read-only tonight; its old host was already deleted. (2) „One hub image" — so the
|
||||||
|
`node_down`-after-delete finding is a row, not a second image. (3) The website is published by a push, so W2 is a patch
|
||||||
|
file, not a commit. (4) The no-swap test ran after the box's last night job, not at 07:00 sharp (see Part 2).
|
||||||
|
|
||||||
|
## 1. Five lines
|
||||||
|
|
||||||
|
1. **Soak (Part 1): NO FREEZE — 3 of the 4 pass lines met, 1 missed.** 11 h on a fresh 8 GB box: no `/api/health` gap (largest interval 16 s), every night job ran, uploads complete except 1 photo of ~26 000 (R-1215); memory „full" pressure passed 20 % for ~3 min in the 03:30 second-drive job (peak 39 % host / 45 % guest over 10 s — R-1214).
|
||||||
|
2. **No-swap test (Part 2): THE FREEZE CAME BACK.** Host swap off, the same burst: the dashboard stopped answering within 20 s, dead up to 289 s at a time (24 of 125 probes failed), Immich lost 207 of 306 photos; it ended only when the Immich stream broke. Swap back on: normal at once. The agent warned once.
|
||||||
|
3. **D22 built and proven: YES** — the put-back covers the local, second-drive and kept restores and a restore cut by a
|
||||||
|
power loss; proven on 9202 (a damaged backup and a killed controller both kept the newest row and the app running).
|
||||||
|
4. **First-hour words: 22 fixed** (controller W9–W14, W21, W22, W24, W25, W28, W29; hub W3–W5; catalog W16–W19; ISO
|
||||||
|
source W1, W7); W6/W8 were already true; W2 is a ready patch (a publish).
|
||||||
|
5. **R-1199:** the hub mailed the household by a list saved two days before the box existed; the box never sent its own
|
||||||
|
— fixed in the controller (decided by CC, may be reversed). **R-1194:** a timing false alarm — the first report runs
|
||||||
|
before the emergency-door setup; fixed in the agent.
|
||||||
|
|
||||||
|
## 2. The morning list (copied from STATUS)
|
||||||
|
|
||||||
|
**Morning list, 2026-10-22.** CC decided one thing (you may reverse it): a box tells the hub the notification list its page shows, even if never saved (R-1199; the rehearsal household got 1 alarm of 9).
|
||||||
|
1. Release controller 0.338.0 (restore put-back incl. power cut, first-hour words, swap sentence)? On the scratch box a damaged backup and a killed controller both kept the newest data, app running (R-1177, R-1195, R-1178, R-1180).
|
||||||
|
2. Release and vouch agent 0.162.0 (swap = RAM, host-swap check, no false door alarm — R-1192, R-1194)? Soak, 11 h on a fresh 8 GB box: no freeze, every check answered, every night job ran; memory pressure passed the 20 % line for 3 min in the 03:30 backup (R-1214). With no host swap the freeze comes back; our installer gives 7.75 GB. I'd release.
|
||||||
|
3. Deploy hub 0.164.0 (host-swap alarm, bind-page words)? And delete the test host „Tester4-9aca27" (machine gone; my token cannot delete).
|
||||||
|
4. How was Tester 2's laptop installed — our installer, or its own Proxmox? (Without host swap the fix does not help.)
|
||||||
|
5. A publish: the website download button (patch ready), and with the next ISO the boot text and the cleared screen (R-1210). Secrets: keep or remove the `REHEARSAL_2026_10_20_*`, `FIXES_2026_10_21_*`, `NIGHT_2026_10_21_*` names in sec.
|
||||||
|
|
||||||
|
## 3. Part 1 — the soak (numbers)
|
||||||
|
|
||||||
|
Box: VM 9322 on demo-hp (4 vCPU, 8 GiB, 256 GB + 200 GB), installed from the public ISO, customer Tester4; agent fixture
|
||||||
|
`0.162.0-night.1` then `night.2` by hand (provenance `P1-soak/00-provenance.txt`). Two channels: the box's own 10-s
|
||||||
|
recorder (`03-rec.jsonl`, 3 632 samples) and a probe from DooPlex (`02-outside.txt`, 3 390 probes + demo-hp's view of the VM's
|
||||||
|
disk reads).
|
||||||
|
|
||||||
|
| | Load window 18:52–20:53 UTC | Night 20:53–04:05 UTC |
|
||||||
|
|---|---|---|
|
||||||
|
| health probes / failed | 684 / 0 | 2 449 / 0 |
|
||||||
|
| worst health gap (between answers) | 10–11 s (slowest answer 1.27 s) | **16 s** at 01:30:36 (slowest answer 4.07 s) |
|
||||||
|
| worst memory „full" (10 s, host / guest) | 31.2 % / 35.2 % (18:53:38, Immich model load; 1 sample) | **39.4 % / 45.5 %** (01:30:42, tier2 job; 12 samples > 20 %, ~3 min) |
|
||||||
|
| worst memory „full" (60 s) | — | 20.1 % / 25.0 % |
|
||||||
|
| OOM kills / guest limit hits | 0 / 0 | 0 / 0 |
|
||||||
|
| guest swap used (max) | 2 614 MB | 2 720 MB |
|
||||||
|
|
||||||
|
- **Guest swap raised:** 512 → 4096 MB by the fixture at 18:37:13 UTC (`04-agent-fixture.txt`).
|
||||||
|
- **Uploads:** Paperless 99 cycles × 53 PDFs, all accepted; Immich 87 cycles × 305 + the first upload — every user holds
|
||||||
|
303 photos (the set has 2 duplicates and 1 file Immich refuses by design), except cycle 36: 302 — **one photo hung 10 min
|
||||||
|
and was not stored** (19:38 UTC, pressure 3.9 %, health 0.26 s; Immich's log lost at its 02:16 restart — R-1215);
|
||||||
|
Nextcloud: both trees 3 031 of 3 033 files byte-identical to the manifest (sha256; the 2 are the uploader's by-design skips).
|
||||||
|
- **Night jobs (box log `10-controller-log.txt`):** db-dump 00:30 (4 DBs, 4 m 14 s), tier2 01:30 (4 m 15 s), off-site
|
||||||
|
02:15 (8 apps, 72 snapshots, 5 m 53 s), abandon-sweep 03:10, proof 03:30 (actualbudget PASSED), integrity 04:00
|
||||||
|
(1 m 22 s) — all completed. The agent's whole-guest backup ran at 18:41 UTC (24 h cadence; next due after the night).
|
||||||
|
- **Host swap the installer gives an 8 GB / 256 GB machine: 7.75 GB** (LVM `pve/swap`, SwapTotal 8 126 460 kB;
|
||||||
|
`02-host-swap-after-install.txt`).
|
||||||
|
- **Before (rehearsal, no fix):** „full" 96 %, dashboard dead for about an hour. **After:** no gap over 16 s.
|
||||||
|
|
||||||
|
## 4. Part 3 — D22, each scenario
|
||||||
|
|
||||||
|
| Scenario (9202, Rallly + Postgres, controller 0.338.0-rc.1) | Result |
|
||||||
|
|---|---|
|
||||||
|
| a normal local restore (the control) | unchanged: „1 adatkötet és az adatbázis visszaállítva", the expected rewind (A,B → A) |
|
||||||
|
| the unit's database volume archive damaged (cut to half) | put back: the live volume poured back, markers **A,C** (C = the newest live row), Rallly running, „…visszaállítása sikertelen. Nem változott semmi…" — on 0.331.0 (2026-10-20) the same case crash-looped Postgres and lost C |
|
||||||
|
| the controller killed (kill -9) right after the volume step, before the replay | on start: „interrupted restore is put back … 1 volume(s) poured back, definition=true", markers **A,C,D** (D newer than the backup), Rallly running, the record: „A visszaállítás félbeszakadt (áramszünet). Az alkalmazás adatai a visszaállítás előtti állapotban vannak." |
|
||||||
|
| off-site restore cut the same way | by test (`TestR1195_AKilledOffsiteRestoreIsPutBackOnStart`) |
|
||||||
|
| second drive: the unit refuses after the file half (R-1178) | by test: nothing moves, the app is never stopped |
|
||||||
|
| the night mirror during a restore (R-1180) | by test with the production rsync: skipped, mirrored the next night |
|
||||||
|
|
||||||
|
**Red-proofs seen failing, then passing:** `TestR1177_*` (4 failed on the old `restore_unit.go`: the old „completed with
|
||||||
|
data errors" text, no copy), `TestR1195_*` (the put-back at start switched off → „nothing poured back"; the off-site
|
||||||
|
journal not written → no undo), `TestR1195_TheRecordSays…` (old „indítsd el újra"), `TestR1178_*` (1 restored +
|
||||||
|
1 replaced), `TestR1180_*` ([a.jpg b.jpg]). Three older tests that pinned the old outcome were changed on purpose
|
||||||
|
(the app started on a half state is now held or put back). 9202 put back as found: controller 0.337.0, the same apps.
|
||||||
|
|
||||||
|
## 5. Part 2 — machines with no swap
|
||||||
|
|
||||||
|
**The test** (`P2-noswap/`): the soak box after its night, `swapoff -a` on the host at 04:06 UTC (48 s; it held 1.9 GB),
|
||||||
|
the burst from 04:08:25 to 04:38:27 (Immich machine-learning restarted so its models load again, as on the first afternoon),
|
||||||
|
then `swapon -a`. The first attempt (04:06–04:08) is VOID and kept: the load script restarted its names at 1, so Immich got
|
||||||
|
duplicates and Nextcloud skipped existing folders (`02-attempt1-VOID-duplicate-uploads.txt`); the rerun used fresh names.
|
||||||
|
|
||||||
|
| | No host swap (04:08–04:38) | Swap back on (04:38–04:44) |
|
||||||
|
|---|---|---|
|
||||||
|
| health probes / failed | 125 / **24** | 34 / 0 |
|
||||||
|
| gaps between answers | **289 s**, 172 s, 68 s | none |
|
||||||
|
| memory „full" (10 s, host / guest) | **36 % / 47 %** (94 of 180 samples > 20 %) | 1.2 % / 1.3 % |
|
||||||
|
| VM disk reads (demo-hp's view) | ~40 GB per 25 s at the worst | quiet |
|
||||||
|
| OOM kills | 0 | 0 |
|
||||||
|
| uploads | Immich 99 of 306, then its stream broke (a login answer that was not JSON); Paperless 52 of 53 in the first cycle, then 22 × 53; Nextcloud cut at the time limit | — |
|
||||||
|
|
||||||
|
**Said plainly:** without host swap the freeze comes back — the same shape as the rehearsal (refaults, disk reads, no OOM
|
||||||
|
kill), shorter only because the Immich stream died at 04:22 and took the load away. The agent logged its WARN once at the
|
||||||
|
swapoff („host_swap_mb=938 guest_swap_mb=4096").
|
||||||
|
|
||||||
|
**Built (commit only):** agent `0da792e` reports `host.host_swap_mb` (Proxmox node status = the kernel's SwapTotal — the
|
||||||
|
same number as `/proc/meminfo`; used instead of reading the file because the agent already holds the node status) and
|
||||||
|
`host.guest_swap_mb`, logs one WARN when short; `GET /guest/memory` carries both. Hub 0.164.0: operator-only
|
||||||
|
`host_swap_short` / `host_swap_ok`, edge-triggered from the host's newest swap event. Controller install page, when memory
|
||||||
|
is tight and the host's swap is below the guest's RAM: **HU** „A gépen nincs elég cserehely (swap), ezért nagy
|
||||||
|
feltöltéskor a doboz lelassulhat, és percekig sem válaszolhat. Szólj a Felhomnak, mi rendbe tesszük." **EN** „The machine
|
||||||
|
has too little swap space, so during large uploads the box can slow down and not answer for minutes. Tell Felhom; we will
|
||||||
|
fix it." Creating host swap on a BYO host is a decision → R-1207 (D27).
|
||||||
|
|
||||||
|
## 6. Part 5 — the two checks
|
||||||
|
|
||||||
|
**R-1199 (the household's mails went to the operator).** Hub DB read with `SELECT` on named non-secret columns only
|
||||||
|
(`P5-checks/r1199-01-hub-db-read.txt`): Tester4's stored list was saved **2026-10-18 18:18** — before the rehearsal box
|
||||||
|
existed — and holds `node_down, backup_failed, disk_critical, host_disk_critical, storage_fill_critical,
|
||||||
|
offbox_repo_orphaned, app_update_undone, app_update_held, app_stopped_unhealthy`: no `disk_warning`, no
|
||||||
|
`storage_disconnected`, no `crossdrive_failed`. The rehearsal day's log: the one household mail was `backup_failed`.
|
||||||
|
**Mechanism:** the box shows `settings.DefaultEnabledEvents` ticked when nothing was saved
|
||||||
|
(`controller/internal/settings/settings.go` `GetNotificationPrefs`), but the start-up sync pushes only when an address or
|
||||||
|
a clear was saved (`settings.go` `SyncOnStartup`, `cmd/controller/main.go` start-up goroutine), and the hub mails the
|
||||||
|
household only by its stored list (`hub/internal/notify/dispatcher.go` `processCustomer` → `isEventEnabled`). **Fix
|
||||||
|
(controller `ba22f9b`):** a box with no saved choice pushes the default list at start, with no address (the hub's
|
||||||
|
no-clobber guard keeps the stored one). The box's page is the source — `08` („the household decides"). *Decided by CC
|
||||||
|
unattended — operator may reverse.* `TestR1199_*` red-proved.
|
||||||
|
|
||||||
|
**R-1194 (emergency SSH door „broken" on a new box).** On the fresh soak box: the agent ran `sshd -t -f
|
||||||
|
/etc/felhom-sshd/sshd_config` **through sudo** at 20:33:59, the config was applied at 20:34:00.434, and the hub mailed
|
||||||
|
`oob_degraded` („config invalid") at 20:34 CEST; as the agent's user the check passes now (rc 0, port 8822 listening);
|
||||||
|
without sudo it fails on the host key — which the agent never does. **True candidate: the first report runs before the
|
||||||
|
felhom-sshd loop's first pass** (port 0, config never rendered) — a false alarm on every fresh box, operator key or not.
|
||||||
|
**Fix (agent `0da792e`):** no OOB stanza until the first pass has run (a failed pass counts). Red-proved.
|
||||||
|
|
||||||
|
## 7. Builds, commits, CI
|
||||||
|
|
||||||
|
| Artifact | Version | From | Digest / sha256 | State |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| controller | `0.338.0-rc.1` | `d582feb` | — | the D22 proof on 9202 |
|
||||||
|
| controller | `0.338.0-rc.2` | `f0abdd0` | `sha256:ae52c93ebb83af1c646986119a1b74b07f992ffb03017624d052ece6266ec748` | smoke on 9202 (pages 200, 0 error lines); 9202 back on 0.337.0 |
|
||||||
|
| hub | `0.164.0` | `6b0d23c2` | `sha256:931594e333a4e8a9784eff2fb144d022ab46f89dd6a865a93e6100dc28dbd9dc` | built, pushed, NOT deployed (manifest 0.163.0) |
|
||||||
|
| agent | `0.162.0` candidate | `0da792e`, clean clone, release flags | `5fd7a23bfe319cb521c4bc399612d5fb3413f1e695f79bc663c386f69c757928` | not published; fixtures `night.1` (`286a7e3`) and `night.2` (`0da792e`) ran on the soak host |
|
||||||
|
|
||||||
|
**Commits.** controller: `d582feb`, `37b79ca`, `8c1e290`, `28b05d6`, `5776d54` (the helper's three: W9–W29), `ba22f9b`,
|
||||||
|
`f0abdd0`, `1de8a55`. agent: `0da792e`, `b310773`. felhom.eu: `3c620f8d`, `f90e2229` (hub), `2fa5b8f3` (ISO source),
|
||||||
|
`6b0d23c2`, `2293b252`, `9d3bf76d`, `57c06ba9`, `307d25b1`, `f576d7f7`, `c2c130ec`, `dee26da3` + this report.
|
||||||
|
catalog: `13f9c9c` (live through the catalog sync). **CI:** controller 2175, 2178, 2181, 2184, 2186, 2187, 2188, 2191 —
|
||||||
|
all success; agent 2176, 2190 — success; catalog 2180 — success; felhom.eu 2177–2192 **failed on one suite only**
|
||||||
|
(`test_gate_decoys.py` hit the 300 s limit on the CI runner; R-1209, fixed in `c2c130ec`) — after the fix: `c2c130ec` 2193, `dee26da3` 2194, `97f088ec` 2195 — all success; the commit carrying this report is checked after its push (the final note)..
|
||||||
|
|
||||||
|
## 8. Teardown (three layers)
|
||||||
|
|
||||||
|
- **Machine:** VM 9322 `night1021-soak` and CT 9332 `night1021-loader` destroyed on demo-hp (names checked first) at
|
||||||
|
~04:47 UTC; their disk folders on nvme-scratch are gone (used 301 → 208 GB). The thin pool `local-lvm` read the same before
|
||||||
|
and after (26 159 563 KiB used) — never written. The public ISO on `local` was there before and stays. demo-hp's own clock
|
||||||
|
and swap untouched. The night's evidence was copied off first (logs, recorder, Immich counts).
|
||||||
|
- **Host (hub):** the host `Tester4-9aca27` stays on the hub (agent `0.162.0-night.2`); CC's token cannot delete a host —
|
||||||
|
**the operator deletes it** (morning list 3). Until then it reads down. The customer Tester4 stays (the kept one).
|
||||||
|
- **Off-site:** ep0 (read only) holds for Tester4 one copy, `ct/9201/2026-10-20T12:17:26Z`, from the rehearsal box —
|
||||||
|
nothing from tonight; nothing removed. The Storage Box holds Tester4's off-site repository with tonight's 72 snapshots —
|
||||||
|
left (a new Tester4 box sets it aside by design).
|
||||||
|
- **9202:** back on controller 0.337.0 with the apps it had; the D22 test app removed through the product.
|
||||||
|
|
||||||
|
## 9. Rows
|
||||||
|
|
||||||
|
Before **176**; after **184**; opened **9** (R-1207 host swap on a no-swap host — D27; R-1208 `node_down` after an
|
||||||
|
operator delete; R-1209 the CI time limit — closed the same night; R-1210 first-hour words waiting for a publish; R-1211
|
||||||
|
the household `backup_failed` mail; R-1212 two texts ignoring the box language; R-1213 operator detail left on two pages; R-1214 memory pressure in the night's second-drive job; R-1215 one Immich upload hung);
|
||||||
|
closed **1** (R-1209). Moved to WAITING-ON-OPERATOR (fixed on main, ship with the morning releases): R-1177, R-1178,
|
||||||
|
R-1180, R-1195 (controller), R-1194 (agent). R-1188 (a) fixed. `unproven.py --summary`: unchanged, 35 of 55 not walked.
|
||||||
|
|
||||||
|
## 10. Observations
|
||||||
|
|
||||||
|
- The decoy suite timed out on CI on every felhom.eu push of the night — **FILED: R-1209** (fixed, closed).
|
||||||
|
- `node_down` mailed for a customer whose host the operator had deleted — **FILED: R-1208**.
|
||||||
|
- A household `backup_failed` mail with no app and no step — **FILED: R-1211** (the helper's).
|
||||||
|
- Two backup texts ignore the box language — **FILED: R-1212**. Operator detail on two pages — **FILED: R-1213**.
|
||||||
|
- `TestR603_*` was red on controller main since 0.337.0 (CI runs gates, not `go test`) — **NOT-A-FINDING:** fixed in
|
||||||
|
the session (two English apostrophes), small.
|
||||||
|
- The first `resend` press on a spent link before the new box registered sent nothing — **NOT-A-FINDING:** a live link
|
||||||
|
existed (minted at the host delete, mailed 16:49); the page says „if it was a real link" by design.
|
||||||
|
- The helper chose different words than the brief for W25 („a következő éjjel", the retry is the same night when pressed
|
||||||
|
by day) and W29 („Leszerelve. Nem költöztettünk át adatot róla; ami rajta volt, a meghajtón maradt." — „adat nem volt
|
||||||
|
rajta" would be false) — **NOT-A-FINDING:** the brief's texts were examples; the reasons are in the CHANGELOG.
|
||||||
|
- On the installed app pages Mealie already showed the right address (R-498); W19 was wrong only on the install page —
|
||||||
|
**NOT-A-FINDING** beyond the catalog fix.
|
||||||
|
- An empty `backups/restore-undo/` folder stays after a put-back (only the per-app folder is removed) —
|
||||||
|
**NOT-A-FINDING:** harmless, no data.
|
||||||
|
- The soak load script's Nextcloud stream ran past the 2 hours (each stream finishes its current pass) —
|
||||||
|
**NOT-A-FINDING:** harness.
|
||||||
|
|
||||||
|
- Memory „full" above 20 % in the night's second-drive job (~3 min) — **FILED: R-1214**. One Immich upload hung and was
|
||||||
|
not stored — **FILED: R-1215**.
|
||||||
|
- Without host swap the freeze comes back — **recorded on R-1207** (the operator's D27), not a new row.
|
||||||
|
- One `execCommand failed: exit status 1` at 00:30:49 — **NOT-A-FINDING:** the memory check's `docker inspect` of all
|
||||||
|
containers ran 1 s after actualbudget was stopped for its dump; the code keeps the other answers by design.
|
||||||
|
- One Radicale health probe timed out at 01:30:24 (the second-drive job starting) — **NOT-A-FINDING:** the next probe passed,
|
||||||
|
no alarm, the household's door answered throughout.
|
||||||
|
- The guest hit its memory limit 101 times at 18:39 UTC, during the app installs, before the load — **NOT-A-FINDING:** page
|
||||||
|
cache reclaim (0 OOM kills, swap 40 KB used), never again in 11 h.
|
||||||
|
- A 637 MB `vzdump-lxc-9100-…` on the fresh box's `local` — **NOT-A-FINDING:** the installer's golden archive (9100 = the
|
||||||
|
golden build guest, `felhom-host-install.sh`); the restore test skips it with a reason.
|
||||||
|
- The byte-check tool `tools/nc_compare.py` turns TLS checking off — **NOT-A-FINDING:** a measurement script against a
|
||||||
|
throwaway box with a self-signed certificate and a test domain that does not resolve; the box is destroyed.
|
||||||
|
- The first no-swap attempt reused the soak's names (Immich duplicates, Nextcloud skips) — **NOT-A-FINDING:** harness;
|
||||||
|
stopped after 2 min, kept as VOID, rerun with fresh names (`tools/load2.py` now takes a name tag).
|
||||||
|
- CC ran an old one-off register script by mistake at 04:07 UTC; it added 16 duplicate rows to the register file —
|
||||||
|
**NOT-A-FINDING:** caught at once by the row count, the file restored from git before anything was committed; the
|
||||||
|
script renamed so it cannot run again.
|
||||||
|
**Fixed without a row:** the R-603 test, the restore-record log line after a put-back, the CHANGELOG blank line.
|
||||||
|
**Secrets:** names only — `NIGHT_2026_10_21_ROOTPW`, `NIGHT_2026_10_21_PAIRING_CODE`, `NIGHT_2026_10_21_BIND_TOKEN`,
|
||||||
|
`NIGHT_2026_10_21_SETUP_CODE` (the last three spent). The bind link and the setup code were read from the mailbox through
|
||||||
|
the mail connector, so both one-time values entered this session's record (spent within minutes). A scan of the evidence
|
||||||
|
for every secret used tonight: 0 hits — all 34 secret names of the night, the rehearsal and the fixes run, across the 75 evidence files, this report, STATUS, CONTEXT and the register (positive control: a known word in the evidence was found). **Poster:** no fact in `felhom-system-poster.facts.md` changed (nothing
|
||||||
|
released). **Instruction files:** none edited.
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
# REPORT — system poster refresh check + OMV guide fixes (2026-10-21)
|
||||||
|
|
||||||
|
Baseline `main` = `7a773207`. Own clone; documentation only; no machine, box or release touched.
|
||||||
|
Binding: `documentation/architecture/felhom-system-poster.facts.md`.
|
||||||
|
|
||||||
|
## Plain words
|
||||||
|
|
||||||
|
- **The new poster is in the repository**, the web page and the PDF. It took three exports.
|
||||||
|
- The first export was a folder of 6 files, and Claude Design had changed four texts. Nothing was committed then
|
||||||
|
(commit `3decdf94`, with the OMV guide fixes and R-1201).
|
||||||
|
- The second export was one file; two texts were still short. The third export passed every check.
|
||||||
|
- The STATUS line „System poster needs a refresh" is gone. R-1201 is closed.
|
||||||
|
|
||||||
|
## Part A — checks
|
||||||
|
Full table, all three rounds: `documentation/audits/poster-refresh-2026-10-21/CHECKS.md`.
|
||||||
|
Round 3: file PASS · shape: plain static HTML (better than the old loader) · two changes PASS · nothing lost PASS ·
|
||||||
|
secrets PASS · look PASS (PDF; HTML render not checked, no browser on DooPlex).
|
||||||
|
|
||||||
|
## Part B — committed (second commit)
|
||||||
|
- `documentation/architecture/felhom-system-poster.html` replaced; `felhom-system-poster.pdf` added (6.25 MB, A1 size).
|
||||||
|
- Facts file: one History line.
|
||||||
|
- STATUS: the poster line removed. R-1201 moved to `CLOSED-ITEMS.md`.
|
||||||
|
- Evidence: round-3 hashes, text diff, three PNG crops.
|
||||||
|
|
||||||
|
## Part C — `documentation/pilot/GUIDE-omv-kvm-hu.md` (first commit, `3decdf94`)
|
||||||
|
1. „Ha valami nem megy" item 7 (HDD_PATH error) removed; 8 → 7.
|
||||||
|
2. Jellyfin: „Ha a géped egy korábbi telepítővel…" sentence removed.
|
||||||
|
3. Memory bullet: „Az első napokban kevés alkalmazással indulunk; a nagy fotógyűjtemény feltöltését együtt csináljuk." added.
|
||||||
|
4. Operator note: dated line, point (4) obsolete since installer 1.33.0; history kept.
|
||||||
|
|
||||||
|
Re-checked: `scripts/iso/felhom-bootstrap.sh:165` fetches `https://felhom.eu/scripts/felhom-host-install.sh`, which
|
||||||
|
serves `SCRIPT_VERSION="1.35.0"` (≥ 1.33.0). `https://felhom.eu/letoltes` lists `felhom-installer-1.29.0-pve9.2-1.iso`
|
||||||
|
and `dceacae5…4829`, the same as the guide's step 3.
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
Opened R-1201 (first commit), closed R-1201 (second commit). Register 178 → 179 → 178.
|
||||||
|
|
||||||
|
## Claims in the task that were wrong
|
||||||
|
- The PDF is not A2: 2400×1696.5 pt ≈ A1 size.
|
||||||
|
- `claude/DESIGN-poster-refresh-2026-10-21.md` is not in this repository: `claude/` is the planning project's folder
|
||||||
|
(as for `claude/MONTHLY-security-retest.md`).
|
||||||
|
- felhom.eu has no root `CHANGELOG.md`; this repo records sessions in `REPORT-<topic>.md` files, so this is one.
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
- The pre-push hook refuses a clone outside `/mnt/5_hdd/felhom.eu`; the clones were moved under it. No `--no-verify`.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
Provisioned nothing. Scratch copies and clones deleted. `~/temp/` untouched.
|
||||||
@@ -0,0 +1,158 @@
|
|||||||
|
# REPORT — Tester-2 rehearsal with a pretend household + an independent data-loss hunt (2026-10-20 → 21)
|
||||||
|
|
||||||
|
> Run 2026-10-20 09:30 UTC → 2026-10-21 ~09:50 UTC. Drafted before the night, finished in the morning. This repo's
|
||||||
|
> shared `REPORT.md` was not touched: a parallel session works in the same clone, and the repo rule says the second
|
||||||
|
> session writes `REPORT-<topic>.md` only (the brief asked for both; this line is why one was skipped).
|
||||||
|
> Evidence: `documentation/audits/rehearsal-2026-10-20/` (journal, A0–B10, screens) and
|
||||||
|
> `documentation/audits/dataloss-hunt-2026-10-20/` (C1–C3). Architecture read for the claims: `07-backup-architecture.md`,
|
||||||
|
> `10-localisation.md`, `00-capability-map.md` (A15).
|
||||||
|
|
||||||
|
## Five sentences
|
||||||
|
|
||||||
|
A pretend household installed a box from the public ISO, linked it, claimed it, put eight apps and about 2 GB of family
|
||||||
|
data on it, used it for a day, and every customer restore path brought its data back. The first night's backups, the
|
||||||
|
clock change, three days off and power cuts in the middle of installs and catch-ups all worked, and the morning data was
|
||||||
|
byte-identical to the evening's. But the 8 GB minimum box froze twice on its first afternoon and only a plug pull helped
|
||||||
|
(R-1192), and the household's warning mails went to the operator instead (R-1199). When the box died, the household could
|
||||||
|
not even start again without the operator, and after the operator's press the rebuilt box could not reach its off-site
|
||||||
|
history at all — proven live (R-1176, P1). Nothing was released; 29 new register rows carry the rest.
|
||||||
|
|
||||||
|
## The ten places a household gets stuck
|
||||||
|
|
||||||
|
`documentation/audits/rehearsal-2026-10-20/B10-stuck/README.md`. In short:
|
||||||
|
|
||||||
|
1. The box froze twice on the first afternoon (≈1 h and ≈35 min); only pulling the plug helped (R-1192, P1).
|
||||||
|
2. The bind page needs an owner passphrase the household does not have — 2 h 4 min waiting for the operator.
|
||||||
|
3. A power cut during a full off-site restore leaves the app on the backup's older data; the page only says „interrupted" (R-1195).
|
||||||
|
4. A pulled drive breaks the last uploads (0 bytes or gone) although each was confirmed (R-1196).
|
||||||
|
5. The family's warning mails went to the operator, not the family (R-1199).
|
||||||
|
6. „Ödön" and „Írisz" are refused as family names (W12).
|
||||||
|
7. A new drive needs a „mount name" nobody knows; the button silently does nothing (W13).
|
||||||
|
8. Sharing waits for ever for a password the page never asks for (W21).
|
||||||
|
9. Errors arrive as raw technical English (W22, W25).
|
||||||
|
10. After the box died, the household was promised a mail that never came, and after the operator's press the rebuilt box
|
||||||
|
could not reach the family's off-site data (R-1203, R-1176). (Close behind: three apps' wrong „first steps", W17–W19.)
|
||||||
|
|
||||||
|
## Data-loss findings, ranked
|
||||||
|
|
||||||
|
| row | Sev | harm | reproduced | design / defect |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| R-1175 | P1 | shares on two drives sharing one second drive: every night one drive's share copy is pruned; the night a drive dies its copy is deleted | unit test on production code (`C3-D1-shares-tier2/`) | defect |
|
||||||
|
| R-1176 | P1 | a box rebuilt after „delete the old host" cannot reach its off-site history: no recovery code asked, a new key, the old history set aside on night one | hub store unit test (`C3-K1-…`) + **LIVE in B7** (2026-10-21) | defect |
|
||||||
|
| R-1191 | P1 | a `.fab` import makes a family-only app open to anyone | LIVE on scratch 9202 | defect (security) |
|
||||||
|
| R-1192 | P1 | the 8 GB box froze under first-day use; two plug pulls | LIVE on Tester4 (twice) | defect — cause not diagnosed (no shell on the box) |
|
||||||
|
| R-1177 | P2 | a failed local restore destroys the live data, keeps no copy | LIVE on 9202 | design question (D22 in STATUS) |
|
||||||
|
| R-1195 | P2 | a power cut mid full off-site restore: app on the backup's data, page says only „interrupted" | LIVE on Tester4 (B6 #2) | defect |
|
||||||
|
| R-1178 | P2 | second-drive „whole restore" can change files, refuse the DB, leave the app stopped and say „untouched" | unit test | defect |
|
||||||
|
| R-1180 | P2 | the nightly mirror (`rsync --delete`) can run during a restore from it | unit test | defect |
|
||||||
|
| R-1181 | P2 | a missing app folder deletes its second-drive copy in one night (the only copy for optional folders) | unit test | defect (D23 in STATUS) |
|
||||||
|
| R-1179 | P2 | a drive marked „away" can still be the second-drive target | code + unit test | defect |
|
||||||
|
| R-1182 | P2 | abandoning one set-aside off-site copy deletes every retained key package | code | defect |
|
||||||
|
| R-1199 | P2 | warning mails reach the operator, not the household | LIVE (two mailboxes) | defect |
|
||||||
|
| R-1204 | P2 | a box that ran once with its clock ahead loses its WireGuard tunnel to ep0 (PBS copies + operator door) until that date | LIVE (B6 #8, ep0 `wg show`) | defect |
|
||||||
|
| R-1203 | P2 | after a box death, the household is promised a link mail that never comes until the operator deletes the old host | LIVE (B7) | defect |
|
||||||
|
| R-1185 | P3 | a dump that fails its own check has already replaced the good one | unit test (docker stub) | defect (order) |
|
||||||
|
| R-1186 | P3 | the nightly off-site copy is skipped silently when another backup holds the flag | code | defect |
|
||||||
|
| R-1184 | P3 | a cut-off shares restore can be reported restored with files missing | code | defect |
|
||||||
|
| R-1196 | P3 | uploads confirmed just before a drive pull are 0 bytes or gone | LIVE (B6 #4) | defect |
|
||||||
|
| R-1183 | P3 | on 2026-10-25 every hub daily job runs twice (DB snapshot keep-2 pruned a day early) | replay of the hub's arithmetic | defect (D24 in STATUS) |
|
||||||
|
| R-1200 | P3 | an afternoon install gets no whole-box backup on night one | LIVE (B5) | design question |
|
||||||
|
| R-1202 | P3 | a box with its clock behind stops an app silently and only says „Hub nem elérhető" | LIVE (B6 #8) | defect |
|
||||||
|
| R-1187 … R-1190 | P3 | the unproven candidates of the four reviewers, grouped | code reading | mixed |
|
||||||
|
|
||||||
|
## Failure scenarios (B6)
|
||||||
|
|
||||||
|
| # | scenario | result | evidence |
|
||||||
|
|---|---|---|---|
|
||||||
|
| ★1 | power cut mid app install | PASS — clean rollback, honest event, reinstall in 15 s | `B6-failures/s1-…` |
|
||||||
|
| ★2 | power cut mid off-site restore | 6 s: safe; 14 s: app on the backup's data, page „interrupted" → R-1195 | `s2-…` |
|
||||||
|
| ★3 | data disk 100 % during backup | PASS — warned at 89 %, refused before writing, healed; SMB error unclear; system disk 97 % not run | `s3-…` |
|
||||||
|
| ★4 | data drive pulled, put back | apps stopped in 42/34 s, back by themselves; last uploads 0 B/missing → R-1196 | `s4-…` |
|
||||||
|
| ★5 | slow lossy line, first big off-site | run failed under 1 Mbit + 5 % loss, raw English; second press OK 3 m 38 s | `s5-…` |
|
||||||
|
| ★6 | off-site blocked | failed in 2.5 min, raw English; catch-up OK 2 m 37 s (shortened, not a night) | `s6-…` |
|
||||||
|
| ★7 | DST | PASS — the database copy ran once, at the second 02:30; W30 (no summer/winter mark in the repeated hour) | `s7-dst/` |
|
||||||
|
| ★8 | wrong clock at boot ±2 y | FINDINGS — −2 y became −6 months (the OS floor): „Hub nem elérhető" only, Paperless stopped silently (R-1202); +2 y: a whole-box archive dated 2028 (R-1189 a), then the ep0 tunnel dead (R-1204) | `s8-wrong-clock/` |
|
||||||
|
| ★9 | two actions at once | PASS — second action refused with a clear sentence | `s9-…` |
|
||||||
|
| ★10 | box off 3 days (+#15 restart in the catch-up) | PASS — ONE catch-up, a clear note to the household, the interrupted catch-up recovered | `s10-three-days-off/` |
|
||||||
|
| 11 | hard reset mid app update | NOT RUN — no update was offered to the box (nothing was released during the run) | — |
|
||||||
|
| 12 | DNS dead 1 h | PASS for data; operator told after ~37 min, household never | `s12-…` |
|
||||||
|
| 13 | second drive added later | not offered as the backup place → R-1198; safe removal OK; blank texts (W29) | `s13-…` |
|
||||||
|
| 14 | remove keeping data → reinstall „use kept data" | PASS — 47/47 documents, 47/47 originals | `s14-…` |
|
||||||
|
| 16 | English browser / English dashboard | follows the household's setting (by design); a few Hungarian strings → fixed W26 (unreleased) | `s16-…` |
|
||||||
|
|
||||||
|
## Restore paths (B8)
|
||||||
|
|
||||||
|
All seven customer paths PASS on the live box (`B8-restores/README.md`): A.1.1 Mealie unit 17 s; A.1.2 Nextcloud from
|
||||||
|
the second drive 3/3 sha256; A.1.3 Radicale verify copy 5 s; A.1.4 Nextcloud missing-only 3/3 sha256 (NFC/NFD: both
|
||||||
|
names come back on the share; Nextcloud itself folds NFD into NFC); A.1.5 BookStack full off-site 30 s; A.1.6 shares
|
||||||
|
2 files; A.1.7 `.fab` Mealie. Gaps: the messages do not say which point in time or how many files.
|
||||||
|
|
||||||
|
## Night chain and DST
|
||||||
|
|
||||||
|
**The first night (B5), no hands:** database copy 00:30Z PASS (3 m 27 s); second-drive copy 01:30Z PASS (8 apps + the
|
||||||
|
share, 58 s); off-site 02:15Z PASS (44 snapshots, 3 m 6 s, nothing deleted; the hub's own listing agrees: 44 files,
|
||||||
|
newest 02:18Z); whole-box backup NOT RUN — not due under the 24 h rule, because the install's own backup at 12:09Z
|
||||||
|
started the clock (R-1200, P3: the family's first day reaches a whole-box copy only on the second night). At 02:46Z
|
||||||
|
demo-hp's own kernel update rebooted the test host and shut the rehearsal VM down (harness, not a Tester4 defect; a BYO
|
||||||
|
host skips that step by design). Health 200 all night before that; memory steady. **The morning data snapshot is
|
||||||
|
identical to the evening one, byte for byte** (`B5-night/README.md`).
|
||||||
|
|
||||||
|
**DST (#7), live:** the box was started with its clock at 2026-10-25 02:20 CEST and left alone: no database copy at the first 02:30, one at the second (01:30:00Z, 1 m 55 s). The code replay agrees for the box; the hub runs every daily job twice on that day (R-1183, D24) and spring 2027 puts the 02:30 dump on the 03:30 copy (`B6-failures/dst-replay/`).
|
||||||
|
|
||||||
|
## Interventions (operator-only acts)
|
||||||
|
|
||||||
|
| # | when (UTC) | what | minutes the household waited |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | 09:57 → 12:00:57 | the owner passphrase handed over (by design, out of band) | 124 |
|
||||||
|
| 2 | asked 09:35, done 12:01–12:15 | off-site switched on for Tester4 | 0 (done before the household needed it) |
|
||||||
|
| 3 | asked 08:59, done ~09:11 (2026-10-21) | delete host Tester4-816e21 before the reinstall (B7) | ~12 (and 17 min of a dead end before it, R-1203) |
|
||||||
|
|
||||||
|
Not counted as interventions: the two plug pulls (14:11, 16:13) — a household can do that, and did.
|
||||||
|
|
||||||
|
## Versions
|
||||||
|
|
||||||
|
controller 0.336.0 (`7ca59e2`, floor), agent 0.161.0 (`4e1621c`), hub 0.162.0, golden 0.331.0, min_agent 0.158.1,
|
||||||
|
installer 1.35.0, ISO 1.29.0 (sha256 `dceacae5…4829`), catalog `c50adfc6` (65 templates). Nothing was released, deployed,
|
||||||
|
vouched, baked or published during the run. One text fix was committed, not released: controller `01c967e` (W26).
|
||||||
|
|
||||||
|
## Browser spike (A4)
|
||||||
|
|
||||||
|
Works: Chromium 155 + Playwright 1.52 in a throwaway LXC on demo-hp, phone and computer, HU and EN, in ~12 minutes.
|
||||||
|
121 screenshots. Limits: no real DNS/TLS for the household's address; not a human. `A4-browser/README.md`.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
Machine: VM 9320, CT 9330 and the ISO removed from demo-hp by their own IDs; no drill firewall rules left; memory back
|
||||||
|
to the start. DooPlex: three watchers stopped by PID. Hub: **host Tester4-648b80 is left for the operator to delete**
|
||||||
|
(the customer Tester4 stays). ep0: one PBS snapshot `ns/Tester4/ct/9201/2026-10-20T12:17:26Z` and the WireGuard peer
|
||||||
|
10.77.0.6 remain; the Storage Box keeps the household's restic history — listed, nothing removed (`D-teardown/`).
|
||||||
|
|
||||||
|
## Secrets (names only)
|
||||||
|
|
||||||
|
All in `sec`, prefix `REHEARSAL_2026_10_20_`: `9202_PW`, `9202_RESETCODE`, `APP_BOOKSTACK`, `APP_RADICALE`,
|
||||||
|
`APP_VAULTADMIN`, `BIND_TOKEN`, `BOOKSTACK_TOKEN`, `DASH_PW`, `FAMILY_IRISZ`, `FAMILY_KATA`, `FAMILY_ODON`,
|
||||||
|
`INSTALL_ROOTPW`, `IRISZ_PW`, `KATA_PW`, `ODON_PW`, `OLD_BIND_TOKEN`, `PAIRING_CODE`, `PASSPHRASE`, `RECOVERY_CODE`,
|
||||||
|
`SETUP_CODE`, `SMB_PW`. Kept (the operator decides); none written anywhere else. Evidence scanned for their values:
|
||||||
|
0 hits.
|
||||||
|
|
||||||
|
## Register
|
||||||
|
|
||||||
|
Before 153 open rows; now 182. Opened by this run: 29 (R-1175 … R-1200, R-1202, R-1203, R-1204). Closed: R-1019 (a dated
|
||||||
|
check read on demo-felhom, read-only). One row (R-1201) came from a parallel session. Updated: R-1176 (proven live),
|
||||||
|
R-1189 (a half-live), R-895 (first verdict), R-435 (moved). Capability map: A15 still PARTIAL with new evidence; A12 and
|
||||||
|
A13 PROVEN-LIVE → PARTIAL (B7). `unproven.py --summary`: unchanged (35 of 55 not walked).
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
FILED: everything in the tables above. NOT-A-FINDING: the dashboard ignores an English browser (by design: the
|
||||||
|
household's setting wins); Nextcloud folding NFD into NFC (upstream behaviour; both forms survive on the share); the
|
||||||
|
films' off-site size (CBR filler compresses; nothing missing); demo-hp's own kernel reboot at 02:46 (a harness effect: a
|
||||||
|
BYO host skips that step by design, `osupdate/leg.go:674`). FILED: everything in the tables above.
|
||||||
|
|
||||||
|
**Harness slips (mine), each recorded in the journal:** a screenshot step blanked a form before submit (15 min lost); the
|
||||||
|
two watcher logs froze at 20:24 when `git pull --autostash` replaced their files (recovered in full from the open files);
|
||||||
|
my first #10 start used a date the #7 records had made meaningless (restarted, recorded before any result). **Secrets:**
|
||||||
|
the nightly DooPlex sweep found three rehearsal secret values in my session record (the old and new bind tokens and a
|
||||||
|
setup code — I had read their mails in full and typed one into a `sec put`); in B7 a second bind link and setup code
|
||||||
|
entered the record the same way. All are one-time values, spent within minutes; nothing secret is in any committed file
|
||||||
|
(323 files scanned, 0 hits; pictures checked by eye; the B7 pairing code was blacked out).
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
# REPORT — the status page back to one screen, and a gate that keeps it there (2026-10-20, day)
|
||||||
|
|
||||||
|
## In plain words
|
||||||
|
|
||||||
|
The status page was 107 lines and 31,695 bytes; now it is 50 lines and about 5,500 characters outside the decide table,
|
||||||
|
which holds the 8 decisions that are still open. The „do" list went from about 25 items to 7 „still needed? (yes/no)"
|
||||||
|
questions, 2 morning-list items and 5 other lines (plus one „Later" line of row ids); CC checked the rest in the hub
|
||||||
|
and on DooPlex, and 7 were already done. **You answer:** the „Still needed? (yes/no)" list below, one word per line.
|
||||||
|
|
||||||
|
## Still needed? (yes/no) — copied from STATUS.md
|
||||||
|
|
||||||
|
1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). Since 2026-10-16.
|
||||||
|
2. The N100 sheet, about 30 minutes: Plex, the admins in Prowlarr, Sonarr and Radarr, your download sources
|
||||||
|
(`documentation/audits/n100-lived-in-2026-10-15/SHEET.md`). Since 2026-10-15.
|
||||||
|
3. Look at these pages on demo-hp when you have time: Calibre-Web („Adatok és mentés"), the file manager, AdventureLog,
|
||||||
|
Plex, the app page, the Plex and Sonarr guides („Útmutató"). Since 2026-10-16.
|
||||||
|
4. Remove the test server „plex-k6" from your Plex account (plex.tv → Settings → Authorized Devices). Since 2026-10-15.
|
||||||
|
5. Ask Google to index `https://felhom.eu/sajat-felho-tarhely` in Search Console (R-1035). Since 2026-10-14.
|
||||||
|
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable.
|
||||||
|
Since 2026-10-14.
|
||||||
|
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
|
||||||
|
|
||||||
|
## What CC checked, and how (method: the hub's GET pages with CC's scoped token, from the running pod; read-only local checks on DooPlex)
|
||||||
|
|
||||||
|
| „Do" item | Verdict | Evidence |
|
||||||
|
|---|---|---|
|
||||||
|
| Vouch agent 0.161.0 | **done** — the Day-0 manifest names 0.161.0 with the asked sha256 | `documentation/audits/status-view-2026-10-20/hub-reads.txt` |
|
||||||
|
| Delete the test customer Tester3 | **done** — not in the customer overview | same |
|
||||||
|
| Discard registrations 40 and 41 | **done** — the Hosts page has no „Unclaimed appliances" section (only `registered`/`bound` rows render) | same |
|
||||||
|
| Delete Tester4's old host `Tester4-31fa0e` | **done** — not on the Hosts page | same |
|
||||||
|
| Say „tear it down" for Tester 1's media tests | **done** — R-1022 closed 2026-10-17 | `CLOSED-ITEMS.md` |
|
||||||
|
| Hub master key into the password manager, then shred the file | **done** — `new.key` is gone; its folder is there (control) | `…/local-checks.txt` |
|
||||||
|
| Hub 0.153.0 „no token can …" | not a task — a statement; moved to history | — |
|
||||||
|
| Pull the failed disk Z6A07P2G | **not done** — still in DooPlex (lsblk; its twin is the control) | `…/local-checks.txt` |
|
||||||
|
| The Facebook `first-post/` leftovers | **not done** — 15 files still there | same |
|
||||||
|
| The `deny`-list edit in homelab-manifests | **not done** — still uncommitted | same |
|
||||||
|
| Old „broken" line: a Longhorn volume at 91 % | **gone** — Alertmanager lists only the two R-934 disk alarms | same |
|
||||||
|
| Old „broken" line: CI jobs get lost (R-887) | **gone** — R-887 closed 2026-10-19 (381 jobs, 0 lost) | `CLOSED-ITEMS.md` |
|
||||||
|
|
||||||
|
Every decided row (D1, D2, D10–D20) left the decide table only after its ruling was found in its closed row in
|
||||||
|
`CLOSED-ITEMS.md`. The whole old page is in `STATUS-HISTORY.md` under „Moved from STATUS on 2026-10-20" (headings
|
||||||
|
moved down two levels, otherwise unchanged).
|
||||||
|
|
||||||
|
## Register
|
||||||
|
|
||||||
|
- **Four numbers:** rows before **156**, after **153**, opened **1** (R-1174), closed **4**.
|
||||||
|
- **Closed:** R-1121 (the rotation file is gone; control `~/.bashrc`), R-1118 (its last part, R-1134, closed
|
||||||
|
2026-10-17; `sec doctor` all OK), R-231 (scripts in git and installed from it, the first run green; its same-disk
|
||||||
|
facts live on in R-232) — evidence in `CLOSED-ITEMS.md`, `audits/status-view-2026-10-20/local-checks.txt`,
|
||||||
|
`audits/answers-2026-10-12/s20-backup-scripts.txt`.
|
||||||
|
- **Merged:** R-993 into R-991 (one decision, D5; the proposal and the „no host floor at install" fact copied into R-991).
|
||||||
|
- **Narrowed:** R-132 (the hub password was rotated 2026-10-15; left: the break-glass hash field in the public repo);
|
||||||
|
R-1049 → WATCHING (the fix is live since hub 0.151.0; it no longer waits on the operator).
|
||||||
|
- **Owner cell fixed:** R-919 `CC` → `operator (upload + the look), then CC (close)` — its next action is the operator's.
|
||||||
|
- **Operator-owned rows (43), still waiting on the operator?** yes for all, except: R-933 — **CC first** (the size
|
||||||
|
measurement), then the operator's go; R-388 — **partly** (slice 1 is built; the scope decision is open);
|
||||||
|
R-920 — parked by the operator's own choice (c). On the page: decide D3–D9, D21; do (still needed, the rest);
|
||||||
|
the „Later" line for the remaining 23.
|
||||||
|
- **Rows over 4.5 KB (not rewritten, §N.7):** R-336 7.6 KB, R-516 7.0, R-462 6.8, R-433 6.7, R-264 5.8, R-919 5.0,
|
||||||
|
R-440 4.6, R-925 9.0.
|
||||||
|
|
||||||
|
## The gate — `scripts/status_view_gate.py` (R-1174)
|
||||||
|
|
||||||
|
- **Checks:** size outside the decide table (≤ 50 lines, ≤ 6,000 characters); one „Updated" paragraph (≤ 3 sentences,
|
||||||
|
no „Earlier" chain); no Hungarian letter outside a quote; every cited R-n open; no decided/closed/done row in the
|
||||||
|
decide table; the P1 row under „What is broken" or „decide"; every open row whose Owner names the operator under
|
||||||
|
„decide" or „do". Exit 0 clean, 1 convicted, 2 inconclusive (no sections, missing section, no or empty register).
|
||||||
|
- **Switch:** warn-first until **2026-10-27** (`SWITCH_DATE`), then a refusal; the date is also a due check for
|
||||||
|
R-1174, so the next push on or after that date names it.
|
||||||
|
- **Tests** (`scripts/test_status_view_gate.py`, 39 cases, all PASS): the genuine fixture before and after the switch;
|
||||||
|
the clean run's „what it looked at" line; per check, decoys that warn before and refuse from the switch — size (50
|
||||||
|
more lines; few lines and too many characters), updated (a prepended paragraph, an Earlier chain, four sentences, no
|
||||||
|
paragraph), english (a Hungarian do-line), open-ids (a closed row cited), decide (a DECIDED row; a „done" pick), p1
|
||||||
|
(the P1 line deleted; moved to What works), operator (the Later line deleted; a row only in „broken"); genuine
|
||||||
|
controls that must pass (a long decide table, version numbers in a sentence, Hungarian inside a quote); four
|
||||||
|
inconclusive cases; and the real `STATUS.md` against the real register.
|
||||||
|
- **Red-proof:** each check removed in turn → exactly its own decoys FAIL (size 4, updated 8, english 2, open-ids 2,
|
||||||
|
decide 4, p1 4, operator 4), then put back → 39/39. The switch moved to 2099 → the 14 „refuses" cases FAIL; moved
|
||||||
|
one day earlier → the 14 „warns on 10-26" cases FAIL. Put back → 39/39.
|
||||||
|
- **Run against the old page** (as it was at `2954b0be`): 75 convictions across all seven checks (81 lines, 25,472
|
||||||
|
characters outside decide; 13 decided rows; 24 operator rows missing; the P1 only in „do").
|
||||||
|
- Wired: `repo_gates.py` (fast set, docstring entry 19), `test_gate_decoys.py` (COVERS + runs the suite), found by
|
||||||
|
`script_tests_gate.py`; `decoy_coverage_gate.py` reports it COVERED.
|
||||||
|
|
||||||
|
## The CLAUDE.md edit (repo `felhom.eu/CLAUDE.md`, end-of-session checklist, line 139–141)
|
||||||
|
|
||||||
|
- **Before:** „… One screen, written for the operator in plain language, and deliberately **not** `CONTEXT.md`."
|
||||||
|
- **After:** the same, plus „(STE English) … How to update it:" and five numbered lines — replace the „Updated"
|
||||||
|
paragraph (never prepend; the old one to `STATUS-HISTORY.md`); a morning list under a dated sub-heading in „do";
|
||||||
|
remove what you finished; the decide-row rule; the gate checks it (R-1174).
|
||||||
|
- **Why allowed:** a format rule, not a loosened safety rule (`PROMPT-TEMPLATE.md` §9 rule 9, decision 150).
|
||||||
|
|
||||||
|
## Commits and the running session
|
||||||
|
|
||||||
|
`2954b0be` register pass · `9ec5f76a` the page · `1d7071e8` the gate · `54d1ee82` the instruction · this report.
|
||||||
|
**The rehearsal session shares this clone.** Its commit `f84f94d9` (audit evidence only) landed between my parts 1
|
||||||
|
and 2; it touched neither `STATUS.md` nor the register, so there was nothing to merge. **Its `git push` carried my
|
||||||
|
parts 2–4 to `origin` with its own** (the shared branch); its hook ran with `--scope=docs`. My own `--fast` run of the
|
||||||
|
same content was green (rc 0), and `scripts/push_scope.py` classifies the full range correctly as CODE. Its decoy
|
||||||
|
suite briefly planted changes in `website/gyik.html` and `scripts/felhom-host-install.sh` while I worked (R-928's
|
||||||
|
shape); I staged only my own paths, and both files were clean again afterwards. `unproven.py --summary`: unchanged
|
||||||
|
(NOT WALKED 35 of 55).
|
||||||
|
|
||||||
|
**Gate runs.** Every push: `repo_gates.py --fast`, rc 0 (the last one on the final tree, this report included). **Not
|
||||||
|
followed: „the full run before the last push".** The full run adds exactly one gate, `iso-bootstrap`, which starts a
|
||||||
|
container from an ISO image; on DooPlex that is a Docker run on production (workspace rule, and a standing feedback
|
||||||
|
memory), and it checks only `scripts/iso/`, which this session did not touch.
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- „My commits were pushed by the other session's push, under a docs-scope hook" — **NOT-A-FINDING:** CI re-runs the
|
||||||
|
whole fast set on every push whatever the hook's scope, and CI was green for every push of this session (by `head_sha` on the `jobs` endpoint, attempt 1 each): `2954b0be` job 2131, `54d1ee82` job 2132 (it carried parts 2–4), `c701b253` job 2133 — all `success`; the
|
||||||
|
shared-clone behaviour is already recorded (R-928, the shared-worktree rule).
|
||||||
|
- Tester4 shows DOWN with no host on the hub — **NOT-A-FINDING:** the rehearsal session reinstalls Tester4 today.
|
||||||
|
- „A Longhorn volume at 91 %" had no row — **NOT-A-FINDING:** the alarm is no longer active (Alertmanager, with the two
|
||||||
|
R-934 alarms as the control).
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
Provisioned nothing: no machine, no host, no hub write (GET only, CC's scoped token).
|
||||||
@@ -85,15 +85,15 @@
|
|||||||
| <a id="a9"></a>[A9](capability-map-detail/A.md#a9) | One name per secret on every surface — „Beállító kód" (the setup code) and „Tulajdonosi jelmondat" (the owner passphrase) — and the re-enrol mail points at a page a rebuilt box shows | controller v0.211.0, hub v0.104.0 + v0.105.0 (R-295, R-323) | **IMPLEMENTED** | written BUILT: source-read; no customer has typed the passphrase | R-295; R-323 |
|
| <a id="a9"></a>[A9](capability-map-detail/A.md#a9) | One name per secret on every surface — „Beállító kód" (the setup code) and „Tulajdonosi jelmondat" (the owner passphrase) — and the re-enrol mail points at a page a rebuilt box shows | controller v0.211.0, hub v0.104.0 + v0.105.0 (R-295, R-323) | **IMPLEMENTED** | written BUILT: source-read; no customer has typed the passphrase | R-295; R-323 |
|
||||||
| <a id="a10"></a>[A10](capability-map-detail/A.md#a10) | The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE** | demo-felhom, real endpoints | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04); `audits/RECON-offsite-dr-chain-2026-08-04.md` (2026-08-04) |
|
| <a id="a10"></a>[A10](capability-map-detail/A.md#a10) | The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE** | demo-felhom, real endpoints | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04); `audits/RECON-offsite-dr-chain-2026-08-04.md` (2026-08-04) |
|
||||||
| <a id="a11"></a>[A11](capability-map-detail/A.md#a11) | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** | demo/drill fleet only | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
|
| <a id="a11"></a>[A11](capability-map-detail/A.md#a11) | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** | demo/drill fleet only | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
|
||||||
| <a id="a12"></a>[A12](capability-map-detail/A.md#a12) | A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE** | the success path; the failure path is not claimed (R-379, R-380) | `audits/DRILL-r201-night-run-2026-08-04.md` (2026-08-04); `audits/DRILL-r356-hot-only-restore-2026-08-22/` (2026-08-22); `audits/DRILL-r356b-driveless-db-restore-2026-08-22/` (2026-08-22) |
|
| <a id="a12"></a>[A12](capability-map-detail/A.md#a12) | A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES — the rebuilt Tester4 box (public ISO) got the household's sealed package from the hub, asked for the recovery code, opened the history; Nextcloud came back with 3 031 of 3 033 manifest files byte-identical (the 2 never reached the old box) and the next off-site runs appended (63 → 64). Earlier the same day it FAILED (R-1176, now closed); the failure paths are not claimed (R-379, R-380) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `audits/DRILL-r201-night-run-2026-08-04.md` (2026-08-04); `audits/DRILL-r356-hot-only-restore-2026-08-22/` (2026-08-22); `audits/DRILL-r356b-driveless-db-restore-2026-08-22/` (2026-08-22) |
|
||||||
| <a id="a13"></a>[A13](capability-map-detail/A.md#a13) | The customer's own UNAIDED recovery journey, end to end | controller v0.206.0, hub v0.98.0, agent v0.127.0 | **PROVEN-LIVE** | fifth walk, scoped — read what it does not claim | `tests/walk5-r201-2026-08-07/journal.md` (2026-08-07); `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md` (2026-08-05) |
|
| <a id="a13"></a>[A13](capability-map-detail/A.md#a13) | The customer's own UNAIDED recovery journey, end to end | controller v0.206.0, hub v0.98.0, agent v0.127.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES with ONE operator act by design (D19: delete the dead host) — install, bind, claim, the recovery offer, a wrong code refused, the right code, „Teljes visszaállítás", all through the household's own pages; the new-link page now tells a household whose old box is still linked to write to Felhom (R-1203). Earlier the same day it stopped at the rebuilt dashboard (R-1176) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `tests/walk5-r201-2026-08-07/journal.md` (2026-08-07); `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md` (2026-08-05) |
|
||||||
| <a id="a14"></a>[A14](capability-map-detail/A.md#a14) | A random night of household actions while random things go wrong — twelve rounds, unattended | controller v0.245.0, agent v0.131.0, hub v0.116.0, golden 0.245.0, ISO 1.28.0 | **PROVEN-LIVE** | nested box, twelve chaos rounds | `audits/DRILL-chaos-night-2026-09-17.md` (2026-09-17) |
|
| <a id="a14"></a>[A14](capability-map-detail/A.md#a14) | A random night of household actions while random things go wrong — twelve rounds, unattended | controller v0.245.0, agent v0.131.0, hub v0.116.0, golden 0.245.0, ISO 1.28.0 | **PROVEN-LIVE** | nested box, twelve chaos rounds | `audits/DRILL-chaos-night-2026-09-17.md` (2026-09-17) |
|
||||||
| <a id="a15"></a>[A15](capability-map-detail/A.md#a15) | A stranger's FIRST HOUR on a fresh install — download → install → claim → two apps → use → backup → remove → restore, plus a power cut and a mistyped code | controller v0.242.0, agent v0.130.0, hub v0.112.0, golden 0.242.0, installer ISO 1.26.1 | **PARTIAL** | every mechanism passed; the unaided journey failed before the first app | `audits/DRILL-fresh-install-0242-2026-09-14.md` (2026-09-14); `audits/DRILL-new-household-2026-09-30.md` (2026-09-30) |
|
| <a id="a15"></a>[A15](capability-map-detail/A.md#a15) | A stranger's FIRST HOUR on a fresh install — download → install → claim → two apps → use → backup → remove → restore, plus a power cut and a mistyped code | controller v0.336.0, agent v0.161.0, hub v0.162.0, golden 0.331.0, installer 1.35.0, ISO 1.29.0 | **PARTIAL** | 2026-10-20: download → install → bind → claim → 8 apps → use → backup → remove → every restore path → power cut → mistyped codes all worked; the 8 GB box FROZE twice in the first afternoon and only a plug pull brought it back (R-1192), and the owner passphrase needed the operator | `audits/rehearsal-2026-10-20/journal.md` (2026-10-20); `audits/DRILL-fresh-install-0242-2026-09-14.md` (2026-09-14); `audits/DRILL-new-household-2026-09-30.md` (2026-09-30) |
|
||||||
| <a id="a16"></a>[A16](capability-map-detail/A.md#a16) | Off-site app-data capture covers the paths an app declares MANDATORY — on BOTH drive layouts | controller v0.197.0 | **PROVEN-LIVE** | both drive layouts | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04) |
|
| <a id="a16"></a>[A16](capability-map-detail/A.md#a16) | Off-site app-data capture covers the paths an app declares MANDATORY — on BOTH drive layouts | controller v0.197.0 | **PROVEN-LIVE** | both drive layouts | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04) |
|
||||||
| <a id="a17"></a>[A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) |
|
| <a id="a17"></a>[A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) |
|
||||||
| <a id="a18"></a>[A18](capability-map-detail/A.md#a18) | Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard | agent v0.104.0 → v0.121.0, hub v0.77.0 → v0.91.0 | **PROVEN-LIVE** | — | *(evidence not on file — not re-verifiable)*: the row cites per-repo CHANGELOGs and the spec `backlog/SPEC-r85-phase4-5-2026-07-26.md` — a spec, not a run record |
|
| <a id="a18"></a>[A18](capability-map-detail/A.md#a18) | Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard | agent v0.104.0 → v0.121.0, hub v0.77.0 → v0.91.0 | **PROVEN-LIVE** | — | *(evidence not on file — not re-verifiable)*: the row cites per-repo CHANGELOGs and the spec `backlog/SPEC-r85-phase4-5-2026-07-26.md` — a spec, not a run record |
|
||||||
| <a id="a19"></a>[A19](capability-map-detail/A.md#a19) | A restore-test can never fill the box's disk: it sizes the restore first (uncompressed), keeps off the tested guest's pool when it can, refuses what does not fit, and retries a failed clean-up on a timer | agent v0.133.0 (RELEASED, NOT DELIVERED), hub v0.124.0 | **IMPLEMENTED** | written BUILT: the refusal proven on demo-hp 2026-09-24; the full-test path not live | `audits/r672-2026-09-24/` (2026-09-24); R-672 |
|
| <a id="a19"></a>[A19](capability-map-detail/A.md#a19) | A restore-test can never fill the box's disk: it sizes the restore first (uncompressed), keeps off the tested guest's pool when it can, refuses what does not fit, and retries a failed clean-up on a timer | agent v0.133.0 (RELEASED, NOT DELIVERED), hub v0.124.0 | **IMPLEMENTED** | written BUILT: the refusal proven on demo-hp 2026-09-24; the full-test path not live | `audits/r672-2026-09-24/` (2026-09-24); R-672 |
|
||||||
| <a id="a20"></a>[A20](capability-map-detail/A.md#a20) | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE** | two real firings | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
|
| <a id="a20"></a>[A20](capability-map-detail/A.md#a20) | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0; purge-first hub v0.142.0 | **PROVEN-LIVE** | two real firings (2026-07-18); third 2026-10-22 on a real customer (Tester-2) with the R-32 purge-first leg — all legs ok, others untouched | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18); `audits/tester2-reset-2026-10-22/` |
|
||||||
| <a id="a21"></a>[A21](capability-map-detail/A.md#a21) | Customer DELETE cascade (top lifecycle tier): one guided operator action → `hosts → RESET → residue → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **IMPLEMENTED** | written UNIT-PROVEN: live leg pending | hub unit tests `hub/internal/web/customer_delete_test.go` (outside `documentation/`) |
|
| <a id="a21"></a>[A21](capability-map-detail/A.md#a21) | Customer DELETE cascade (top lifecycle tier): one guided operator action → `hosts → RESET → residue → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **IMPLEMENTED** | written UNIT-PROVEN: live leg pending | hub unit tests `hub/internal/web/customer_delete_test.go` (outside `documentation/`) |
|
||||||
| <a id="a22"></a>[A22](capability-map-detail/A.md#a22) | Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | GL6-F2 open | `pilot/DRILL-GL6-2026-07-08.md` (2026-07-08); R-9 |
|
| <a id="a22"></a>[A22](capability-map-detail/A.md#a22) | Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | GL6-F2 open | `pilot/DRILL-GL6-2026-07-08.md` (2026-07-08); R-9 |
|
||||||
| <a id="a23"></a>[A23](capability-map-detail/A.md#a23) | Felhom in a virtual machine on OpenMediaVault 8 (KVM plugin): the whole NVMe passed to the VM, installed from the public ISO, bound, claimed, apps, backup, and back by itself after an OMV restart | ISO 1.29.0, agent 0.156.1, controller 0.314.0, OMV 8.5.11-1 + openmediavault-kvm 8.3.4 | **PARTIAL** | live on an exact OMV copy (VM on demo-hp), not yet on the pilot's real hardware; one-disk Jellyfin not proven (R-1068); NAS films not usable by Jellyfin (R-1071); no SMART inside the VM (R-1073) | `audits/omv-vm-2026-10-15/README.md` (2026-10-15); guide `pilot/GUIDE-omv-kvm-hu.md` |
|
| <a id="a23"></a>[A23](capability-map-detail/A.md#a23) | Felhom in a virtual machine on OpenMediaVault 8 (KVM plugin): the whole NVMe passed to the VM, installed from the public ISO, bound, claimed, apps, backup, and back by itself after an OMV restart | ISO 1.29.0, agent 0.156.1, controller 0.314.0, OMV 8.5.11-1 + openmediavault-kvm 8.3.4 | **PARTIAL** | live on an exact OMV copy (VM on demo-hp), not yet on the pilot's real hardware; one-disk Jellyfin not proven (R-1068); NAS films not usable by Jellyfin (R-1071); no SMART inside the VM (R-1073) | `audits/omv-vm-2026-10-15/README.md` (2026-10-15); guide `pilot/GUIDE-omv-kvm-hu.md` |
|
||||||
@@ -181,6 +181,7 @@
|
|||||||
| <a id="d13"></a>[D13](capability-map-detail/D.md#d13) | Media apps see EVERY drive; a drive added, pulled, back or safely removed is handled per app (notice, real restart, wiring, torrents paused/resumed) | controller v0.313.0, catalog `0188430`, hub v0.151.0 | **PROVEN-LIVE** | Tier-0 boxes (Tester 1, demo-hp 9201); the fleet with golden 0.313.0 + the floor | `audits/media-build-2026-10-15/G/` (2026-10-15); `audits/media-drive-events-2026-10-14/` (2026-10-14) |
|
| <a id="d13"></a>[D13](capability-map-detail/D.md#d13) | Media apps see EVERY drive; a drive added, pulled, back or safely removed is handled per app (notice, real restart, wiring, torrents paused/resumed) | controller v0.313.0, catalog `0188430`, hub v0.151.0 | **PROVEN-LIVE** | Tier-0 boxes (Tester 1, demo-hp 9201); the fleet with golden 0.313.0 + the floor | `audits/media-build-2026-10-15/G/` (2026-10-15); `audits/media-drive-events-2026-10-14/` (2026-10-14) |
|
||||||
| <a id="d14"></a>[D14](capability-map-detail/D.md#d14) | A second local copy COUNTS only on a proven different physical disk: the agent knows each place's disks; a VM box makes the copy but says it cannot check; a same-disk copy is not made; a NAS never hides the one-copy notice; a box with no drive refuses a data-folder app in words | agent v0.157.0, controller v0.317.0, hub v0.154.0 | **PARTIAL** | live: VM `unknown` (Tester 1) and two real-disk boxes `proven` (demo-hp, N100); unit only: the same-disk skip, the NAS notice, the no-drive sentence; `usb_bridge` never seen on a device | `audits/storage-identity-buildA-2026-10-15/README.md` (2026-10-15); `architecture/16-storage-physical-identity.md` §13 |
|
| <a id="d14"></a>[D14](capability-map-detail/D.md#d14) | A second local copy COUNTS only on a proven different physical disk: the agent knows each place's disks; a VM box makes the copy but says it cannot check; a same-disk copy is not made; a NAS never hides the one-copy notice; a box with no drive refuses a data-folder app in words | agent v0.157.0, controller v0.317.0, hub v0.154.0 | **PARTIAL** | live: VM `unknown` (Tester 1) and two real-disk boxes `proven` (demo-hp, N100); unit only: the same-disk skip, the NAS notice, the no-drive sentence; `usb_bridge` never seen on a device | `audits/storage-identity-buildA-2026-10-15/README.md` (2026-10-15); `architecture/16-storage-physical-identity.md` §13 |
|
||||||
| <a id="d15"></a>[D15](capability-map-detail/D.md#d15) | A one-disk box gets its own data area („Belső tárhely") outside every whole-box copy, registered as the default drive with no press, so media and data-folder apps install with no extra step; a free disk's partitions can be formatted or used one by one | installer 1.33.0, agent v0.158.1, controller v0.318.0, golden 0.318.0 | **PROVEN-LIVE** | fresh 512 GB one-disk VM on demo-hp; partitions on a scratch disk; the old-box signed job and the D4 sentence unit-proven only; an empty part asks for confirmation (R-1093) | `audits/storage-identity-buildB-2026-10-15/README.md` (2026-10-16) |
|
| <a id="d15"></a>[D15](capability-map-detail/D.md#d15) | A one-disk box gets its own data area („Belső tárhely") outside every whole-box copy, registered as the default drive with no press, so media and data-folder apps install with no extra step; a free disk's partitions can be formatted or used one by one | installer 1.33.0, agent v0.158.1, controller v0.318.0, golden 0.318.0 | **PROVEN-LIVE** | fresh 512 GB one-disk VM on demo-hp; partitions on a scratch disk; the old-box signed job and the D4 sentence unit-proven only; an empty part asks for confirmation (R-1093) | `audits/storage-identity-buildB-2026-10-15/README.md` (2026-10-16) |
|
||||||
|
| <a id="d16"></a>[D16](capability-map-detail/D.md#d16) | A failed or cut restore puts the household's data back EXACTLY as it was and starts the app — local unit, second drive, kept load and off-site, and after a power cut or a killed controller (D22) | controller `main` (0.338.0-rc.1), unreleased | **IMPLEMENTED** | proven live on 9202 (rc.1): a damaged unit and a controller killed between the volume step and the replay; the second-drive pre-check (R-1178), the mirror skip (R-1180) and the off-site power cut by test only; a database outside the replaced volumes is not covered | `audits/night-2026-10-21/P3-restore/02-run.log` (2026-10-21) |
|
||||||
|
|
||||||
## E. Access, networking & household use
|
## E. Access, networking & household use
|
||||||
|
|
||||||
|
|||||||
@@ -481,6 +481,18 @@ box. The escrow is how `K` comes back **without** Felhom ever being able to read
|
|||||||
(hub v0.8.0, `PUT /api/v1/hosts/{host_id}/escrow`). Validated end-to-end on a throwaway in
|
(hub v0.8.0, `PUT /api/v1/hosts/{host_id}/escrow`). Validated end-to-end on a throwaway in
|
||||||
`documentation/tests/slice7-escrow-spike-findings.md`. Restore-mode *serving/consumption* is slice 10.
|
`documentation/tests/slice7-escrow-spike-findings.md`. Restore-mode *serving/consumption* is slice 10.
|
||||||
|
|
||||||
|
#### The rebuilt box gets the household's package back — the CARRY [DESIGN, operator ruling 2026-10-21, hub v0.163.0, R-1176]
|
||||||
|
|
||||||
|
After the operator deletes a dead box's host (escrow acknowledged), the package is demoted to retained custody under
|
||||||
|
the OLD host id. The hub now **re-attaches** it: when the customer has exactly one live host, no live host holds a
|
||||||
|
package, and its newest escrow-acknowledged deletion left a retained row with an identity blob, that row becomes the new
|
||||||
|
host's current escrow row — moved, in one transaction, on the agent's first report (and again on the controller's
|
||||||
|
report, idempotently). **Zero-knowledge is unchanged:** the hub moves the same ciphertext from one row to another; it
|
||||||
|
never opens, re-seals or logs it; the audit (`escrow_carried`) carries host ids and the non-reversible hash. Two live
|
||||||
|
hosts → no carry, `escrow_carry_refused` to the operator. The box then sees „identity present", its mint guard (R-241)
|
||||||
|
holds, and it offers the recovery screen. Proven live 2026-10-21 (`audits/fixes-2026-10-21/P1-live/`: the right code
|
||||||
|
opened the household's history, Nextcloud's 3 031 files came back byte-identical, the next off-site run appended).
|
||||||
|
|
||||||
#### The separation principle (the rule that governs every posture)
|
#### The separation principle (the rule that governs every posture)
|
||||||
Reading customer data needs **BOTH** the encrypted chunks **AND** a usable key. **Zero-knowledge
|
Reading customer data needs **BOTH** the encrypted chunks **AND** a usable key. **Zero-knowledge
|
||||||
holds for exactly as long as Felhom never holds both at once.** Every posture below is just a
|
holds for exactly as long as Felhom never holds both at once.** Every posture below is just a
|
||||||
@@ -643,6 +655,25 @@ run as a **journaled reconcile job**; a mid-flight failure is compensating-rolle
|
|||||||
the just-restored guest — allowed unsigned per §4, same-transaction provenance). They diverge in
|
the just-restored guest — allowed unsigned per §4, same-transaction provenance). They diverge in
|
||||||
the *archive* and the *back half*, **and in identity policy** (below).
|
the *archive* and the *back half*, **and in identity policy** (below).
|
||||||
|
|
||||||
|
**Guest swap = guest RAM, and the host's swap behind it (R-1192; agent 0.162.0, released and vouched 2026-10-22).**
|
||||||
|
`[FACT]` 2026-10-21 (`audits/fixes-2026-10-21/P4-freeze/`): Proxmox gives an LXC a 512 MB swap cap when nothing sets
|
||||||
|
`swap`, and the agent never set it; on the 8 GB minimum machine (guest 4096 MB) the household's first uploads filled the
|
||||||
|
guest with its swap full, the kernel could evict only program code, and the box froze (memory „full" pressure 96 %, load
|
||||||
|
162, the dashboard dead 6½ minutes). `[DESIGN]` The size step sets `swap` = the guest's memory: at bring-up, at a RAM
|
||||||
|
resize (`POST /guest/memory`), and once a minute after agent start for every pool LXC whose swap is below its RAM (live,
|
||||||
|
no restart, never lowers). Proven on a fresh box the night of 2026-10-21 (`audits/night-2026-10-21/P1-soak/`: 512 →
|
||||||
|
4096 one minute after the fixture agent started; the 2-hour household load then answered every 10-second probe). **The
|
||||||
|
host must have swap behind it:** our installer gives an 8 GB / 256 GB machine 7.75 GB (measured the same night); a BYO
|
||||||
|
host may have none. The agent reports `host.host_swap_mb` and `host.guest_swap_mb` on every host report and logs one WARN
|
||||||
|
when the first is below the second (its own, strict line — a log, nothing more). **The judgement is the hub's and the
|
||||||
|
box's, by one rule (R-1216, D28 = A, operator 2026-10-22):** short = no host swap, or less than 90 % of min(the guests'
|
||||||
|
swap allowance, 8192 MB) — 8192 MB being the most Proxmox's installer gives any machine (`Install.pm compute_swapsize`:
|
||||||
|
the RAM, at least 4096 MB on a disk ≥ 64 GB, at most 8192 MB). The hub mails the operator once (`host_swap_short`,
|
||||||
|
operator-only); the controller's install page adds „A gépen nincs elég cserehely (swap)…" when memory is tight and the
|
||||||
|
rule says short. One case table in both repos, byte-identical (`scripts/host_swap_rule_gate.py`). The first rule (host
|
||||||
|
swap < the guests' whole allowance) flagged every machine with a guest above 8 GB; the 8 GB soak used 2.7 GB. The
|
||||||
|
agent never creates or enlarges host swap — changing a host its owner set up is a decision (register).
|
||||||
|
|
||||||
**Identity reset is scenario-specific — this is a correctness boundary, not a detail.** "Reset
|
**Identity reset is scenario-specific — this is a correctness boundary, not a detail.** "Reset
|
||||||
identity" is shorthand for two different operations:
|
identity" is shorthand for two different operations:
|
||||||
|
|
||||||
|
|||||||
@@ -331,7 +331,9 @@ customer instead of a single controller.
|
|||||||
- **DROP both** `infra_backup_versions` (current/live) **and** `infra_backups` (legacy mirror) — the DR
|
- **DROP both** `infra_backup_versions` (current/live) **and** `infra_backups` (legacy mirror) — the DR
|
||||||
recipe, tenantsync and the escrow blob replace them (R-105).
|
recipe, tenantsync and the escrow blob replace them (R-105).
|
||||||
- **KEEP** `customer_configs`, `events`, `customer_notifications`, `notification_log`,
|
- **KEEP** `customer_configs`, `events`, `customer_notifications`, `notification_log`,
|
||||||
`app_telemetry`, `app_log_issues`.
|
`app_telemetry`, `app_log_issues`. `customer_notifications.household_saved` (hub 0.165.0, R-1199 reopened) marks a
|
||||||
|
list the household chose on a box; a box's default never overwrites it, and a box reads it back with `GET
|
||||||
|
/api/v1/preferences` (customer-scoped like the save). The rule is in `08` (the 2026-10-22 `[DESIGN]` paragraph).
|
||||||
- **Authz cleanup the cutover enables:** several endpoints today use global-or-any-customer-key auth
|
- **Authz cleanup the cutover enables:** several endpoints today use global-or-any-customer-key auth
|
||||||
rather than customer-scoped (the infra-backup GETs, `/notify`). Most retire with the infra-backup
|
rather than customer-scoped (the infra-backup GETs, `/notify`). Most retire with the infra-backup
|
||||||
push; any that carry over should scope to the resolved host/guest → customer under §2.
|
push; any that carry over should scope to the resolved host/guest → customer under §2.
|
||||||
@@ -575,6 +577,14 @@ are deleted at start-up and at each new day (`TestAccessLog_Keeps90DaysThenDelet
|
|||||||
ingress log: ≈ 1,900 hub requests a day at ≈ 110 bytes a line → ≈ 0.2 MB a day, ≈ 19 MB for 90 days. A logging
|
ingress log: ≈ 1,900 hub requests a day at ≈ 110 bytes a line → ≈ 0.2 MB a day, ≈ 19 MB for 90 days. A logging
|
||||||
failure never fails or delays a request: lines go through a non-blocking buffer to one writer (`internal/accesslog`).
|
failure never fails or delays a request: lines go through a non-blocking buffer to one writer (`internal/accesslog`).
|
||||||
|
|
||||||
|
### 16.5 Daily jobs and the clock change [DESIGN — hub v0.163.0, operator ruling D24 2026-10-21, R-1183]
|
||||||
|
|
||||||
|
Every daily hub job (`scheduleDaily`: db-snapshot 02:00, prune, deadline-check 05:00, offsite-key-audit 07:10) runs at
|
||||||
|
its Budapest wall-clock time **once per calendar day**: the next run is the calendar day after (`time.Date`, day+1),
|
||||||
|
never „now + 24 h", and the slot that ran last never runs again. On a 25-hour day (the last Sunday of October) the old
|
||||||
|
arithmetic ran every job after 02:00 twice; on a 23-hour day it ran them an hour late. Pinned by `TestR1183_*` for
|
||||||
|
2026-10-25 and 2027-03-28.
|
||||||
|
|
||||||
## 17. Operator access tokens [DESIGN — hub v0.152.0, operator ruling 2026-10-15 11:38 (`09` §3 decision 251)]
|
## 17. Operator access tokens [DESIGN — hub v0.152.0, operator ruling 2026-10-15 11:38 (`09` §3 decision 251)]
|
||||||
|
|
||||||
**The problem.** Scripts — Claude Code's floor step first — drove the hub with the operator's own password (HTTP Basic,
|
**The problem.** Scripts — Claude Code's floor step first — drove the hub with the operator's own password (HTTP Basic,
|
||||||
|
|||||||
@@ -736,6 +736,42 @@ stayed (37 tables, the live app had 36); the definition did not move (same versi
|
|||||||
answered 409. The restore page's „the database is back as it was" is false for it. Design note with the two options and
|
answered 409. The restore page's „the database is back as it was" is false for it. Design note with the two options and
|
||||||
the pick (A: a copy of the app's live volumes, put back on failure — one copy's disk for the restore's length, a fit
|
the pick (A: a copy of the app's live volumes, put back on failure — one copy's disk for the restore's length, a fit
|
||||||
check before the stop): `audits/night-2026-10-19/P3/design-R-893.md`; the decision is on `STATUS.md`.
|
check before the stop): `audits/night-2026-10-19/P3/design-R-893.md`; the decision is on `STATUS.md`.
|
||||||
|
**`[DESIGN]` 2026-10-20 — „put back exactly as it was" (R-893 option A, operator decision D20 = A, `09` §3 decision 192
|
||||||
|
second half; controller 0.336.0 is the floor).** Before the stop, every named volume the snapshot will replace is measured
|
||||||
|
and its copy is added to the restore's own free-space need (R-357's exact rule); too little room or an unmeasurable volume
|
||||||
|
refuses with nothing stopped. Right after the stop the volumes are tarred into `<namespace>/backups/restore-undo/<app>/`
|
||||||
|
(outside every capture set). On ANY failure after a volume or the definition moved, the copies are poured back, a volume
|
||||||
|
the snapshot added is removed, the live definition is written back and the app is STARTED; the household reads that the
|
||||||
|
restore failed and nothing changed (with placed files: its database and volumes are back, the copied files stay); the
|
||||||
|
operator gets `backup_run_failures` leg `restore-put-back`. The hold (`restore_mixed`) remains only for a put-back that
|
||||||
|
fails — then the copy is kept for support — and the double failure (replay AND rollback) holds as before. On success the
|
||||||
|
copy is removed. **`[FACT]` 2026-10-20 (9202, controller 0.336.0-rc.1, `audits/night2-2026-10-20/P2/live/`):** the same
|
||||||
|
forced failure as on 2026-10-19 → 36 tables, the backup-only table absent, schema/data/row-count md5 equal to before,
|
||||||
|
the app running, no hold, the copy (69 MB, 0.9 s) gone; a normal restore of the same snapshot then worked. The R-893
|
||||||
|
known limit above is closed for named volumes; placed FILES are still not taken back (the plain-files mix, as ruled).
|
||||||
|
|
||||||
|
**`[DESIGN]` 2026-10-21 — the put-back covers EVERY restore, and a power cut (operator ruling D22 = yes, 2026-10-21
|
||||||
|
20:05; controller `main`, unreleased).** (1) **The local unit restore** — the app's own unit, the second drive's unit,
|
||||||
|
the second drive's „Teljes visszaállítás" and a kept load all run `RestoreFromRecoveryUnitAtWith` — now takes the same
|
||||||
|
measure → copy → put-back (`offbox_restore_undo.go`, one implementation, no second copy of the logic): the copy must fit
|
||||||
|
beside the unit before anything stops; ANY failure after the stop (volume replay, definition write, DB-only start, DB
|
||||||
|
replay, start) pours the live volumes back, writes the live definition back and starts the app; a put-back that fails
|
||||||
|
holds and keeps the copy; with no copy (a unit with no volume archive) a half-done DB replay HOLDS (the R-379 ruling)
|
||||||
|
instead of starting on it. The window carries the app-stop marker (`unit_restore`). (2) **A restore cut by a power loss
|
||||||
|
or a controller kill** (R-1195): once the copy is whole the restore writes `undo.json` and the live definition files
|
||||||
|
beside it and records the folder in the app-stop marker; at start the crash recovery pours it back BEFORE it restarts the
|
||||||
|
app, and the household reads „A visszaállítás félbeszakadt (áramszünet). Az alkalmazás adatai a visszaállítás előtti
|
||||||
|
állapotban vannak." A cut before the copy was whole restarts as before (nothing had moved). Local and off-site alike.
|
||||||
|
(3) **The second drive's „Teljes visszaállítás" asks the unit's refusals before its file half** (R-1178), so a refusal
|
||||||
|
moves nothing and its „érintetlenek" is true; a later refusal starts the app and names the changed files. (4) **The
|
||||||
|
night's second-drive mirror skips an app while a restore of it runs** (R-1180). **`[FACT]` 2026-10-21 night (9202,
|
||||||
|
controller 0.338.0-rc.1, `audits/night-2026-10-21/P3-restore/`):** Rallly + Postgres; a damaged unit volume → the live
|
||||||
|
volume poured back, the newest row still there (A,C), the app running, „…Nem változott semmi…" (on 0.331.0 the same case
|
||||||
|
crash-looped Postgres and lost the row, C3 2026-10-20); the controller killed (kill -9) right after the volume step →
|
||||||
|
on start „interrupted restore is put back … 1 volume(s) poured back, definition=true", markers A,C,D (D = the row newer
|
||||||
|
than the unit), the app running, the record reads the D22 sentence. Known limit kept: a database whose data is NOT in a
|
||||||
|
named volume the unit replaces (none in the catalog today — a drive-file app's unit restore is refused by R-538) is not
|
||||||
|
covered by the volume copy.
|
||||||
|
|
||||||
**[DESIGN] 2026-08-22 — the failure ladder of a database restore: replay → rollback → hold.**
|
**[DESIGN] 2026-08-22 — the failure ladder of a database restore: replay → rollback → hold.**
|
||||||
Recorded here rather than only in a closed register row, because a decision that survives only inside
|
Recorded here rather than only in a closed register row, because a decision that survives only inside
|
||||||
@@ -1038,6 +1074,14 @@ here — §8 has rows where it is the actual state.
|
|||||||
|
|
||||||
### 7.0 What a customer can and cannot do ALONE — the four steps the drill found
|
### 7.0 What a customer can and cannot do ALONE — the four steps the drill found
|
||||||
|
|
||||||
|
> **[DESIGN] 2026-10-21 — the box death, alone (operator ruling, hub v0.163.0, R-1176).** After the operator's one press
|
||||||
|
> (delete the dead host — D19), the household reinstalls, binds, claims, and the rebuilt box offers „A korábbi, házon
|
||||||
|
> kívüli mentéseid visszaszerezhetők" and asks for the recovery code: the hub carries the sealed package to the new
|
||||||
|
> host (03 §8a, „the CARRY"). A wrong code is refused and changes nothing; the right code unlocks the history; each
|
||||||
|
> app is reinstalled and brought back with „Teljes visszaállítás"; the night's off-site run appends to the same
|
||||||
|
> history. Measured end to end 2026-10-21 (`audits/fixes-2026-10-21/P1-live/`). Before this ruling the rebuilt box saw
|
||||||
|
> no package, minted a fresh key and set the history aside on its first night (rehearsal 2026-10-20 B7).
|
||||||
|
|
||||||
> **[FACT] Added 2026-08-05.** The 2026-08-04 R-201 drill (`audits/DRILL-r201-night-run-2026-08-04.md`)
|
> **[FACT] Added 2026-08-05.** The 2026-08-04 R-201 drill (`audits/DRILL-r201-night-run-2026-08-04.md`)
|
||||||
> is the first end-to-end proof that a customer's file survives a machine rebuild and comes back
|
> is the first end-to-end proof that a customer's file survives a machine rebuild and comes back
|
||||||
> byte-identical. **It passed with an operator present, and four manual interventions stood between
|
> byte-identical. **It passed with an operator present, and four manual interventions stood between
|
||||||
@@ -1531,6 +1575,19 @@ absent or hollow primary unit from the mirror **inside the call, before returnin
|
|||||||
observe the hollow state. **The capture itself is deliberately not guarded:** a capture that describes
|
observe the hollow state. **The capture itself is deliberately not guarded:** a capture that describes
|
||||||
an empty drive as empty is correct, and guarding it would make the manifest lie.
|
an empty drive as empty is correct, and guarding it would make the manifest lie.
|
||||||
|
|
||||||
|
### 8.2a A second-drive copy whose SOURCE is gone is kept until the household releases it [DESIGN, operator ruling D25, 2026-10-21]
|
||||||
|
|
||||||
|
**[DESIGN]** The derived-copy rule (row 5) rebuilds the secondary from the source. When the SOURCE is gone — the drive
|
||||||
|
is away or decommissioned, a registered share's folder or an app's declared folder is missing — there is nothing to
|
||||||
|
rebuild from, and the copy may be the only one left. **D25:** such a copy is kept, recorded, and listed on the restore
|
||||||
|
page („Megőrzött másolatok") with „Eltávolítás"; no night job removes it. This is a different case from „data legs are
|
||||||
|
not guarded" for a leg that LEAVES the class set (the household's or the catalog's own choice). A share the household
|
||||||
|
deleted or renamed on a present drive is still pruned. An EMPTIED present folder is not mirrored over a copy that
|
||||||
|
holds files (the run warns). Removal is by the record (path inside the copy tree, same device + inode), never by size
|
||||||
|
or pattern. **Why:** the data-loss hunt of 2026-10-20 proved two nights that deleted a household's only second copy
|
||||||
|
(R-1175: one share group's reconcile removed the other groups' copies, and a dead drive's copy; R-1181: a missing or
|
||||||
|
emptied folder took its copy within one night). Controller: `internal/backup/tier2_kept.go`.
|
||||||
|
|
||||||
### 8.1 The blank cells, listed explicitly
|
### 8.1 The blank cells, listed explicitly
|
||||||
|
|
||||||
Per the rule that a blank is a finding, here they are:
|
Per the rule that a blank is a finding, here they are:
|
||||||
|
|||||||
@@ -499,6 +499,23 @@ app-mail settings).*
|
|||||||
**Pick: A.** It follows the direction (the household decides three things, not 18), keeps every existing
|
**Pick: A.** It follows the direction (the household decides three things, not 18), keeps every existing
|
||||||
mechanism, and each slice is reversible on its own. B can be added later as a fourth choice if households ask.
|
mechanism, and each slice is reversible on its own. B can be added later as a fourth choice if households ask.
|
||||||
|
|
||||||
|
**`[DESIGN]` 2026-10-21 night — the box's page is the list the hub mails by (R-1199; decided by CC unattended — operator
|
||||||
|
may reverse).** Measured: a fresh box shows its default list ticked, but pushed nothing until the household saved the
|
||||||
|
page, so the hub kept mailing by an older stored list (Tester4: saved 2026-10-18, without disk warning, drive
|
||||||
|
disconnected and second-drive failure — 1 household alarm of 9 on the rehearsal day). Now a box whose household never
|
||||||
|
saved a choice pushes the default list at start (no address — the hub keeps the stored one); a saved choice is pushed
|
||||||
|
only as before (R-922). Controller `ba22f9b`, shipped in 0.338.0.
|
||||||
|
|
||||||
|
**`[DESIGN]` 2026-10-22 — a household's saved list is never overwritten (R-1199 reopened; the day brief).** `[FACT]`
|
||||||
|
0.338.0's push overwrote demo-felhom's own saved list (13 events, written by an earlier box) with the 17 defaults at
|
||||||
|
its first start, and repeated at every start. So the hub now records WHO chose a list (`customer_notifications.
|
||||||
|
household_saved`, set only by a push the box declares `source: household` — the settings page, or the re-push of a
|
||||||
|
list saved on the box — and never cleared) and refuses any other push over a household's list (`{"status":"kept"}`;
|
||||||
|
an older controller's push with neither address nor clear counts as not the household's). A box with no list of its
|
||||||
|
own READS the hub's first (`GET /api/v1/preferences`, self-scoped): a household's list is adopted and shown, nothing
|
||||||
|
pushed; only when no household ever saved one does the box push its default, once, and save it. Hub 0.165.0,
|
||||||
|
controller 0.339.0. Existing rows were marked once: written after creation = a box wrote it (`household_prefs.go`).
|
||||||
|
|
||||||
**What the operator still gets:** everything, unchanged — the operator channel never reads the household's switches
|
**What the operator still gets:** everything, unchanged — the operator channel never reads the household's switches
|
||||||
(`processOperator` is independent of `enabled_events`).
|
(`processOperator` is independent of `enabled_events`).
|
||||||
|
|
||||||
|
|||||||
@@ -367,6 +367,8 @@ here; every sentence this product writes is display, and a decision reads a sign
|
|||||||
a sentinel for `errors.Is` — never `fmt.Errorf("%w: …")`, which would prepend the sentinel's own text
|
a sentinel for `errors.Is` — never `fmt.Errorf("%w: …")`, which would prepend the sentinel's own text
|
||||||
to the customer's sentence.
|
to the customer's sentence.
|
||||||
|
|
||||||
|
**2026-10-21 night (W25, controller `5776d54`, unreleased):** the off-site page's „Utolsó hiba" no longer shows the run's raw text to the household. It is built from the classifier's class and the run's own result (the failed apps by name, „the other apps are fine" when true, „the box tries again the next night" when a retry can help). The raw text — restic's included — stays verbatim in the log and, sanitised, in the operator's event; the „foreign error prints verbatim" rule below now holds for those two places only on this page.
|
||||||
|
|
||||||
**[FACT] One exception, with an end date:** a box upgraded to 0.251.0 carries the OLD persisted note
|
**[FACT] One exception, with an end date:** a box upgraded to 0.251.0 carries the OLD persisted note
|
||||||
with no kind until its next off-site run, so `offboxWarningDisplay` keeps the substring test for
|
with no kind until its next off-site run, so `offboxWarningDisplay` keeps the substring test for
|
||||||
`kind == ""` only. **Slice 2 (R-557) must not translate that producer until R-570 closes.**
|
`kind == ""` only. **Slice 2 (R-557) must not translate that producer until R-570 closes.**
|
||||||
|
|||||||
@@ -294,3 +294,29 @@ part next to a mounted part can only be used, not formatted (the root wrapper tr
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## D16 — A failed or cut restore puts the data back exactly (D22)
|
||||||
|
|
||||||
|
**What it is:** a restore that fails after it began changing an app — or is cut by a power loss or a killed controller
|
||||||
|
— leaves the app on the household's own data as it was before the restore, running, and says so. Before (2026-10-20,
|
||||||
|
C3 on 9202): a damaged local unit removed the live database volume, Postgres crash-looped, the newest row was gone and
|
||||||
|
nothing could put it back; a power cut mid off-site restore left the app on the backup's data (rehearsal s2).
|
||||||
|
|
||||||
|
**Components:** controller `main` (`d582feb`, unreleased): `restore_unit.go` reuses `offbox_restore_undo.go` (measure,
|
||||||
|
fit, copy, put back, hold on a double failure); `restore_crash_putback.go` + the app-stop marker's `undo` record and
|
||||||
|
`Recover`'s put-back; `tier2_whole.go` pre-check (R-1178); `RunAllTier2` skip during a restore (R-1180). Design: `07` §6.3
|
||||||
|
(the 2026-10-21 `[DESIGN]` paragraph), operator ruling D22.
|
||||||
|
|
||||||
|
**Status:** **IMPLEMENTED** — proven live on scratch 9202 with 0.338.0-rc.1 (2026-10-21): a damaged unit volume → the
|
||||||
|
live volume poured back, the newest row kept (A,C), the app running, „…Nem változott semmi…"; the controller killed
|
||||||
|
(kill -9) right after the volume step → on start the copy poured back (A,C,D — D newer than the unit), the app running,
|
||||||
|
„A visszaállítás félbeszakadt (áramszünet). Az alkalmazás adatai a visszaállítás előtti állapotban vannak." A normal
|
||||||
|
restore is unchanged (the control). PROVEN-LIVE when the release reaches the fleet.
|
||||||
|
|
||||||
|
**Unit only:** the off-site power cut (`TestR1195_AKilledOffsiteRestoreIsPutBackOnStart`), the second drive's pre-check
|
||||||
|
(`TestR1178_*`), the mirror skip (`TestR1180_*`), a failed put-back at start (`TestR1195_AFailedPutBackOnStart…`).
|
||||||
|
**Known gap:** a database whose data is not in a named volume the unit replaces is not covered by the copy (none in the
|
||||||
|
catalog today; a drive-file app's unit restore is refused by R-538).
|
||||||
|
|
||||||
|
**Evidence:** `audits/night-2026-10-21/P3-restore/` (the run log, the red-proofs).
|
||||||
|
|
||||||
|
---
|
||||||
|
|||||||
@@ -16,6 +16,8 @@
|
|||||||
> renamed here the same day, with the operator's three Claude Design fixes folded in (marked *fix
|
> renamed here the same day, with the operator's three Claude Design fixes folded in (marked *fix
|
||||||
> (a)/(b)/(c)* below) and one measured correction: felhom.eu is served from DooPlex with
|
> (a)/(b)/(c)* below) and one measured correction: felhom.eu is served from DooPlex with
|
||||||
> **Cloudflare DNS only**, not through Cloudflare.
|
> **Cloudflare DNS only**, not through Cloudflare.
|
||||||
|
> 2026-10-21: poster regenerated in Claude Design (one-disk data area, https-only); PDF print version added next to it
|
||||||
|
> (`felhom-system-poster.pdf`).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# C1 reviewer 2 — delete and retention (fresh, read-only; clones at controller 7ca59e2)
|
||||||
|
|
||||||
|
| # | Sev guess | harm | mechanism |
|
||||||
|
|---|---|---|---|
|
||||||
|
| D1 | P1 (mech high, reach medium) | Shares second-drive copy: one drive group's reconcile deletes ANOTHER drive's share mirrors (on the night a drive is decommissioned/away; or every night when two groups target one drive, then a full re-copy) | `internal/backup/tier2_shares.go:135` groups by source; `:201-202` own destBase/legRels; `:230` `tier2ReconcileRoots(destBase, sharesTier2DestRoots(destBase), legRels)`; `:266-267` ReadDir lists every drive key on disk; `tier2.go:299-329` removes any child not in this group's legs; `shares_payload.go:233-251` away/decommissioned drives drop out |
|
||||||
|
| D2 | P1 optional legs / P2 mandatory; design question with a defect corner | A missing or emptied app folder on the drive deletes its second-drive copy within one night; optional legs (audiobooks, comics, ROMs, Immich external library) are in no other tier | `tier2_capture.go:55-61` missing leg skipped (optional silently) → `tier2.go:455` reconcile deletes its mirror; run recorded success `:465`; emptied → `rsync -a --delete` `tier2.go:751` via `:443`; catalog optional legs `audiobookshelf/.felhom.yml:24-26`, komga, romm, immich `:32-34`; 07 §8.2 „data legs are NOT guarded" covers a leg leaving the class set, not a declared-but-missing leg |
|
||||||
|
| D3 | P2 high (mech) | A host clock ahead during one whole-guest backup silences that tier until real time passes it; the hub reads the same clock and says OK | agent `internal/localapi/server.go:1178-1190` negative age never due; `internal/backup/runner.go:607-629` newest ctime unbounded; hub `monitor/deadline_tiers.go:155`, `deadline.go:243`; R-973 cap counts future entries `backup_cap.go:61`; restic: `offbox_window.go:133-136` pinned guard refuses every window → quota fills; NAS tier no guard `:209-216`; `offbox_abandon.go:248,287` set-aside rm on box clock |
|
||||||
|
| D4 | P2 | A failed LOCAL unit restore leaves the app running on mixed data, no undo copy (same as R1 of reviewer 1 — independent agreement) | `restore_unit.go:430-489`; `writeSafetyDump` only in `offbox_reconstitute.go`; 07 §6.3 ladder not scoped to off-site |
|
||||||
|
| D5 | P2 order defect / depth design | A dump that fails its own content check has ALREADY replaced last night's good dump; Tier-2 mirrors it an hour later | `appbackup/dbdump.go:388-397` size-only check, `:399` rename, `:419` `ValidateDump` after; `backup.go:688-690`; volume tars promoted with no content check `backup.go:902` |
|
||||||
|
| D6 | P3 med | „Megőrzött adatok" can list a RUNNING app's live folder as deletable leftover if its app.yaml is unreadable | `stacks/manager.go:672`; `kept.go:220-235` skips `!Deployed || AppConfig==nil`; `ListKept :330-358`; `DeleteKept :385-402` RemoveAll; `deploy.go:1180-1190` write+rename without fsync |
|
||||||
|
| D7 | P3 low-med | A unit restore can point the app back at an OLD drive; F5 then deletes the current drive's unit | `restore_unit.go:90-111, 507`; `cmd/controller/main.go:3252`; `deploy.go:820`; `backup.go:1517-1574` `pruneStalePrimaryDirs` even when this run's capture failed (`:726-733`) |
|
||||||
|
| D8 | P3/P4 design | local whole-guest keep=1 prunes the only good archive before the new one is proven | agent `runner.go:229`, `:139-161`, restore-test settle 24 h `:462` |
|
||||||
|
|
||||||
|
**Checked SAFE (reviewer):** hub off-site deletions by exact identity (`offsitekeys.go:366-515`); pinned off-site retention guard refuses future/too-new snapshots and big plans (`offbox_window.go:132-161`); dump and tar writes crash-safe (tmp+fsync+rename); R-403 hollow-unit guard; drive migration deletes source only after verify (`stacks/migrate.go:614-667`); app removal refusals and protected folders (`delete.go:136-153, 241-301, 839-846`); mirror/undo clean-ups allow-listed/prefix-guarded; agent prune never on PBS unless opted in; restore-test skips unsettled/future/foreign-key archives; `ForgetRetiredDrives` mount units only; SMB share removal deletes no folder. Not reviewed: agent `vzpartial` janitor, tenantsync/ep0 prune config, hub RESET beyond `PurgeRepos`.
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# C1 reviewer 3 — keys that gate recovery (fresh, read-only; controller 7ca59e2, hub 5c48731, agent 4e1621c)
|
||||||
|
|
||||||
|
| # | Sev guess | harm | mechanism |
|
||||||
|
|---|---|---|---|
|
||||||
|
| K1 | P2 (P1 with K2 / a RESET) high | **A reinstall through „delete the old host first" (D19/R-1154) cuts the new box off from the household's escrow: the recovery code is never asked, a NEW key is minted, and the old off-site history is set aside by itself on night one; typing the old code answers „wrong code"** | hub `api/passphrase_scope.go:58` 409 forces the delete; `store.go:3204-3210` delete demotes escrow under the OLD host_id; `handler.go:1261` new host_id; `GetEscrowStatusForCustomer` `store.go:3500-3527` joins on `hosts` → nil (superseded join on deleted host also gone) → `identity_blob_present`/`superseded_present` false; controller `offbox.go:542-555` mints new pw, `:1788` no recovery offer, `:905-906` firstNight → `resetOrphanedRepo`; `handleHostEscrowRetainedGet` 404 for a deleted host (`handler.go:1691`); contrast purge `store.go:4185-4190` DOES include `host_deletions` |
|
||||||
|
| K2 | P2 high | Abandoning ONE set-aside copy deletes EVERY retained recovery package of the customer — other set-aside copies become ciphertext nobody can open | hub `handler.go:626-627` → `PurgeSupersededEscrowForCustomer` `store.go:4185-4190` deletes all rows for the customer (incl. deleted hosts); the comment at `:4176-4179` states the invariant it breaks |
|
||||||
|
| K3 | P3 med-high | A ceremony while off-site is not configured seals a package with NO repo password and supersedes the one that had it | controller `escrow_handlers.go:217` re-stage only when configured; agent `main.go:2975-2981`; hub `store.go:3392`; `escrow_confirm.go:222-258` stale card local only |
|
||||||
|
| K4 | P3 med | The global „create your recovery code" bar links to the ceremony the backups page hides while a recovery offer is outstanding | `backups_remote.html:144-160` vs `escrow_banner.go:23-33` + hu.json:1682; no `recoveryOffer()` refusal in the wizard/start handlers; settle gate `main.go:876`, `offsiteapply.go:84` |
|
||||||
|
| K5 | P3 med-high | After a reinstall recovered on the reuse path, the NEW PBS key K is never escrowed and nothing compares fingerprints | `felhom-pbs-apply:99` autogen; `escrow_confirm.go:182-205` auto-confirm on hash; no compare of `host_escrow.key_fingerprint` vs live key (agent `internal/backup/runner.go:407-429` only) |
|
||||||
|
| K6 | P3/P4 design | Whole-guest off-site (PBS) copies run before any escrow exists; host lost before the ceremony → unreadable archives | controller gates Tier 3 (`offbox.go:781`); agent `internal/backup`, `internal/pbsdr` have no escrow gate |
|
||||||
|
| K7 | P4 low | escrow PUT split in two statements; a failure leaves a row whose hash claims a key its blob lacks | hub `handler.go:1388, 1412-1420`; `store.go:3402-3408` |
|
||||||
|
| K8 | P4 | operator off-site re-issue resets the provider password before saving it | hub `offsite.go:217-245` |
|
||||||
|
|
||||||
|
**Checked SAFE (reviewer):** mint guard while the hub holds a package (`offbox.go:542-555, 659-720`); orphaned repo moved aside, never re-inited/deleted (`offbox.go:886-943, 332-397`); supersession retains both blobs (`store.go:3343-3416`); host delete demotes not destroys (`store.go:3164-3229`); ceremony two-word confirm; `escrow.Create` self-verifies; recovery refuses to replace a different local key; `ClearOffboxTarget`; PBS wrapper has no delete verb; hub abandon delete limited to one path after a delay (`offsitekeys/service.go:464-496`).
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
# C1 reviewer 1 — restore and rollback (fresh, read-only; clones at controller 7ca59e2 / v0.336.0)
|
||||||
|
|
||||||
|
Raw candidates as returned by the reviewer, condensed; each is UNVERIFIED until C2 marks it. File:line are in the controller unless named.
|
||||||
|
|
||||||
|
| # | Sev guess | harm | mechanism |
|
||||||
|
|---|---|---|---|
|
||||||
|
| R1 | P2 high | A local / second-drive unit restore that fails halfway restarts the app on half-replaced data, no undo copy; the next night's capture then overwrites the only restore point with the broken state | `internal/backup/restore_unit.go:429-431` failed stop only warns; `restore.go:217-218` `docker volume rm -f` error ignored before untar (10-min limit `:237-241`); `restore_unit.go:466-486` DB replay failure → app started anyway; one restore point per app, promoted in place (`atomicPromoteTar`) |
|
||||||
|
| R2 | P2 high | A power cut / controller restart during a restore starts the app on a mixed state; the undo copy is never used at boot | `offbox_reconstitute.go:942` marker → `appstop_marker.go:307` StartStack with no restore check; `offbox_restore_undo.go:137` undo dir only dropped, never read at start; unit restore and .fab import set no marker → `bootrecon/bootrecon.go:210-213,269` starts the app; updates have a journal (`stacks/update.go:1420-1489`), restores do not |
|
||||||
|
| R3 | P2 high | A cut-off shared-folder restore can be „placed" and reported as restored while files are missing; scratch deleted | `shares_restore.go:138` readiness = `rErr == nil && len(entries) > 0` (R-358 class, not fixed for shares); 30-min ctx `web/offbox_handlers.go:716`; place reports N files and deletes scratch `:301`; free-space gate fails open and fixed 2 GiB `:107` |
|
||||||
|
| R4 | P2 high | A `.fab` export silently drops a whole drive folder that contains a symlink; import never sets owners | `appexport/fabplan.go:292` `tar.FileInfoHeader(info,"")` then `os.Open`+`io.Copy` on a symlink; `export.go:714` warn-and-continue; `assertBundleDataComplete` refuses only when no data at all; `restore.go:1157-1176` extracts only dirs/regular files, no chown |
|
||||||
|
| R5 | P2 med-high | Second-drive whole restore swaps files, then refuses the DB half, leaves the app STOPPED and says „the app is untouched" | `tier2_whole.go` stops + merges first, then `RestoreFromRecoveryUnitAtWith` whose refusals (`restore_unit.go:366-371, 386-389, 413-416, 420-423`) assume nothing happened; en.json:2667 „the restore did not start. The app is untouched" |
|
||||||
|
| R6 | P2 med | `.fab` import into a STOPPED app merges into existing volumes; the default export (stop box ticked) has no SQL dump | `appexport/restore.go:361-374` volumes removed only if running; `docker cp` merge `:721`; `templates/app_data.html:221` stop default; `export.go:262` stop before `dumpDatabase` `:366`; `appbackup/dbdump.go:142` discovers only running containers |
|
||||||
|
| R7 | P2/P3 high | `.fab` import has no concurrency guard, no undo, no record | `restore.go:181-186`; `web/handler_export.go:312-353` no `restoreOpBlocked()`; `RemoveStackVolumes` before anything else, no copy |
|
||||||
|
| R8 | P2 med | `.fab` import drops the family gate, sign-up lock and version pin → a family-only app opens to anyone | `cmd/controller/main.go:3517-3530` bare `AppConfig`; `stacks/family_gate.go:173-190` removes the gate file when `FamilyGate == nil`; R-773 fix only on the unit path (`stacks/deploy.go:840-870`) |
|
||||||
|
| R9 | P3 med | `.fab` import trusts the bundle's HDD_PATH; empty → relative path; older userdata overwrites newer | `restore.go:589-631` |
|
||||||
|
| R10 | P3 high | `.fab` DB import: plain psql without ON_ERROR_STOP; postgis not classified | `restore.go:1052-1053`, `:928-937` |
|
||||||
|
| R11 | P3 med | volume replace never checks the old volume is gone | `restore.go:218` |
|
||||||
|
| R12 | P3 high | interrupted second-drive whole restore leaves `<dst>.felhom-restoring` (O_EXCL) → every retry fails „file exists" | `tier2_whole.go:86` |
|
||||||
|
| R13 | P3 design | full off-site download capped at 30 min per press; household not told to retry | `web/offbox_handlers.go:455` |
|
||||||
|
| R14 | P4 med | place/full restore re-read the LATEST snapshot instead of the downloaded one | `offbox_restore.go:675-679`, `offbox_reconstitute.go:657-661` |
|
||||||
|
|
||||||
|
**Inventory drift (July → now):** three more customer restore entries exist: the Tier-2 unit restore (R-102), the Tier-2 whole restore (decision 26, `tier2_whole.go`), the kept-data Load (`kept_load.go`).
|
||||||
|
|
||||||
|
**Checked SAFE (reviewer):** `.fab` path confinement (`appexport/restore.go:1104-1107,1152-1155`; `validate.go:28-66`; `web/handler_export.go:385-394`); off-site placement map refuses `..`/outside/`backups/` (`offbox_restore.go:603-640`); R-358 marker after restic success; full off-site restore fit/safety-dump/undo checks before stop, fail closed (`offbox_reconstitute.go:719-818, 909-927`); Tier-1 refusals before change; main loader `ON_ERROR_STOP=1 --single-transaction` (`appbackup/dbdump.go:775-784`); Tier-2 file restore `rsync --ignore-existing`, never `--delete`; whole restore newer-wins + Lchown; **NFC/NFD: no normalisation anywhere — every lane compares bytes, so the missing-only compare sees TWO names (consistent on one filesystem)**; agent RestoreLXC force only when asked. Not reviewed: `kept_load.go`, hub operator restore paths.
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
# C1 reviewer 4 — time and concurrency (fresh, read-only; controller 7ca59e2, agent 4e1621c, felhom.eu 5c48731)
|
||||||
|
|
||||||
|
## (A) Schedulers and what each does on Sunday 2026-10-25 (03:00 CEST → 02:00 CET; 02:00–02:59 happens twice)
|
||||||
|
|
||||||
|
| owner | job | schedule | zone | file:line | 2026-10-25 | after 3 days off |
|
||||||
|
|---|---|---|---|---|---|---|
|
||||||
|
| controller | db-dump | daily W (default 02:30) | Budapest (`time.Date`, day+1) | main.go:1262, scheduler.go:499-518 | once, at the SECOND 02:30 (CET = 01:30Z) — Go normalises the ambiguous time to the later offset | not replayed by the scheduler; the catch-up runs it once 15 min after start (nightchain.go:277-345) |
|
||||||
|
| controller | tier2-backup | W+60m | Budapest | main.go:1352 | once, 03:30 CET | once, by the catch-up |
|
||||||
|
| controller | offbox-backup + app-update leg | W+105m | Budapest | main.go:1519 | once, 04:15 CET | catch-up runs the off-site leg only |
|
||||||
|
| controller | offsite-abandon-sweep 05:10, offsite-proof 05:30, offsite-integrity 06:00 | fixed | Budapest | main.go:1561, 1613, 1591 | once | **no catch-up** (finding T6) |
|
||||||
|
| controller | metrics-prune 04:00, fill-watch 03:30, asset-sync 05:00, selfupdate-auto 04:30 (off by default) | fixed | Budapest | main.go:1621,1671,1768,1741 | once | floor update runs on any hub reply |
|
||||||
|
| controller | whole-guest quiesce loop | poll 5 min; gate [W+2h, W+6h) by wall minute; valve when age > cadence+24h | Budapest wall | quiesce.go:150, 191-205, 790-818 | gate 04:30–08:30 CET (4 h real) | valve fires ~5 min after start, any time of day |
|
||||||
|
| controller | ~20 `Every` tickers | monotonic | n/a | main.go:898-2146 | unaffected | restart at boot |
|
||||||
|
| agent | whole-guest due | age ≥ cadence | UTC | localapi/server.go:1182-1192 | unaffected | due at once |
|
||||||
|
| agent | restore-test | 30 min after start, then 6 h | UTC | backup/schedule.go:148-177 | unaffected | after 30 min |
|
||||||
|
| agent | OS leg (+ kernel reboot if hub „Tonight") | after any non-manual primary backup +90 s, 20 h min gap | no wall clock | server.go:982-986, main.go:1272-1280, leg.go:655-664, kernel.go:101-114 | follows the backup | follows the valve backup = daytime (T8) |
|
||||||
|
| agent | fstrim | hourly; due Wed 10:00, only 10:00–20:59 | host local | fstrim/fstrim.go:39-50,110 | n/a | next eligible hour |
|
||||||
|
| hub | scheduleDaily: db-snapshot 02:00 (keep 2), prune 04:30, deadline-check 05:00, offsite-key-audit 07:10 | `next.Add(24*time.Hour)` | Budapest | hub cmd/hub/main.go:1049-1072 | **every job runs TWICE** (T9) | n/a |
|
||||||
|
| host timers | mgmt-watchdog, crash-guard-check | monotonic | n/a | felhom-agent/configs/*.timer | unaffected | after boot |
|
||||||
|
| DooPlex timers | hub-db-backup 02:30; dooplex-offsite 00:20; hub-db-restore-test Sun 04:30; offsite-restore-test Sun 05:30; ep0-copy-gc 08:00 | OnCalendar local | DooPlex | felhom.eu/scripts/*/*.timer | Oct 25 is a Sunday: both weekly restore tests run on the DST day; 02:30 is in the repeated hour (systemd behaviour not verified) | Persistent=true |
|
||||||
|
|
||||||
|
## (B) Candidates
|
||||||
|
|
||||||
|
| # | Sev guess | harm | mechanism |
|
||||||
|
|---|---|---|---|
|
||||||
|
| T1 | P2 med-high | The nightly second-drive mirror (`rsync --delete`) can run DURING a restore from that same mirror and delete the files not yet copied back | `tier2.go:479-520` RunAllTier2 checks held/version/disconnected only; `:443` mirror, `:744-751` `--delete`; restore `tier2_whole.go:301` takes only `acquireRunning`; Tier-2 leg takes only `LegLock` (main.go:1348-1352); `update_guard.go:595-601` isHeld ignores restores |
|
||||||
|
| T2 | P2 if true; low-med | The Debian 13 controller image may lack tzdata → every schedule on UTC silently | `Dockerfile:77-100` debian:13.7-slim, no tzdata, no `time/tzdata` import; fallbacks scheduler.go:19-23 (logs), quiesce.go:749-751 (silent), funcmap.go:62-66 |
|
||||||
|
| T3 | P3 med | Whole-guest start creeps ~5 min/night until a night is skipped silently | agent `server.go:1184` `age >= Cadence` no slack; age from archive start; controller poll 5 min `quiesce.go:150`; gate refusal `:807-818` |
|
||||||
|
| T4 | P3 med-high | The whole-guest quiesce stops apps in the middle of a restore or an update | `quiesce.go:213-275` no `IsRunning()`/restore check; `RunningAppStacks` (stacks/manager.go:620-636); `TriggerNow` (:419-445) same; `DumpAppVolumesSafe` (backup.go:1001-1045) can restart an app quiesce stopped |
|
||||||
|
| T5 | P3 high | The nightly off-site copy is skipped SILENTLY when anything else holds the backup flag, and the catch-up never retries | `offbox.go:981-996` INFO „skipped — another backup is running" return nil; `withLeg` MarkEnded unconditionally (main.go:1246); proof 05:30 / integrity 06:00 take the same flag |
|
||||||
|
| T6 | P3 high | Off-site check and proof never run on a box off at 05:30–06:00; the code comment says the opposite | main.go:1575-1577 comment vs 1591-1619; `catchUpLegs` nightchain.go:17-18; no hub alarm on integrity/proof age |
|
||||||
|
| T7 | P3 high | Hub raises a false „no DB dump" error every morning for W after ~04:45 or evening W | hub `monitor/deadline.go:416-419` midnight Budapest; `:556-565` → `expected_dbdump_missed`; hub has no idea of W |
|
||||||
|
| T8 | P3 design | The OS leg and kernel reboot follow ANY scheduled backup, including the daytime valve after power-on | agent server.go:982-986; main.go:1279 labels „night"; kernel.go:101-114; hub osupdates/kernel.go:46-48,181-185 |
|
||||||
|
| T9 | P4 high | Hub daily jobs run TWICE on 25 October (db-snapshot keep-2 prunes a day early; duplicate deadline alarms); once an hour late in spring | hub cmd/hub/main.go:1058-1062 `next.Add(24*time.Hour)`; the controller fixed this (scheduler.go:511-516) |
|
||||||
|
| T10 | P4 high | Spring forward 2027-03-28: db-dump and tier2 fire at the same instant; tier2 may mirror yesterday's unit | scheduler normalisation; `LegLock` main.go:1237 plain mutex |
|
||||||
|
| T11 | P4 design | A clock ahead stops whole-guest backups silently (same as D3) | server.go:1182-1192; quiesce.go:816-817; deadline.go:244; backup_cap.go:61 |
|
||||||
|
| T12 | P4 high | self-update busy check before the image pull, swap after it with no re-check | updater.go:532-545, :764 |
|
||||||
|
| T13 | P4 med | double-clicking a restore shows it as failed while it runs | handlers.go:1853, :1874 `BeginRestoreOp` after the check; second POST's `EndRestoreOp(false)` |
|
||||||
|
|
||||||
|
**Checked SAFE (reviewer):** controller DST day+1 arithmetic and late-fire skip (scheduler.go:342-348, 511-516); night ledger persisted, seeded, one catch-up for many nights; quiesce marker persisted and recovered; in-memory backup flag (no stale lock); flock-based OS-leg locks; restic stale locks self-heal; two restores blocked by `m.running`; Remove refused during restore/quiesce; catch-up and whole-guest wait for each other; agent due/cap UTC durations; off-site prune guard refuses future snapshots; host power-off re-checks; no cron/OnCalendar in the installer; hub staleness uses the hub's own receive clock.
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
# C2 register search, 2026-10-20T10:11:13Z, OPEN-ITEMS.md + CLOSED-ITEMS.md at c701b253; grep -ciE
|
||||||
|
1 open / 2 closed : R-893
|
||||||
|
0 open / 0 closed : tier2_shares|_shares.*tier|shares.*second.drive|Megoszt.*2\. ment
|
||||||
|
0 open / 2 closed : host_deletions|retained.*delet|deleted host|escrow.*reinstall
|
||||||
|
2 open / 7 closed : scheduleDaily|DST|daylight|oraallit|time.?zone
|
||||||
|
1 open / 8 closed : restore-undo|undo copy|safety dump|unit restore.*fail
|
||||||
|
1 open / 0 closed : ValidateDump|dump.*replac.*valid
|
||||||
|
0 open / 0 closed : SharesScratchReady|shares.*scratch|shares restore
|
||||||
|
0 open / 4 closed : tier2_whole|whole restore|Teljes vissza
|
||||||
|
0 open / 3 closed : optional leg|missing on disk|--delete
|
||||||
|
0 open / 0 closed : PurgeSuperseded|abandon.*purge
|
||||||
|
0 open / 1 closed : future.dated|clock ahead|clock.*future|negative age
|
||||||
|
0 open / 0 closed : another backup is running
|
||||||
|
1 open / 2 closed : expected_dbdump_missed
|
||||||
|
2 open / 1 closed : symlink
|
||||||
|
0 open / 1 closed : family.gate.*fab|fab.*family|\.fab
|
||||||
|
0 open / 0 closed : felhom-restoring
|
||||||
|
0 open / 0 closed : quiesce.*restore|restore.*quiesce
|
||||||
|
1 open / 4 closed : offsite-proof|integrity.*catch|proof.*catch
|
||||||
|
0 open / 0 closed : double.click|dupla
|
||||||
|
0 open / 1 closed : disconnected.*target|away.*target
|
||||||
|
Every non-zero hit was read: none describes the candidate (positive control R-893 found in both).
|
||||||
@@ -0,0 +1,51 @@
|
|||||||
|
# C2 — the main session's verdict on every candidate (2026-10-20)
|
||||||
|
|
||||||
|
Method: each candidate's `file:line` read in LIVE source (controller `7ca59e2` = v0.336.0, agent `4e1621c`, felhom.eu
|
||||||
|
`5c48731`+); then (a) the register searched, open AND closed, with ASCII fragments — positive control `R-893` found in
|
||||||
|
both; (b) the design test against `07-backup-architecture.md` and `CONTEXT.md`. Verdict words:
|
||||||
|
|
||||||
|
- **CONFIRMED** — the mechanism read true in live source AND a proof ran (unit test against production code, or live).
|
||||||
|
- **CONFIRMED-CODE** — the mechanism read true in live source; no proof ran yet.
|
||||||
|
- **PLAUSIBLE** — the reviewer's reading, spot-checked only.
|
||||||
|
- **NOT-A-FINDING** — with the reason.
|
||||||
|
|
||||||
|
Register: none of the candidates below is in OPEN-ITEMS or CLOSED-ITEMS (searches in `C2-register-search.txt`).
|
||||||
|
|
||||||
|
| id | verdict | design or defect | note |
|
||||||
|
|---|---|---|---|
|
||||||
|
| **D1** shares second-drive copy pruned by another drive's group | **CONFIRMED** (unit test, production code, fake mirror seam — `C3-D1-shares-tier2/`) | defect | **Worse than reported:** with shares on two drives that pick the same target, EVERY night the second group's reconcile deletes the first group's copy (`removed stale dest dir hdd_2/b_share`) — one drive's shares never have a second copy while the run says ok. And the night a drive goes away, its copy is deleted (`removed stale dest dir hdd_3/csaladi_fotok`). |
|
||||||
|
| D1b an AWAY drive is still chosen as the second-drive target | CONFIRMED-CODE (+ the same unit test: `hdd_3/iskola => hdd_1/...` after hdd_1 was marked disconnected) | defect | `tier2.go:151-180` loops `GetSchedulableStoragePaths()` (`settings.go:1593-1603` excludes only Decommissioned); no Disconnected check. On a real box the away drive's empty mount point is on the guest's root disk |
|
||||||
|
| **K1** reinstall after „delete the old host" loses the route to the old escrow | **CONFIRMED** at the hub store (unit test against production code, `C3-K1-escrow-after-host-delete/`); the controller half (new key minted, history set aside on night one) CONFIRMED-CODE | defect (03 §8a / 07 §7.0 promise the route; D19+R-1154 removed it unrecorded) | hub `store.go:3164-3229` demotes into `host_escrow_superseded` under the OLD host_id and deletes the `hosts` row; `GetEscrowStatusForCustomer` `store.go:3499-3527` joins on `hosts` → nil for the new box; `customer_reset.go:48,155` and `store.go:4189` DO include `host_deletions`. Live proof needs Tester4 off-site ON (B7) |
|
||||||
|
| K2 abandoning one set-aside copy purges every retained package | CONFIRMED-CODE (`store.go:4185-4190`) → R-1182 | defect (breaks `store.go:4176-4179`'s own invariant) | `store.go:4185-4190` customer-wide delete read true |
|
||||||
|
| **R1/D4** a failed LOCAL unit restore: no copy of the live state, app started anyway | **CONFIRMED LIVE** on 9202 (`C3-R1-local-restore-9202/`): live row destroyed, DB crash-loops, no undo copy, raw English Docker text to the household | **design question** for the operator: decision 154 (R-638 A) chose „start anyway"; D20 (R-893 A) gave „put back exactly" to the OFF-SITE path only | `restore_unit.go:428-489` |
|
||||||
|
| R2 a crash mid-restore starts the app on a mixed state; the undo copy is never read at start | PLAUSIBLE | defect | `appstop_marker.go:307`, `offbox_restore_undo.go:137` |
|
||||||
|
| R3 a cut-off shares restore can be placed and reported restored | CONFIRMED-CODE | defect (R-358 class, fixed for apps, not shares) | `shares_restore.go:131-140` readiness = non-empty dir |
|
||||||
|
| R4 `.fab` export drops a drive folder holding a symlink | PLAUSIBLE | defect | `appexport/fabplan.go:292`, `export.go:714` |
|
||||||
|
| R5 second-drive whole restore: files swapped, then the unit half refuses → app left STOPPED, text says „untouched" | **CONFIRMED** (unit test, production code: `C3-R5-whole-restore/` — 1 file restored + 1 replaced, app stopped, no start, message „az alkalmazás és az adatai érintetlenek") | defect | `tier2_whole.go:301-340`: stop + merge, then `RestoreFromRecoveryUnitAtWith` whose early refusals return before its own stop/start; `hu.json` `err.backup.unit_versions_mixed` „Az alkalmazás érintetlen" |
|
||||||
|
| R6 `.fab` import into a stopped app merges volumes | PLAUSIBLE | defect | `appexport/restore.go:361-374` |
|
||||||
|
| R7 `.fab` import has no guard / undo / record | PLAUSIBLE | defect | |
|
||||||
|
| R8 `.fab` import drops the family gate | **CONFIRMED LIVE** on 9202 (`C3-R1187e-fab-family-gate/`: stranger 401 before, 200 after the import) → **R-1191 (P1)** | defect | `cmd/controller/main.go:3517-3530`, `stacks/family_gate.go:173-190` |
|
||||||
|
| R9–R11, R13, R14 | PLAUSIBLE | defect / design (R13) | low |
|
||||||
|
| R12 interrupted whole restore leaves `*.felhom-restoring`, every retry fails | **CONFIRMED** (unit test: `C3-R12-leftover-temp/`) → R-1188 (c) | defect | `tier2_whole.go:86-87` O_EXCL; removed only on an in-process error |
|
||||||
|
| **D2** a missing / emptied app folder deletes its second-drive copy within one night | **CONFIRMED** for a MISSING folder (unit test, production code + real copy seam: `C3-D2-missing-leg/` — mirror of `photos` deleted, run „ok", the warning says only „nem kerültek a másodlagos mentésbe"); the EMPTIED case CONFIRMED-CODE (`rsyncMirror` = `rsync -a --delete`, `tier2.go:744-751`; the test seam does not delete) | defect-shaped corner of a choice: 07 §8.2 „data legs are NOT guarded" is about a leg LEAVING the class set; this is a declared leg that is MISSING, which the code itself logs as a gap (`tier2_capture.go:55-61`) and then reconciles away (`tier2.go:454`) | P1 for optional legs (no other tier holds them) |
|
||||||
|
| D3 / T11 a host clock ahead silences the whole-guest tier; the hub reads the same clock | PLAUSIBLE (B6 #8 on the rehearsal VM is its live test) | defect | agent `localapi/server.go:1178-1190` |
|
||||||
|
| D5 a dump that fails its own check has already replaced the good one | **CONFIRMED** (unit test with a docker stub: `C3-D5-dump-replaced/`) → R-1185 | defect for the ORDER | `appbackup/dbdump.go:388-422`: size check, rename (`:399`), then `ValidateDump` |
|
||||||
|
| D6 kept-data page can list a running app's folder | PLAUSIBLE | defect | needs an unreadable app.yaml |
|
||||||
|
| D7 unit restore can point the app at an old drive | PLAUSIBLE (low) | defect | |
|
||||||
|
| D8 local whole-guest keep=1 | design (07 §6.1 R-684 chose it) | design | **NOT-A-FINDING: the operator chose keep=1 for demo-hp; recorded in 07** |
|
||||||
|
| T1 the nightly mirror (`rsync --delete`) can run during a restore from that mirror | **CONFIRMED** (unit test, production rsync mirror: `C3-T1-mirror-during-restore/`) → R-1180 | defect | `RunAllTier2` (`tier2.go:479-520`) takes no backup flag; the whole restore holds only `acquireRunning` (`tier2_whole.go:301`) |
|
||||||
|
| T2 controller image without tzdata → UTC schedules | **NOT-A-FINDING:** measured on 9202 (controller 0.331.0, Debian 13 base): `/usr/share/zoneinfo/Europe/Budapest` present, scheduler logs „scheduled for … CEST" | — | |
|
||||||
|
| T3 whole-guest start creeps ~5 min/night | PLAUSIBLE | defect | needs the agent journal over weeks |
|
||||||
|
| T4 whole-guest quiesce stops apps mid-restore | PLAUSIBLE | defect | `quiesce.go:213-275` |
|
||||||
|
| T5 nightly off-site copy skipped SILENTLY when the flag is held; catch-up never retries | CONFIRMED-CODE → R-1186 | defect (the code chose nil for the nightly path; the catch-up gap is not chosen) | `offbox.go:981-996`, `main.go:1246` |
|
||||||
|
| T6 proof / integrity never run on a box off at 05:30–06:00 | PLAUSIBLE | defect (comment `main.go:1575-1577` says the opposite) | |
|
||||||
|
| T7 hub false „no DB dump" for late or evening W | PLAUSIBLE | defect | hub `monitor/deadline.go:416-419, 556-565` |
|
||||||
|
| T8 the OS leg follows a daytime valve backup | design question | design | 07 §6.1 / 11 §8.1 set no time bound |
|
||||||
|
| **T9** hub daily jobs run TWICE on 2026-10-25 | **CONFIRMED** (replay of the hub's exact arithmetic, `C3-T9-hub-dst/replay.txt`) | defect | deadline-check fires 04:00 CET AND 05:00 CET; db-snapshot twice (keep-2 → the previous day's copy pruned a day early). **In five days, on the live hub.** |
|
||||||
|
| T10 spring forward collision | PLAUSIBLE | defect (P4) | |
|
||||||
|
| T12, T13 | PLAUSIBLE | defect (P4) | |
|
||||||
|
| K3–K8 | PLAUSIBLE | K6 design | |
|
||||||
|
|
||||||
|
**NFC/NFD (the brief's question for A.1.4):** reviewer 1 found no normalisation anywhere — rsync, restic and the Go
|
||||||
|
merge compare names byte for byte, so the missing-only compare sees **two names** for the NFC/NFD pair. The household
|
||||||
|
data carries the pair; B8 measures it live.
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# go test at controller main 7ca59e2 (v0.336.0), scratch clone, fake mirror seam (no rsync/docker); 2026-10-20T10:04:31Z
|
||||||
|
=== RUN TestRehearsalD1_DeadDriveShareCopyIsPruned
|
||||||
|
zz_rehearsal_d1_test.go:46: drive keys: map[hdd_1:hdd_1 hdd_2:hdd_2 hdd_3:hdd_3]
|
||||||
|
zz_rehearsal_d1_test.go:47: POSITIVE CONTROL — copies after run 1 (both drives present): [hdd_1: hdd_3/iskola hdd_2: hdd_1/csaladi_fotok]
|
||||||
|
zz_rehearsal_d1_test.go:48: mirror calls run 1: [hdd_1/csaladi_fotok=>hdd_2/backups/secondary/_shares/hdd_1/csaladi_fotok data/shares-payload=>hdd_2/backups/secondary/_shares/_payload hdd_3/iskola=>hdd_1/backups/secondary/_shares/hdd_3/iskola data/shares-payload=>hdd_1/backups/secondary/_shares/_payload]
|
||||||
|
zz_rehearsal_d1_test.go:56: FAULT — copies after run 2 (hdd_1 away): [hdd_1: hdd_3/iskola hdd_2: hdd_1/csaladi_fotok]
|
||||||
|
zz_rehearsal_d1_test.go:57: mirror calls run 2: [hdd_3/iskola=>hdd_1/backups/secondary/_shares/hdd_3/iskola data/shares-payload=>hdd_1/backups/secondary/_shares/_payload]
|
||||||
|
--- PASS: TestRehearsalD1_DeadDriveShareCopyIsPruned (0.02s)
|
||||||
|
=== RUN TestRehearsalD1_TwoGroupsOneTarget
|
||||||
|
zz_rehearsal_d1_test.go:73: copies after two runs: [hdd_1: hdd_3/c_share hdd_2: hdd_1/a_share]
|
||||||
|
zz_rehearsal_d1_test.go:75: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_1 → hdd_2/backups/secondary/_shares (18 B)
|
||||||
|
zz_rehearsal_d1_test.go:75: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_3 → hdd_1/backups/secondary/_shares (36 B)
|
||||||
|
zz_rehearsal_d1_test.go:75: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_1 → hdd_2/backups/secondary/_shares (18 B)
|
||||||
|
zz_rehearsal_d1_test.go:75: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_3 → hdd_1/backups/secondary/_shares (36 B)
|
||||||
|
--- PASS: TestRehearsalD1_TwoGroupsOneTarget (0.01s)
|
||||||
|
=== RUN TestRehearsalD1_SameTargetThenDriveDies
|
||||||
|
zz_rehearsal_d1_test.go:89: POSITIVE CONTROL run 1: [hdd_1: hdd_3/csaladi_fotok]
|
||||||
|
zz_rehearsal_d1_test.go:91: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_2 → hdd_1/backups/secondary/_shares (18 B)
|
||||||
|
zz_rehearsal_d1_test.go:91: LOG [INFO] [backup] Tier 2 reconcile: removed stale dest dir hdd_2/b_share
|
||||||
|
zz_rehearsal_d1_test.go:91: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_3 → hdd_1/backups/secondary/_shares (42 B)
|
||||||
|
zz_rehearsal_d1_test.go:100: FAULT run 2 (hdd_3 away): [hdd_1: hdd_2/b_share]
|
||||||
|
zz_rehearsal_d1_test.go:102: LOG [INFO] [backup] Tier 2 reconcile: removed stale dest dir hdd_3/csaladi_fotok
|
||||||
|
zz_rehearsal_d1_test.go:102: LOG [INFO] [shares] tier-2 copied 1 share(s) from hdd_2 → hdd_1/backups/secondary/_shares (18 B)
|
||||||
|
--- PASS: TestRehearsalD1_SameTargetThenDriveDies (0.01s)
|
||||||
|
PASS
|
||||||
|
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.049s
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate D1). Never committed.
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"sort"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func listShareCopies(env *sharesEnv) []string {
|
||||||
|
var out []string
|
||||||
|
for name, d := range env.drives {
|
||||||
|
base := filepath.Join(NamespaceRoot(d, true), "backups", "secondary", SharesPseudoStack)
|
||||||
|
keys, _ := os.ReadDir(base)
|
||||||
|
for _, k := range keys {
|
||||||
|
if !k.IsDir() || k.Name() == filepath.Base(sharesPayloadDestRel) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
subs, _ := os.ReadDir(filepath.Join(base, k.Name()))
|
||||||
|
for _, s := range subs {
|
||||||
|
out = append(out, name+": "+k.Name()+"/"+s.Name())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
sort.Strings(out)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
func keyOwner(env *sharesEnv) map[string]string {
|
||||||
|
m := map[string]string{}
|
||||||
|
for name, d := range env.drives {
|
||||||
|
m[sharesDriveKey(d)] = name
|
||||||
|
}
|
||||||
|
return m
|
||||||
|
}
|
||||||
|
|
||||||
|
// A drive whose shares were mirrored goes away; another share group's run lands on the same target.
|
||||||
|
func TestRehearsalD1_DeadDriveShareCopyIsPruned(t *testing.T) {
|
||||||
|
env := newSharesEnv(t, "hdd_1", "hdd_2", "hdd_3")
|
||||||
|
env.addShare(t, "hdd_1", "csaladi_fotok", true)
|
||||||
|
env.addShare(t, "hdd_3", "iskola", true)
|
||||||
|
if err := env.m.RunSharesTier2(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Logf("drive keys: %v", keyOwner(env))
|
||||||
|
t.Logf("POSITIVE CONTROL — copies after run 1 (both drives present): %v", listShareCopies(env))
|
||||||
|
t.Logf("mirror calls run 1: %v", env.mirrored)
|
||||||
|
if err := env.sett.SetDisconnected(env.drives["hdd_1"], true, nil); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
env.mirrored = nil
|
||||||
|
if err := env.m.RunSharesTier2(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Logf("FAULT — copies after run 2 (hdd_1 away): %v", listShareCopies(env))
|
||||||
|
t.Logf("mirror calls run 2: %v", env.mirrored)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Shares on two drives that both pick the SAME target: does group A's reconcile remove group B's copy?
|
||||||
|
func TestRehearsalD1_TwoGroupsOneTarget(t *testing.T) {
|
||||||
|
env := newSharesEnv(t, "hdd_1", "hdd_2", "hdd_3")
|
||||||
|
env.addShare(t, "hdd_1", "a_share", true)
|
||||||
|
env.addShare(t, "hdd_3", "c_share", true)
|
||||||
|
var removed []string
|
||||||
|
env.m.logger = newCaptureLogger(&removed)
|
||||||
|
if err := env.m.RunSharesTier2(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := env.m.RunSharesTier2(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Logf("copies after two runs: %v", listShareCopies(env))
|
||||||
|
for _, l := range removed {
|
||||||
|
t.Logf("LOG %s", l)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Shares on hdd_2 and hdd_3 (both pick hdd_1 as target). Then hdd_3 dies.
|
||||||
|
func TestRehearsalD1_SameTargetThenDriveDies(t *testing.T) {
|
||||||
|
env := newSharesEnv(t, "hdd_1", "hdd_2", "hdd_3")
|
||||||
|
env.addShare(t, "hdd_2", "b_share", true)
|
||||||
|
env.addShare(t, "hdd_3", "csaladi_fotok", true)
|
||||||
|
var logs []string
|
||||||
|
env.m.logger = newCaptureLogger(&logs)
|
||||||
|
if err := env.m.RunSharesTier2(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Logf("POSITIVE CONTROL run 1: %v", listShareCopies(env))
|
||||||
|
for _, l := range logs {
|
||||||
|
t.Logf(" LOG %s", l)
|
||||||
|
}
|
||||||
|
logs = nil
|
||||||
|
if err := env.sett.SetDisconnected(env.drives["hdd_3"], true, nil); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := env.m.RunSharesTier2(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Logf("FAULT run 2 (hdd_3 away): %v", listShareCopies(env))
|
||||||
|
for _, l := range logs {
|
||||||
|
t.Logf(" LOG %s", l)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
import (
|
||||||
|
"log"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
type capW struct{ dst *[]string }
|
||||||
|
|
||||||
|
func (c capW) Write(p []byte) (int, error) {
|
||||||
|
s := string(p)
|
||||||
|
if strings.Contains(s, "reconcile") || strings.Contains(s, "tier-2 copied") {
|
||||||
|
*c.dst = append(*c.dst, strings.TrimSpace(s))
|
||||||
|
}
|
||||||
|
return len(p), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func newCaptureLogger(dst *[]string) *log.Logger { return log.New(capW{dst}, "", 0) }
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
# go test at controller 7ca59e2 (v0.336.0), scratch clone; the real copyTree mirror seam (rsync --delete stands in by copyTree for the emptied case — see note); 2026-10-20T10:19:17Z
|
||||||
|
=== RUN TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror
|
||||||
|
zz_rehearsal_d2_test.go:22: POSITIVE CONTROL after run 1: mirror photo=true doc=true
|
||||||
|
zz_rehearsal_d2_test.go:33: run 2 err=<nil>; status="ok" warning="Figyelmeztetés: a(z) app alkalmazás egyes adatmappái nem kerültek a másodlagos mentésbe: appdata/app/photos."
|
||||||
|
zz_rehearsal_d2_test.go:34: FAULT after run 2: mirror photo=false doc=true (the second-drive copy is the household's only local copy of these files)
|
||||||
|
--- PASS: TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror (0.02s)
|
||||||
|
PASS
|
||||||
|
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.030s
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate D2). Never committed.
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRehearsalD2_MissingAndEmptiedLegDeleteTheirMirror(t *testing.T) {
|
||||||
|
m, src, target, prov := newTier2V2(t, "app")
|
||||||
|
prov.has["app"] = true
|
||||||
|
prov.binds["app"] = []ClassifiedBind{mHDD("appdata/app/photos"), mHDD("appdata/app/docs")}
|
||||||
|
mustWrite(t, filepath.Join(src, "appdata", "app", "photos", "family.jpg"), "PHOTO")
|
||||||
|
mustWrite(t, filepath.Join(src, "appdata", "app", "docs", "contract.pdf"), "PDF")
|
||||||
|
if err := m.RunTier2("app"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
destBase := filepath.Join(target, "backups", "secondary", "app")
|
||||||
|
photo := filepath.Join(destBase, "hdd", "appdata", "app", "photos", "family.jpg")
|
||||||
|
doc := filepath.Join(destBase, "hdd", "appdata", "app", "docs", "contract.pdf")
|
||||||
|
t.Logf("POSITIVE CONTROL after run 1: mirror photo=%v doc=%v", exists(photo), exists(doc))
|
||||||
|
|
||||||
|
// The household (or an app bug, or fsck → lost+found) loses the live photos folder; the docs folder is emptied.
|
||||||
|
if err := os.RemoveAll(filepath.Join(src, "appdata", "app", "photos")); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.Remove(filepath.Join(src, "appdata", "app", "docs", "contract.pdf")); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
err := m.RunTier2("app")
|
||||||
|
cd := m.settings.GetCrossDriveConfig("app")
|
||||||
|
t.Logf("run 2 err=%v; status=%q warning=%q", err, cd.LastStatus, cd.LastWarning)
|
||||||
|
t.Logf("FAULT after run 2: mirror photo=%v doc=%v (the second-drive copy is the household's only local copy of these files)", exists(photo), exists(doc))
|
||||||
|
}
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
# go test at controller 7ca59e2 (v0.336.0), scratch clone; a docker STUB on PATH (no real docker reached); 2026-10-20T11:07:38Z
|
||||||
|
=== RUN TestRehearsalD5_InvalidDumpReplacesTheGoodOne
|
||||||
|
zz_rehearsal_d5_test.go:26: POSITIVE CONTROL: last night's dump, 152 B, ValidateDump={Valid:true TableCount:1 Error: FileSize:152 ModTime:2026-10-20 13:07:39.022566077 +0200 CEST UserTableFound:false UserRows:0 LooksEmpty:false}
|
||||||
|
2026/10/20 13:07:39 [WARN] [backup] ValidateDump FAIL: /tmp/TestRehearsalD5_InvalidDumpReplacesTheGoodOne2769942191/002/unit/app-postgres.sql — dump file too small (< 100 bytes)
|
||||||
|
zz_rehearsal_d5_test.go:31: tonight's dump: err=<nil> validation={Valid:false TableCount:0 Error:dump file too small (< 100 bytes) FileSize:99 ModTime:2026-10-20 13:07:39.026566071 +0200 CEST UserTableFound:false UserRows:0 LooksEmpty:false}
|
||||||
|
zz_rehearsal_d5_test.go:32: FAULT: the file on disk now = 99 B, still holds CREATE TABLE photos: false
|
||||||
|
--- PASS: TestRehearsalD5_InvalidDumpReplacesTheGoodOne (0.00s)
|
||||||
|
PASS
|
||||||
|
ok gitea.dooplex.hu/admin/felhom-controller/internal/appbackup 0.008s
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
package appbackup
|
||||||
|
|
||||||
|
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate D5 / R-1185). Never committed. A docker STUB on PATH
|
||||||
|
// (no real docker is reached) prints a header-only dump, as a database that was re-initialised empty would.
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRehearsalD5_InvalidDumpReplacesTheGoodOne(t *testing.T) {
|
||||||
|
bin := t.TempDir()
|
||||||
|
stub := "#!/bin/sh\ncase \"$1\" in\n inspect) echo true ;;\n exec) printf -- '-- PostgreSQL database dump\\nSET statement_timeout = 0;\\n\\n--\\n-- PostgreSQL database dump complete\\n--\\n' ;;\nesac\n"
|
||||||
|
if err := os.WriteFile(filepath.Join(bin, "docker"), []byte(stub), 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Setenv("PATH", bin)
|
||||||
|
dumpDir := filepath.Join(t.TempDir(), "unit")
|
||||||
|
final := filepath.Join(dumpDir, "app-postgres.sql")
|
||||||
|
good := "-- PostgreSQL database dump\nCREATE TABLE public.photos (id int);\nCOPY public.photos (id) FROM stdin;\n1\n2\n\\.\n\n--\n-- PostgreSQL database dump complete\n--\n"
|
||||||
|
os.MkdirAll(dumpDir, 0o755)
|
||||||
|
os.WriteFile(final, []byte(good), 0o644)
|
||||||
|
t.Logf("POSITIVE CONTROL: last night's dump, %d B, ValidateDump=%+v", len(good), ValidateDump(final, DBTypePostgres))
|
||||||
|
|
||||||
|
db := DiscoveredDB{ContainerName: "app-db", ContainerID: "0123456789abcdef", DBType: DBTypePostgres, DBUser: "u", DBName: "d", StackName: "app"}
|
||||||
|
res := DumpOneTo(context.Background(), db, final, log.New(io.Discard, "", 0), false)
|
||||||
|
b, _ := os.ReadFile(final)
|
||||||
|
t.Logf("tonight's dump: err=%v validation=%+v", res.Error, res.Validation)
|
||||||
|
t.Logf("FAULT: the file on disk now = %d B, still holds CREATE TABLE photos: %v", len(b), contains(string(b), "CREATE TABLE public.photos"))
|
||||||
|
}
|
||||||
|
|
||||||
|
func contains(s, sub string) bool { return len(s) >= len(sub) && (func() bool { for i := 0; i+len(sub) <= len(s); i++ { if s[i:i+len(sub)] == sub { return true } }; return false })() }
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
# go test, hub store at felhom.eu 5c48731 (scratch clone), in-memory SQLite; 2026-10-20T10:12:12Z
|
||||||
|
=== RUN TestRehearsalK1_NewBoxAfterHostDeleteSeesNoEscrow
|
||||||
|
zz_rehearsal_k1_test.go:20: POSITIVE CONTROL (old host live): status=&{IdentityBlobPresent:true ResticPwSHA256:SHA_OLD CreatedAt:2026-10-18T00:00:00Z SupersededPresent:false SupersededAt:} err=<nil>
|
||||||
|
zz_rehearsal_k1_test.go:27: retained rows under the DELETED host_id: 1 (identity blob present: true)
|
||||||
|
zz_rehearsal_k1_test.go:34: FAULT (new box, what its ACK carries): status=<nil> err=<nil>
|
||||||
|
zz_rehearsal_k1_test.go:36: => GetEscrowStatusForCustomer returns nil: identity_blob_present=false AND superseded_present=false for the new box
|
||||||
|
zz_rehearsal_k1_test.go:39: retained rows reachable under the NEW host_id: 0
|
||||||
|
--- PASS: TestRehearsalK1_NewBoxAfterHostDeleteSeesNoEscrow (0.04s)
|
||||||
|
PASS
|
||||||
|
ok gitea.dooplex.hu/admin/felhom-hub/internal/store 0.048s
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
package store
|
||||||
|
|
||||||
|
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate K1). Never committed.
|
||||||
|
import "testing"
|
||||||
|
|
||||||
|
func TestRehearsalK1_NewBoxAfterHostDeleteSeesNoEscrow(t *testing.T) {
|
||||||
|
s := newTestStore(t)
|
||||||
|
const cust = "Tester4"
|
||||||
|
// The household's first box, with an escrowed package (K blob + the identity blob that opens the off-site repo).
|
||||||
|
if err := s.UpsertHost(&Host{HostID: "Tester4-old", CustomerID: cust, APIKey: "k1"}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, _, err := s.SaveHostEscrow("Tester4-old", []byte("k-blob"), "fp", "zk", "2026-10-18T00:00:00Z", "SHA_OLD"); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := s.SaveHostDRBundle("Tester4-old", []byte("identity-opening-the-old-repo")); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
st, err := s.GetEscrowStatusForCustomer(cust)
|
||||||
|
t.Logf("POSITIVE CONTROL (old host live): status=%+v err=%v", st, err)
|
||||||
|
|
||||||
|
// The operator deletes the old host (the installer's 409 tells them to), escrow acknowledged → demoted.
|
||||||
|
if err := s.DeleteHost("Tester4-old", true); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
ret, _ := s.ListSupersededEscrow("Tester4-old")
|
||||||
|
t.Logf("retained rows under the DELETED host_id: %d (identity blob present: %v)", len(ret), len(ret) == 1 && len(ret[0].IdentityBlob) > 0)
|
||||||
|
|
||||||
|
// The reinstalled box enrolls under a NEW host_id for the same customer.
|
||||||
|
if err := s.UpsertHost(&Host{HostID: "Tester4-new", CustomerID: cust, APIKey: "k2"}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
st, err = s.GetEscrowStatusForCustomer(cust)
|
||||||
|
t.Logf("FAULT (new box, what its ACK carries): status=%+v err=%v", st, err)
|
||||||
|
if st == nil {
|
||||||
|
t.Logf("=> GetEscrowStatusForCustomer returns nil: identity_blob_present=false AND superseded_present=false for the new box")
|
||||||
|
}
|
||||||
|
ret2, _ := s.ListSupersededEscrow("Tester4-new")
|
||||||
|
t.Logf("retained rows reachable under the NEW host_id: %d", len(ret2))
|
||||||
|
}
|
||||||
@@ -0,0 +1,306 @@
|
|||||||
|
Tue Oct 20 10:09:14 UTC 2026
|
||||||
|
filebrowser Up About an hour (healthy)
|
||||||
|
felhom-controller Up About an hour (healthy)
|
||||||
|
paperless-webserver Up 3 hours (healthy)
|
||||||
|
paperless-postgres Up 3 hours (healthy)
|
||||||
|
paperless-redis Up 3 hours (healthy)
|
||||||
|
immich-server Up 3 hours (healthy)
|
||||||
|
immich-postgres Up 3 hours (healthy)
|
||||||
|
immich-machine-learning Up 3 hours (healthy)
|
||||||
|
immich-redis Up 3 hours (healthy)
|
||||||
|
traefik Up 4 days
|
||||||
|
0a4fb1a71c72cf704d1afd44e1ab24a8a68e11b8d82ef1dcb71a75d61a9dc1a3
|
||||||
|
0ce36c2e6db54e085f71a1436c4c89a9807a930c373198f451fa1665f90f0cab
|
||||||
|
0e2ca5e8738b95134285a666b1a7622bfcabc4728fb9269cc972dfaccbd59d45
|
||||||
|
0f4002ab75695fb312775a8203cc5213b256f1b01d255210cb5cdd0e50cd9070
|
||||||
|
1a8fe48409be3defe01713e2f621091630b965e1805cbb5872d1738043efd35c
|
||||||
|
1afe3420067df65e0d3e53baaec3debc39d2d42d333c7a3c266e4f6ff98e04f7
|
||||||
|
1b41940af5c4fa82af59320d6c81ba622d97f2e43d8ec51c50dcf65c2fdcb1e8
|
||||||
|
1f5120663962c7b03539bb993ab736be4800096e3d1f7147eb922d44b308ae6a
|
||||||
|
01feef94eb7e38a3679921b66cfe206e08adfd9b360ed4a60414cde606de1643
|
||||||
|
2a71e17dfa1e930ffe0c213cd1615c8d1adcbbf8d61615a38064d331c265c02e
|
||||||
|
2aaab84bf843817d9e6f32b673277fd1e0c2c8c2d038b8126df44927b1b41006
|
||||||
|
2ac49fc9cc2839f55c8abae76039950d15a5c301ea5b3918fa3e5d8715a15f42
|
||||||
|
2bbdb463b29493d60d352bafefccbe57856cf80bd2cc193b61f9bcc1310fb45e
|
||||||
|
2bc95b104191ffd31bdfdbe1e038d284d0624c04735094494c2c77d957e8e905
|
||||||
|
2dcb4bd2c3c618d71f582d33d5b1243e6ab075d3b00abc238d4430668a2b6f55
|
||||||
|
2e4c79db25c47436b44245376b1f6ff1991208ba081fddce1938c3897fc63c07
|
||||||
|
2ef8994a563ae88024ba29db4e349d26e6e3889e0683d30d1fb258208c9a10b4
|
||||||
|
2f08e882c5d23373e2df5486dce28d4728330f9f0131a2735ebc938bfb4d7b3a
|
||||||
|
02f42b55ee403ceef63a472c3624da0b48fab9e6aa6dc0c7800ec68b2f2041ef
|
||||||
|
3a3d7bc33a1ba21ae4a39c8cb345254cf66cd1dc60afae64e2299513b094b93b
|
||||||
|
3ac9820f84d0a50f233c63a62ee1b63b92acac026d68c60a8227f370b0be00b6
|
||||||
|
3d9a93769342e83a3c512d517b99cd64f8e794eb305c009e25c73a6f6627ad29
|
||||||
|
3d9c1f23abbcef771a4c38a75d75865a12d76a4359fe83c1828652dba3f91710
|
||||||
|
3f015f020f49d4967d2ac71b94060e6eaec86bc8852cf9a0b5f3dd8133dd528c
|
||||||
|
3fa030697dbaffb6809991bb69a335dc4e9accfafd4ed5ee953ed676673912e6
|
||||||
|
4a9ed2aed1fe75a0b02f688e29a20acc2bf7e081ba7518a7d75451be30fdc4e1
|
||||||
|
4bbaae5ae8cc8cc3330a73bb872b16e1420805197fa9bad903f7db9a1f358041
|
||||||
|
4c54a4cda3c5280d5b13facf3089f8a0710c4570876aaeb679025aabf92df99f
|
||||||
|
4e052b4ef5602a68eddcea049bbc4e79b306012cf8b7f0c9cfc3e6c39af64fce
|
||||||
|
4ea780981a54c3e87aa72eb80436e02f3c6b3aaeab96ecf869602f5f5b0b546e
|
||||||
|
4f2ebb779645de5cf9d24e93c6fbd9d95d358b390dec94eed6f0223016785e42
|
||||||
|
4f2266bb0b8a9b902eaa439ae9225963d6fe4359fe30569c2d8afa926b582585
|
||||||
|
004ae946f5997471f4554ab7158a3b982b72391b59c59be40ea79b501eb1dab9
|
||||||
|
5b84763ca63a28ae79c1393d700c534e7dd6e610f890f540981e49972e3854e9
|
||||||
|
5c7dd57f445d87a39eea1e7b299e1a0acbbc946d6bfc92ba165c4af8933b322a
|
||||||
|
5d47a8950cfd56492e0f8b4ed6bfbf470a2e22811206ac0da0b79e520890b41e
|
||||||
|
5def44013bf255d4cd97c3829ac7ae895c607333a3ee586a4090bda128d379fd
|
||||||
|
5ec44fe3223976727132521b3055187e88783716a1f5fd07d1625205acff9888
|
||||||
|
6a84846fd8e476d206eafb8f5a529ecbdf6b1021daab890429077e32ef4c2b6b
|
||||||
|
6aef4d0808bef53a15fc6d19614b641a5e36c941f2729572cfef8cfc8e36bbfd
|
||||||
|
6b5a55f08cc63514e70a93a3eb5d294e9b109e990521bb0ddcea4818a8df1456
|
||||||
|
6b8acdd640ae067c2ee5235ddea8ad4c0e1eec2aaef1277b426923f56303e07d
|
||||||
|
6bcaa587af8bf282f83605713213c20275b1c8178c28de97b154af25bb6a674f
|
||||||
|
6bdf1f89b902e5a7134a87e0269229531ec6c8391256ea1bf466d2d6b66dd8c9
|
||||||
|
6bf7012e27730a047bff7d69ceadef9c06433783d0b9582faefa9c9d0f966c86
|
||||||
|
6ce965b12082d21ed8ade00884de6477faea9f97cf7a85e4b5ab3302d3517fce
|
||||||
|
6ddb2a450b19b940661c677b20936e92e8de54b8bc76f8dbacc56398d7d13a38
|
||||||
|
6e21c6398b13ebac045a11b136fbb2d289e6abe99401eed9108b5464754a4c87
|
||||||
|
6e0521e4314af444375b56e348a5e3d34aceb88d2d5e2f48b6986c27250e395e
|
||||||
|
6f19aa563c4e3e2000ed997157bc7f5bc8a45ee1d77c2cdcd1fd239117a62705
|
||||||
|
7b5530f7f440ff6477f2ee5885b37ff9cfbf933d7908225998875b233b4d2a2a
|
||||||
|
7bfe063b0155297f5bca22c7ed02ae7ef9ce1b968670607415f7720e1e8649f0
|
||||||
|
7ef24d6bf9fb190db2572505b489e45731a326c5c9c4ae6e36ffd7874560bbab
|
||||||
|
7fc41ddc47424ea11d4940f7062cdae69b92af645e6759f8759f47b689a19cb7
|
||||||
|
8b44a3824395c930d0db784e5650ba7b58663090c3bd72a44f9dc716008a9642
|
||||||
|
8bdcbc26d645ae3bc60742debe32650510bcfb7c69ad4416688af5b1815e5645
|
||||||
|
8c2a7cf3f809218d20576368d4d419a63a07a89c3b12cc389fb2f3424fb45b73
|
||||||
|
8c4d45c90f336c0ffd5d619765d8d2a45e802d619548f187bb3f2672b3534969
|
||||||
|
8cae3c834dd9284d1d0069de042c2d50e49a138a78d93a59eb752272e4a5e4f4
|
||||||
|
8d2ab3e77a72ebd708630e22f2eefc0a66f73cd94ed2abbfc4f666c504a3f172
|
||||||
|
008dfd2a2e98c6b6673ee471d72b9e150cfe02859eb3a3bbd31a3c582b5a3a46
|
||||||
|
9a2e6958ba1ea4443ade7619d326c84c50dbc444810b47904ff625969725a091
|
||||||
|
9a86729a66d00d8be95f93062381b9bd91b16f94b654636681553fec6c7c7985
|
||||||
|
9abdd18e6a7b07800067237c93d48448d49d9ad2388a91d35653323c2bed4389
|
||||||
|
9c59fdcefb7ed258e615f8cb688334a69aebdb8e38fd1075a60d6fd98167065a
|
||||||
|
9c033896f0b00023816a49cea882a342c9b62ca4528640969a0d22c21f507505
|
||||||
|
9db05859a329f9544f5a48cbd987f02fd00e48f160dcf6ee8331d8accfb423b4
|
||||||
|
9e9fa98cb2344c5eda9bae7417557f2bc2503de1eed8219fdc2d2d378e1883fe
|
||||||
|
9e06437718bd8f9a27240eff7b425406aae50f061ea47a6991b9830ee2af9f59
|
||||||
|
9ee6f261789b97ea0887a6a905bf5bba61a7070cc91ff64cda0091923d9cdf25
|
||||||
|
09ea67268ce8c0442357ba316f6431cd96143da6bb54008d3c342be8b6ddb169
|
||||||
|
13b744ec13f00bd3e5e2ecaa37bf5385af38cf0ff94fd05e660cfed3339a0c05
|
||||||
|
13d76d014354ffb4e5aabfe00eeed5206b98c502dcfd515545fdc7210316bf61
|
||||||
|
18e51d43a77ac282e108fd8a8e55207282cdabd2219af1e47c2c214117179f51
|
||||||
|
20ee459190ea3442ab7cf0c288182033fb84b75a15bdfc14540e80ab60dec542
|
||||||
|
22ab14c474cdb6cf9f873dcb6c8ce85dfc588abb46e2bd0925cdeff384437c46
|
||||||
|
24a30fa240cec28f9a8b6d02a47ad1fda39674f094ae095b0d6ec3cbf099ed44
|
||||||
|
24da0da42dc2eb9b6d35dcb7cd513b1e5c7d2ee38b6d59d9a1d51f0d01f6f920
|
||||||
|
27a56be2953f040882ecd07fd8b91937f8cb17c25d1ceed56656ed2075518f86
|
||||||
|
31f3b426743b1cff7952948995915586480486ed7fd06eef3379cf54b8fb3d9a
|
||||||
|
38ba44b0ff38f8df925512a477b8b5b4cba216a8ec0bd51b7463a86a88c42fea
|
||||||
|
43d6928bc4bb7a1e097122424e35f22393a0ccf2e03863d7dc3cce44163c8ae8
|
||||||
|
49d0cebfc3199cef395ae27a6ffa479340c1c2f69da9248a777d7bff9b6b9ca9
|
||||||
|
49dbe15539d7170822958bb0699d140c4d493215236b32679340588710d5452b
|
||||||
|
054cfa9b87316799ecd24ecfb480a38aa04b5f85a478a3db95c41706e5b63523
|
||||||
|
57e9dc8d409d939d877a513023c95146400dd10078919ab9402d8144d5f30ca0
|
||||||
|
62e86d5d6aded4a8ba536696c0df0f0af89f2cd7b4b893a7383e3abc00eeadc5
|
||||||
|
63ddb5c60e6cf39dea412903cf1d6c0164a1146bcf4ab98ee94c9b1f9e45c6bb
|
||||||
|
63e2655474549ef03aaf7d9d741502287d2f01170c5ca008d7b09b8e00b1be66
|
||||||
|
67eff4c5c60d493dc2a9f7841f7f138b610dc389c32793a27a62ddc13d417a1c
|
||||||
|
68d18a8d3519b6ea6928bbca40f9db7500b700f66e6eab236e7c32a214de2cc7
|
||||||
|
68f83c9613471fe0e046a1f9252b71e86aa50d36315c675e92dc44a7cdd0267e
|
||||||
|
71cb6156444948894aec343c9a2c0c7674aca84d292d090c7f786d13146709ed
|
||||||
|
72ed18f5962e48dd5fb54e7d91d2fd0479017c27b2c60fc58a5d6b6e55242757
|
||||||
|
74c6b39b7b926dd56db16f175026c2bf9e80a912916f273c122227672f181d03
|
||||||
|
075b39bc7ae8c56a6d99923e4938ddbbb4174df6f0e653abf9c99b420b401a31
|
||||||
|
78c01e39d1997eb54344f02f14dc02c7230cfb14ba58abc1a11feab2e9af85be
|
||||||
|
78e7acb9228c1df4ca4c706d459b97c8e79e8399fe06476ce72bd919a3b2b77f
|
||||||
|
83a8287d1e8700aa7e207b2c617a68557ce6f83f373f2c3c6d1443ae50032826
|
||||||
|
83bc388f3ae110e72ba51a8f0b68f7bbdec3685907c1ff08095b63a1759502af
|
||||||
|
85d6f7309436e59cce9e373b0aa1626fa5e12d8d81163773f5f2c45da9d56543
|
||||||
|
88c6573974e81f1fff96999bf54b80c66bc6f1e9af6746b3cd6801616198e0fd
|
||||||
|
92e2e73e3aef6e5330d2ce8aed3c58e7e3632a6f191f2b03c997ccabd245bb65
|
||||||
|
93c2b9d0a1f54c2e6355d206d93bf2309f81076628aae9d868406c288b0f6bf4
|
||||||
|
98a80f5f7903ef45f92f74a91f34dc01d527f8a2fe98b4fbe76711f5259416a5
|
||||||
|
99f7aaac2646c66dd0eeeebc157b8563b56299c27ab4bf64e13920d089ca7156
|
||||||
|
104e16e4796c52fe212c04b06417161e0bab20fc01632bfcf71421b73b979fbc
|
||||||
|
122fe66bacc9f04b3f3e4f6d0a3b15abdef8321eed36637a1284e3bd145463b8
|
||||||
|
158c590779b033e48b5efcf895fd0bad4c6f277303d72b963cd0802ecac1f440
|
||||||
|
218a186ab13a38d514cf9f523e5798452ae4e5ec76ecf26fe1a46ae22b47a508
|
||||||
|
283e9d53742a13c471e5f59f53a57c179d461f0017ad5f8db27881cdd994cd04
|
||||||
|
305b75817cf5c06105b19dc8c56ff0d796dc6e267ae5bcfc6018aa3d12037744
|
||||||
|
325ec2aa2a6c307bb82d4dcb600fa3ff127aa6d1f35dc683ab61124ef5f0f5b3
|
||||||
|
338f7535ae9ef91754e76604238ca7652273ffce9e6f0a1300de0b1edb61da77
|
||||||
|
343d41a3b38b5a7eac10750eb0213b81780edd3075e9a1fdf3a2c479f4d2474c
|
||||||
|
346eaddf98e91d6d6e7b8c526074c27847e4a18be01b41027bec93abc2b7b2fb
|
||||||
|
355ba69bab94a95e00e87c6c4808587881b9675699083e087917ec8492bebc65
|
||||||
|
426ccc08685def1c9025c94d343de0e59713073984634ee226e69634d6a18fb8
|
||||||
|
552a4b340eb6be03c188cb17628aee0f916b544142465f8e52e7e9d2325206ab
|
||||||
|
556c805068afd3f6784bda2eb59642d7dc03756580e959e1c75262711a30d4d9
|
||||||
|
0642eb9eee222e235de9310b9bd86b42493e5f4d71d430619b5832484dd09cd6
|
||||||
|
645d70ab53013564fd243eae61db209e3793b826a9eeffd77395b18e27458789
|
||||||
|
670e6447721f667df5222cb06a68c2682eb92803184a44faead8aea0dff44df6
|
||||||
|
0717b5f7615c345fe47ab8212c06f506e08585551d93c6a8083b56a216adb71b
|
||||||
|
734e6b63f955d0b508dea39a8637755e75fbb353d54430956a0010f607eb727d
|
||||||
|
748a1560ef6216c8bf80545dbeadd29f99fd5c984b6f34bb678febf8b369adcc
|
||||||
|
748cce6aa136bdc0b3111c73a7c9434508200f8c231933b925d42b37b9cca17c
|
||||||
|
760cd2f9203dc3dc3f933e4a34bbaae857dee0afed66e727846a4c7f750ab5a8
|
||||||
|
762e15e05b2f84f035a7011e10c64558dc6637c9fdde1846dc77f4056f3962d2
|
||||||
|
787dc06078f3718da6faaba82269a7d21b213482c9e45025315bdb6cf59010b5
|
||||||
|
866fc46ac37407bbf74761cecd21fa6322c4e7621f3dfbe963a85671d5753e5c
|
||||||
|
921d92a1c692ea0fd42cbe500bbdb1f3defcd5fb9b51df216451aa260d95ad45
|
||||||
|
932bae2149e79ef113eb4929614a895c642960160d5f5cb2f2e9afac7dfa6af2
|
||||||
|
968a5d7a54fb1d8ef4163f287e128cbd541ab46a2cc54db721dd3eafe1397b65
|
||||||
|
991a5d344112325fb2c40700cd0ee96a784ce2150cba0a7b0be3ed6eb2411878
|
||||||
|
2469a8c88a452ed226dd6d1082f0cbfbe3107d76f919b930b6bc7ce3b839ed90
|
||||||
|
2634a60c3c37d5eba4be4827c08c0e99a951b934604c29d116fcddc27808e480
|
||||||
|
2788da1d8f917a9d5b8acab2c7fe7336b6a7810f1c7edab63ed327a6022ceddf
|
||||||
|
3000c7863b2e80a72329ca1fa0dd03d2fd3994a2391e82450a8c92199cbb56b6
|
||||||
|
3461d48cd9e88caf5363b0e4d96e799f6759ab8b8bc6ba4c148492a1637c01f0
|
||||||
|
4681b8c67ad20fdd3ad04db895084d37a9296ac76b74ea7bc9465bb3ac93df8f
|
||||||
|
4690cfe7c61625190732c028ad4b918930a65e295005171d48114aa8f27b567b
|
||||||
|
5206b60d49110c01e05dc24034e12b767efa739e2ad21c252ab83db47e56d526
|
||||||
|
5234fcace55ee7e44631499ed1f95fd0289918a84bd9f5cf6d9f761d43c34f17
|
||||||
|
07402fbe22b572302f3317958f8cc4c6ca30317e0ee5bbbe9ff51c60b3da5a05
|
||||||
|
7458b4c4b975debbd2bb705e9575d2d1ff5e9308ed9e70c302f5643136cd0cc0
|
||||||
|
8097f97f4ae00b654b1c0b4408ad585f5bfce7999e482f244bd698268e1f6061
|
||||||
|
8158fe1f757d26cadb484ed9690ad4d20d4cb382eeec7c9b0627b365afd6576f
|
||||||
|
9917d608ac0ad05856d31873432d641828ef519825a3f451e6a1ab2519dbf0d1
|
||||||
|
13175cc97740c834e2404c11ba9fa9418d0c390d27a87e6e869977d6a86c20c0
|
||||||
|
36571aa758bc335acc463c3ea074422a5123c309a37e1a7fc818a2069941c493
|
||||||
|
59438a5f6378d218a73d3106ea1984d1c160fe4fcbbb7ec1177067f17430dcb3
|
||||||
|
60496cd3a1e1ad1dcaed30427b70a57b8b14a122bae7f2cd882f8154a4549978
|
||||||
|
61544bee8e2c1cd65e999121bb63011a9c1b4139b20b04fa32e3a850ca8c82b7
|
||||||
|
84046f48a922cd337284bc2448a5d2682ea6a2e3479e71531122aff9ee4cbc93
|
||||||
|
86022b0cd1043facf7b1c933c3b6a6ff36277a76d50bdb2da6a5690d57381b9e
|
||||||
|
98129d711a1e807ab7c3ae4c97187ab8505cebf1012218e8530d0829a2ad436f
|
||||||
|
177451abc7ed0606468250a7793e018725e8c7f370ee8937d4941a11571dcb9d
|
||||||
|
0217869eb1be5181b7053b993443dac73d9cd0804cd5bedb604edeb43f911702
|
||||||
|
219118eb01fa7be03fcce2124b74708cacafe66aea617fd2bafa32f0b2d4fd80
|
||||||
|
394985d0c3db0791b8c7569fccabd36959d00deb285075f9852568c4c81d32eb
|
||||||
|
722380f945884d4dea921ffb958513e02a4802483ec0886712e5ed3fa28d751c
|
||||||
|
774878c58b6a8cae6fbecdf656f2c1f09f37a89190068663e923763dbd6969ce
|
||||||
|
925226a74657c7ed0fcdbcfa9b504a47f7e21d0982c44f7ed9576140fb417146
|
||||||
|
963079a9efb903a575b363b27e15927312569528b9066511b4c282e01e1ec931
|
||||||
|
3830913f0478ec59a7a31d538f858f8ef612e1b2ce3bb9058a2d18f7b43abe9c
|
||||||
|
8579968bcd362a1aa9dcf14e22956b2fd15fd74a2fc2e7369ffb1457c2ce6693
|
||||||
|
8678479fa55ea3871598eb6914eb8428ba599da4c619c62f231e6931f0cb1f54
|
||||||
|
13694424a3895e8e73cc1e06af29aae0c3179c3cb5587b9daadeadcdb29b7cc7
|
||||||
|
67711784cb23efbc0762e83f03365b837ec5216c38f3ba682a82eaff6a7863db
|
||||||
|
92236992ea73ecd9befe3330e88e96a7aedacd9cbca0bb8c32c31d0e2c914e5b
|
||||||
|
134995217d29dc47fb0ca1a274ceb5ea74225d36a98c4bbe6e07bf8c8d6898c7
|
||||||
|
481706314b835ab042583c3107e168932844a162a2adf3b9ef2fa27433a7fd1d
|
||||||
|
4098005900d38c71b152787ce4bdedfb8e2b2b8b061f87645b2b4a14e0004335
|
||||||
|
4699188665aa6eb965b7bb9f4938187f377e7a4ad55fc97e44802ea76f3612e9
|
||||||
|
04762469942ae24c353f929fced777f245bb9f2f888a9f2044ff27fc45933665
|
||||||
|
a6c297b9282b39caca7e513724219af096a3cf6fd453c3736ee7c4c833ef417d
|
||||||
|
a22cb584ba3444bc1dbd90b3723b1ad9424352251f2218a92b1884ef74e63b9f
|
||||||
|
a68fad1bcb1f1030af3811590cc18d74eb1da3c67190af4459fa69118aa8f035
|
||||||
|
a74b9b2333ee936c17edd0525b49fec1337768ccab68f5576a39486c875ff968
|
||||||
|
a74b41238ed9ded9e866679ab00f85813d800f92e4bb2642d3f9de28a1655b81
|
||||||
|
a84ff849188b2bab211fefe8bc61a70f58f48f6a2a6ca25c2645e0bb5c384cbf
|
||||||
|
a2815ec8af59e0a137aa0cdf2baa48ca6d1b86e291c5728fab9449280d15fa75
|
||||||
|
aa5cf2d08c68b8d3dfe3330861935fd97ff8fb9169645f6c632d970ad71b575c
|
||||||
|
aadb8f0be78890f1e08630b7a762a82853dc1402326bd16cd2a2844fa63fef1d
|
||||||
|
ac48d92438e8d19cf546764cc051172af82092223daad8ccfbfb4cb05100f5d6
|
||||||
|
ac78e0a5ac10c7d0239e0e1ba3b604cd6fd418d87a6f32b66f3ce98c674164d8
|
||||||
|
ad79ebffc7d8225cfbee6ad5fec4092ba7376f38dca912d6df8ea932f1eba20b
|
||||||
|
ad7084f002b17a1a27f70c796e3d7424a064b2ef5b71eadb7b3b08dcc9f59042
|
||||||
|
ae2a94eac2960e95f944c9a4a0649c94c405fb0e518adfba2e6fa0f80f2a5af2
|
||||||
|
ae9d5c15cba1adf881044a4449eaf72f5f2ed753c301edc1e43bcfa4560e8d31
|
||||||
|
ae5499adb73000b6f460e63d6e045fe492702bd73187f868d9988690cb72a453
|
||||||
|
b0e341cb8fb7f2fb42f7a8512a9b922591cfa40bd3522f31a422a9fdb4586008
|
||||||
|
b4bc9ff2d9b533e896dfc5082ba865a4ea7676d71259d73aa33d515a5a60dd91
|
||||||
|
b8d2fbe7e6afc931581bf5f1edacaefb9071bcc9f709e294a3f5ead104f73890
|
||||||
|
b8dd8697ad52f98dc44ed3c940752aa52e16d6ab7563d36b1c9e221db22d1e11
|
||||||
|
b21ec6b73a9f1f51623355f2bf8229b5f6f9da1484e1ce105cf8e0363475a5b8
|
||||||
|
b081c3329ed821e22b0ee9cecbc2f52a5e9be70be2b1cb83bb262c21c84640d1
|
||||||
|
b274ece5f6e1d506ca0646077964797c3e7b4413c9a9f1e453d9257e6b4d09f9
|
||||||
|
b676fcf4f5272015b942c8589135a18fa16204c5aec457f1d0898ce6c84638eb
|
||||||
|
b1591e2a2604b6404d0d18e4b2b7599aafbcfb2bbacba06b5de5ff14abf14be3
|
||||||
|
b7000ea518cd327f7dfb42858434fdb80c451aecf0a0325b6ccbff1f30545803
|
||||||
|
beb17a9d330ce361923600aac8e13f8b10e62d0b4067380ad7a95c4d77f0110e
|
||||||
|
bfb91f5666855fd57228801ed0a4294981bd2a6665a93d1093bf3c4a0e89cb2c
|
||||||
|
c0fb7358aabef2c38e6921e2ed06b7fdddddecd29d530814545968323eba964e
|
||||||
|
c1d4fc53203c4b65abb1ff76f9505a46175c57afcd9c95cf445c6a3c443839fb
|
||||||
|
c3fbcb924a928bca859b69c75e4df275eb5e0abf03029785247313b0fa1cdd27
|
||||||
|
c4bccf1c99f4990b0a23848b6fa58977dea0bdbf6ab69b7c0c2b90f7f0a2f5c3
|
||||||
|
c5e875c34f4e4f1fd4edb1c6dbf4862a6543e26387ee480eeba0f5e3f44af7b6
|
||||||
|
c8bb03639a4d7c41c233fa28d5a1c06dee874fe00614f764df9db3df14da86b5
|
||||||
|
c16aa92f949bb89a034c6ef22ea8ece859883ab309086a204e43cedced8dd059
|
||||||
|
c31f2070f92b36708a62c1d53a7aea495fd997440f2c01c851ad83c66357e379
|
||||||
|
c35e0b961694a882dcc4c7bdb8e657d4b0685687cb5c4469559ce5078c95302a
|
||||||
|
c56e78b468dfe3991fc787ab7fe30b4282a73d7fd2e79e4dfc9305bb064685ea
|
||||||
|
c67a888239bc5e11ed44ca31307f3b46f7f76698d706ffc764024110d1a69ceb
|
||||||
|
c424d546895a59e0f88e2296a665c03f4618ac304ef19742b6f1c543665b9bb3
|
||||||
|
c716ebc8ba8fa1556e1dddfaae98c7019618d39ece0e78dbd1b9f58dd0760d94
|
||||||
|
c797d99e68b0efce9b1337dc0e3e9cbf42806bef03021ade362cb4e4d1de4222
|
||||||
|
c5708e0a13ababf73203b1e297b3b73229cd6370e3fec1653b7cf1dc143a0d0a
|
||||||
|
c52417cfa3d923ce7168b7c2b082eb3ab31bc1750ff3f3edfc2c1cd59fb74f3d
|
||||||
|
c53655b36c7abc28db3020511a0ebdffe8fa37d86412753fca86f4caa7ae7a6b
|
||||||
|
c782734d8be184e0721d0a984b6ac3e8e0052206a9e46d22188a7fb2f923face
|
||||||
|
cb71ab036375acb25c69afd915261acf65f919008fbd59417989f6ebef110c7a
|
||||||
|
cb613ceec1bf86b718059ef3e864dce7ed60e5b6a7bd5ab9f7c633350df48f3a
|
||||||
|
cb10732ef556d77ec0a0a5d7b29c0789b47a9757b67bdb3fa9d5be22809debd0
|
||||||
|
cf799a4b43f87d63eba6042a73b9eba3f17b354be1fd2d94dfac2048e13f3fce
|
||||||
|
d3c941cd28b5a692c2532f1381afdc57740d603fbba5eddbe3539f561f26a0ca
|
||||||
|
d5bd7d95fbb1f56d7617d96a5a6deb65417b4448692cb45f4a0bedd824165fc1
|
||||||
|
d6a74fc39944848c79012bb06d915c609b92c1099881b90b4cda59074bd1124f
|
||||||
|
d58b42f643175b6272bb579d354a9510bc34fc88e4e70dda08346997ba3cec13
|
||||||
|
d256b9490d623a136093e459b9f466dedadaa0d00235d6934044e6591ee7e2c5
|
||||||
|
d352bb11678cf6cb47a16740ee9b79cfbf4c19d4da23f4fb95621bbdea6b91b2
|
||||||
|
d053441be51f569ac393c26c220b6369ae3455ae76f3627eba223e8b4e40eacd
|
||||||
|
daa717bf6363a3afe8dbcd3ab863a81ca737e0999242bd99c618daa0e4fa085b
|
||||||
|
db0c37c3ce9df2f2090c622eb4194e6860e26846398d4b4a092b9c4c21441af4
|
||||||
|
db2434b8bc6fb7858aa5eea51b23eb3fe17ed8e8ac025596ebb4f857d00f9ba9
|
||||||
|
db6910df9d6429a8ecffced74f22c26f45a11d48efea51e48e8097bf43331540
|
||||||
|
dbcfeebe1162c457fdf6f7cbfff66c971dfaca8a1a8ae2fb5472fb9007f83ddc
|
||||||
|
dc5c95c9a8f2f01a8eab63c7d591a7d008dbaf5f4b6157380faeeaeab168cd59
|
||||||
|
de81766fb8e806f93c97771ae1a0ff529831bc5e7f50e2187d41dd606b8e23f1
|
||||||
|
e0e25d3ed1797b0092c897ad904a4ac7a2bd77e12b96e1790a939aa009851803
|
||||||
|
e0ee75c206f1b9be41e608c6de05f8fad8532e3bda240cfad0640c673ae46395
|
||||||
|
e1ae949c5339eca43334dd6720b49df19082036b8f53151b8dfac97f2d2c0b6d
|
||||||
|
e2ca0579185aa6d7324f144357323b4d5521c1e94e8dc7929f0ffa9a73e1e4cd
|
||||||
|
e5f9a9a8f7e4b6d097aef1b0493180a547dbf19006920433e7b80aa8aac0026b
|
||||||
|
e6a40ba40f9ac5f43ba2276b3cb6110bb4c0e20003e866a55ca28c954ddd0e9a
|
||||||
|
e9e1b0601616187e8934c7212e2fde5b590b50308aa9c8c5dd12e419c6bebc18
|
||||||
|
e45c71e0680d2856e2945881fd01ad6320868e0a36427c6bca637c53dfecc625
|
||||||
|
e62af03dd8d599dfbdbecb8c6516d95e582ee803f2ddd3798479fbc769229030
|
||||||
|
e72bfb25201770edb1889f678f5a844a57c9f5308acd57a86c5f4936b800d441
|
||||||
|
e311fa91eca24a97e0c558e91d06dca353aa9348a7ea4b111e694e181868fc27
|
||||||
|
e339a495e0c048e30483e4fd3bef66186712975f3c33c90ef63a3c9686da3160
|
||||||
|
e668b1c1f44dae28ca34f530e313c1e544751b63bf0c480300be8e242683fecb
|
||||||
|
e4638cbb5b12548a76a2cb6e87c7f2ada75d61d3c7d1062535ef0bf2b3cde05d
|
||||||
|
e7945baeafe0ac6d9a1e6761786c35fddad76c81d86e544d369188c186a0389b
|
||||||
|
ea35c704202ee36415d4f7a0873bcb254d30268450d7c9ed2dc611c1e2f4d5d1
|
||||||
|
eaeb108b23d455d01ec0c9e44d39ff395522942794d26f3c149e29615be9bf1f
|
||||||
|
eb420ae07f9c459abc101c1264fa8cf779002ef20748ef97abe7c3764c24f032
|
||||||
|
ec30ca07d0b0af16afe60f217dedf15573045ed9adc45e21c3592ee6716fbe39
|
||||||
|
ed02b9c56e48d34910bc97ebff834f1cb082c23ebcae97b0f51f3a2633664466
|
||||||
|
edcc77bb3d279e3dfe7d687fe30e22f95f5d9391b9908e61b2206a6cf309ea91
|
||||||
|
eea963e8499939d36d09e42d8d5cd86798e81e264c4d4c012191e42926e508b8
|
||||||
|
f1b01ee5efb2117033ddcad5f5eafe8e6f65ad6fc3e933902bfc494c13a7d3c0
|
||||||
|
f01b151ee48f9a3a631457069ece3a2ac3e111df28110f3c06f221ee7c54eff8
|
||||||
|
f2ec30f9f836ed26cb13fdb80064d1dcabf953d1952fea2d85bb6410f4cb2271
|
||||||
|
f8a87cab6bc0890c7a2e36f73e533eac1a1cabb13b886b8ed71afa4c8b7d90be
|
||||||
|
f9b0fe025aedb49e94d80119a8a8dc0f85800bc592e568dc95b0f1e5439c54fc
|
||||||
|
f56a1ab986e83e118ff174e9805f6151e47ff693c17a2f475fcedd83cb98e051
|
||||||
|
f82a956615af08264b64e051e09c1441152d92371d4d2db8d5c0ac4ab47a7182
|
||||||
|
f287b03a768126d72af210dceab12e329573779e7aad56c123ac0dbb14fc4f4e
|
||||||
|
f347e779fb8c09daba2a757dfc3ee7c415f8f5a2eaace19535d363cc3ab3171a
|
||||||
|
f547c232f5e1785aa11200ac03dcfda6f056400bf1456a3388db6d73a8158ca0
|
||||||
|
f6822a239614fc82462535db5b2806dd616696c9552e1d469922f4b7cd2bf56e
|
||||||
|
fa453f6fb451a564d31e704bbca0bcbcc9ad1109a25a8e5ad5c704789a424c4b
|
||||||
|
fb5ad60698efb6ee9f45fd440e56c060ba01ba03a854f556e42f8090b2057c7e
|
||||||
|
fbc407de7b12a4975b60dac518ac287f076659213cce708871162342419ab383
|
||||||
|
fbf633eb758b7b66ab761c34eeebc759791e2f73020e839b3cfc2d692c9ba21f
|
||||||
|
fc1bddcbef8c02474db71bf11dace70e1e51d94f3c3fe6d9d45f4a669ac4a9fd
|
||||||
|
fd453dd47dc82ff7625bd4093c62140fe14a348814f094b43b519f7f59293af7
|
||||||
|
fdc05b3bfda3101bec7d66b5e0cfb589e06f1f4b4d6b2f9d84dac470b0abf53b
|
||||||
|
fdebde6f8bcdd5281c50bcb4669c45dc54e33563bec872f8d23d76d2fe3ebfdf
|
||||||
|
fe503d3e9a34b113d4c56b59c5986e13c44b6e401b6e6681eb62712d24368329
|
||||||
|
felhom-controller-data
|
||||||
|
filebrowser_filebrowser_data
|
||||||
|
immich_immich_ml_cache
|
||||||
|
immich_immich_postgres_data
|
||||||
|
immich_immich_redis_data
|
||||||
|
paperless-ngx_paperless_data
|
||||||
|
paperless-ngx_paperless_postgres_data
|
||||||
|
paperless-ngx_paperless_redis_data
|
||||||
|
privatebin_privatebin_data
|
||||||
|
ls: cannot access '/mnt/felhom-drives/scratch_hdd/felhom/backups/primary/': No such file or directory
|
||||||
|
Filesystem Size Used Avail Use% Mounted on
|
||||||
|
/dev/loop1 69G 11G 54G 17% /var/lib/felhom
|
||||||
|
/dev/nvme0n1 938G 210G 681G 24% /mnt/felhom-drives/scratch_hdd
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
10:10:26Z == 1 deploy
|
||||||
|
10:10:27Z deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||||
|
10:11:12Z state: running
|
||||||
|
10:11:42Z == 2 marker A + backup
|
||||||
|
10:11:46Z CREATE TABLE
|
||||||
|
INSERT 0 1
|
||||||
|
10:11:49Z [2] DB markers: 'A'
|
||||||
|
10:11:49Z POST /api/backup/run -> 200 {'ok': True, 'message': 'Mentés elindítva'}
|
||||||
|
10:12:59Z backup finished: {"db_dump": {"count": 3, "duration": "1m9.055678483s", "last_run": "2026-10-20T10:12:58.690358712Z", "success": true}, "enabled": true, "running": false}
|
||||||
|
10:13:04Z unit on disk:
|
||||||
|
UNIT /mnt/sys_drive/felhom-data/backups/primary/rallly
|
||||||
|
2026-10-20 10:12 487 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/app.yaml
|
||||||
|
2026-10-20 10:12 4071 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/docker-compose.yml
|
||||||
|
2026-10-20 10:12 11894 /mnt/sys_drive/felhom-data/backups/primary/rallly/compose/.felhom.yml
|
||||||
|
2026-10-20 10:12 52228608 /mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar
|
||||||
|
2026-10-20 10:11 111371 /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql
|
||||||
|
2026-10-20 10:12 3195 /mnt/sys_drive/felhom-data/backups/primary/rallly/manifest.json
|
||||||
|
2026-10-20 10:12 1191 /mnt/sys_drive/felhom-data/backups/primary/rallly/data-stamps.json
|
||||||
|
10:13:04Z == 3 POSITIVE CONTROL: marker B (live) then a normal unit restore
|
||||||
|
10:13:09Z INSERT 0 1
|
||||||
|
10:13:12Z [3-before] DB markers: 'A,B'
|
||||||
|
10:13:12Z [3] restore points offered: [{"time": "2026-10-20T10:11:50Z", "short_id": "helyi", "tier": 1, "drive_label": "Belső SSD (rendszer)"}]
|
||||||
|
10:13:12Z [3] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started']
|
||||||
|
10:13:12Z [3] restore-status: {"running": true, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:13:12.817369857Z", "last": {"op": "offbox-reconstitute", "stack": "rallly", "ok": true, "message": "A(z) rallly: 0 fájl és 1 adatkötet és az adatbázis visszaállítva (mentés: 2026. 10. 20. 09:30) — az alkalmazás újraindult.", "finished_at": "2026-10-20T07:36:57.852528534Z"}}
|
||||||
|
10:13:40Z [3] restore-status: {"running": false, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:13:12.817369857Z", "last": {"op": "restore", "stack": "rallly", "ok": true, "message": "A(z) rallly: 1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-20T10:13:39.161573115Z"}, "last_recent": true}
|
||||||
|
10:14:00Z [3] after: state=running hold=None
|
||||||
|
10:14:18Z [3-after] DB markers: 'A'
|
||||||
|
10:14:18Z == 4 marker C (the newest live row), then THE FAULT: the unit's DB volume tar truncated to half
|
||||||
|
10:14:22Z INSERT 0 1
|
||||||
|
10:14:25Z [4-live] DB markers: 'A,C'
|
||||||
|
10:14:29Z TRUNCATED /mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar 52228608 -> 26114304
|
||||||
|
10:14:29Z == 5 the unit restore over the damaged copy
|
||||||
|
10:14:29Z [5] restore points offered: [{"time": "2026-10-20T10:11:50Z", "short_id": "helyi", "tier": 1, "drive_label": "Belső SSD (rendszer)"}]
|
||||||
|
10:14:29Z [5] POST /backup/restore -> HTTP/2 302 ['location: /backups/restore?flash=flash.restore.started']
|
||||||
|
10:14:29Z [5] restore-status: {"running": true, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:14:29.288204925Z", "last": {"op": "restore", "stack": "rallly", "ok": true, "message": "A(z) rallly: 1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult.", "finished_at": "2026-10-20T10:13:39.161573115Z"}, "last_recent": true}
|
||||||
|
10:14:44Z [5] restore-status: {"running": false, "op": "restore", "stack": "rallly", "started_at": "2026-10-20T10:14:29.288204925Z", "last": {"op": "restore", "stack": "rallly", "ok": false, "message": "Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1\nstderr: ...twork rallly_rallly-internal Creating \n Network rallly_rallly-internal Created \n Network rallly_rallly-internal Created \n Container rallly-postgres Creating \n Container rallly-postgres Created \n Container rallly Creating \n Container rallly Created \n Container rallly-postgres Starting \n Container rallly-p
|
||||||
|
10:15:04Z [5] after: state=degraded hold=None
|
||||||
|
10:15:38Z [5-after] DB markers: 'Error response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running\nError response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running\nError response from daemon: Container 6012fe2936d0bbea71dd1226f64d872b9d017ae5f5062a2ea234d84f1522029c is restarting, wait until the container is running'
|
||||||
|
10:15:41Z containers:
|
||||||
|
rallly Created
|
||||||
|
rallly-postgres Restarting (1) 5 seconds ago
|
||||||
|
10:15:44Z db container log tail:
|
||||||
|
Data page checksums are enabled.
|
||||||
|
|
||||||
|
initdb: error: directory "/var/lib/postgresql/18/docker" exists but is not empty
|
||||||
|
initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/18/docker" or run initdb with an argument other than "/var/lib/postgresql/18/docker".
|
||||||
|
The files belonging to this database system will be owned by user "postgres".
|
||||||
|
This user must also own the server process.
|
||||||
|
|
||||||
|
The database cluster will be initialized with locale "en_US.utf8".
|
||||||
|
The default database encoding has accordingly been set to "UTF8".
|
||||||
|
The default text search configuration will be set to "english".
|
||||||
|
|
||||||
|
Data page checksums are enabled.
|
||||||
|
|
||||||
|
initdb: error: directory "/var/lib/postgresql/18/docker" exists but is not empty
|
||||||
|
initdb: hint: If you want to create a new database system, either remove or empty the directory "/var/lib/postgresql/18/docker" or run initdb with an argument other than "/var/lib/postgresql/18/docker".
|
||||||
|
10:15:47Z any undo / safety copy of the live state on disk?
|
||||||
|
/mnt/sizing-bench/backups/secondary/paperless-ngx/recovery-unit/db-dumps/pre-restore-20260927T094411Z-paperless-ngx-postgres.sql
|
||||||
|
/mnt/sys_drive/felhom-data/backups/primary/rallly
|
||||||
|
/mnt/sys_drive/felhom-data/backups/primary/rallly/volume-dumps/rallly_rallly_postgres_data.tar
|
||||||
|
/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql
|
||||||
|
10:15:51Z controller log (restore lines):
|
||||||
|
Container rallly Created
|
||||||
|
Container rallly-postgres Starting
|
||||||
|
Container rallly-postgres Started
|
||||||
|
Container rallly-postgres Waiting
|
||||||
|
Container rallly-postgres Error dependency rallly-postgres failed to start
|
||||||
|
dependency failed to start: container rallly-postgres is unhealthy
|
||||||
|
2026/10/20 10:14:42 manager.go:1362: [ERROR] [stacks] Stack rallly start failed after 1.1s: exit code 1
|
||||||
|
stderr: ...twork rallly_rallly-internal Creating
|
||||||
|
Network rallly_rallly-internal Created
|
||||||
|
Network rallly_rallly-internal Created
|
||||||
|
Container rallly-postgres Creating
|
||||||
|
Container rallly-postgres Created
|
||||||
|
Container rallly Creating
|
||||||
|
Container rallly Created
|
||||||
|
Container rallly-postgres Starting
|
||||||
|
Container rallly-postgres Started
|
||||||
|
Container rallly-postgres Waiting
|
||||||
|
Container rallly-postgres Error dependency rallly-postgres failed to start
|
||||||
|
dependency failed to start: container rallly-postgres is unhealthy
|
||||||
|
2026/10/20 10:14:42 handlers.go:1881: [ERROR] [web] Restore failed (async): stack=rallly: starting rallly after restore from unit: starting stack rallly: exit code 1
|
||||||
|
stderr: ...twork rallly_rallly-internal Creating
|
||||||
|
Network rallly_rallly-internal Created
|
||||||
|
Network rallly_rallly-internal Created
|
||||||
|
Container rallly-postgres Creating
|
||||||
|
Container rallly-postgres Created
|
||||||
|
Container rallly Creating
|
||||||
|
Container rallly Created
|
||||||
|
Container rallly-postgres Starting
|
||||||
|
Container rallly-postgres Started
|
||||||
|
Container rallly-postgres Waiting
|
||||||
|
Container rallly-postgres Error dependency rallly-postgres failed to start
|
||||||
|
dependency failed to start: container rallly-postgres is unhealthy
|
||||||
|
2026/10/20 10:14:49 manager.go:920: [DEBUG] [stacks] restart-policy of down member "rallly" = "unless-stopped"
|
||||||
|
2026/10/20 10:14:59 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||||
|
2026/10/20 10:15:09 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||||
|
2026/10/20 10:15:09 main.go:4054: [INFO] [deadapp] rallly crossed the crash_loop threshold (8 in 10m0s) while a backup/quiesce/update holds it — NOT stopped
|
||||||
|
2026/10/20 10:15:19 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||||
|
2026/10/20 10:15:29 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||||
|
2026/10/20 10:15:39 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||||
|
2026/10/20 10:15:49 healthprobe.go:105: [DEBUG] [stacks] RunHealthProbes: rallly has a health check but no container to probe — candidates: [rallly(stopped) rallly-postgres(restarting)]
|
||||||
|
exit 0
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
2026/10/20 10:14:29 handlers.go:1873: [WARN] [web] Restore requested (async): stack=rallly, snapshot=helyi from 172.18.0.3:36322
|
||||||
|
2026/10/20 10:14:29 restore_dbcreds.go:114: [INFO] [backup] Restore rallly: database password(s) [DB_PASSWORD] come from the backup, [] stay as installed (older than the data) — values not logged (R-1106)
|
||||||
|
2026/10/20 10:14:29 restore_unit.go:393: [INFO] [backup] Restoring rallly from recovery unit /mnt/sys_drive/felhom-data/backups/primary/rallly: images=2, secrets recovered=2/2, data_keys=0
|
||||||
|
2026/10/20 10:14:29 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||||
|
2026/10/20 10:14:40 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 10.8s)
|
||||||
|
2026/10/20 10:14:40 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly
|
||||||
|
2026/10/20 10:14:40 restore.go:246: [ERROR] [backup] Failed to populate volume rallly_rallly_postgres_data: tar: short read — exit status 1
|
||||||
|
2026/10/20 10:14:40 restore_unit.go:448: [ERROR] [backup] volume restore for rallly: failed to restore 1 volume(s): [rallly_rallly_postgres_data]
|
||||||
|
2026/10/20 10:14:40 [INFO] [stacks] SaveAppConfig: saved config for rallly
|
||||||
|
2026/10/20 10:14:40 deploy.go:883: [INFO] [stacks] Redeploying rallly from recovery unit with 4 env vars
|
||||||
|
2026/10/20 10:14:40 pin.go:93: [INFO] [stacks] pin rallly: rallly=lukevella/rallly:4.15.3, rallly-postgres=postgres:18-alpine
|
||||||
|
2026/10/20 10:14:40 [INFO] [stacks] SaveAppConfig: saved config for rallly
|
||||||
|
2026/10/20 10:14:40 restore_unit.go:467: [ERROR] [backup] rallly: NOT replaying the database copy — the volume restore failed, so the database is not known to be the copy's own
|
||||||
|
2026/10/20 10:14:40 manager.go:1351: [INFO] [stacks] Starting stack: rallly
|
||||||
|
2026/10/20 10:14:42 manager.go:1704: [ERROR] [stacks] Command failed: docker compose up -d (in /opt/docker/stacks/rallly) — exit code 1 (took 1.1s)
|
||||||
|
2026/10/20 10:14:42 manager.go:1712: [ERROR] [stacks] stderr (last part): ...twork rallly_rallly-internal Creating
|
||||||
|
2026/10/20 10:14:42 manager.go:1362: [ERROR] [stacks] Stack rallly start failed after 1.1s: exit code 1
|
||||||
|
2026/10/20 10:14:42 handlers.go:1881: [ERROR] [web] Restore failed (async): stack=rallly: starting rallly after restore from unit: starting stack rallly: exit code 1
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
200
|
||||||
|
{
|
||||||
|
"ok": true,
|
||||||
|
"message": "Stack rallly stop completed"
|
||||||
|
}
|
||||||
|
200
|
||||||
|
{
|
||||||
|
"ok": true,
|
||||||
|
"data": {
|
||||||
|
"removed": "rallly",
|
||||||
|
"volumes_removed": [
|
||||||
|
"rallly_rallly_postgres_data"
|
||||||
|
],
|
||||||
|
"hdd_paths_removed": [],
|
||||||
|
"hdd_paths_preserved": [],
|
||||||
|
"hdd_note": "Az alkalmazás nem tárolt saját adatot külső meghajtón, így ott nem volt mit törölni.",
|
||||||
|
"userdata_kept": [],
|
||||||
|
"backup_paths_removed": [
|
||||||
|
"/mnt/sys_drive/felhom-data/backups/primary/rallly (26M)"
|
||||||
|
],
|
||||||
|
"verified": true
|
||||||
|
},
|
||||||
|
"message": "Stack rallly removed"
|
||||||
|
}
|
||||||
|
0
|
||||||
|
0
|
||||||
|
ls: cannot access '/mnt/sys_drive/felhom-data/backups/primary/rallly': No such file or directory
|
||||||
|
/opt/docker/stacks/rallly
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# C3 — R1/D4 proven live: a failed LOCAL restore leaves a broken app and no copy of what it replaced
|
||||||
|
|
||||||
|
Scratch guest 9202 on demo-hp, controller **0.331.0** (the local-restore code is unchanged since v0.328.0 —
|
||||||
|
`git log` on `restore_unit.go`/`restore.go`; 0.336.0 runs the same lines). Hub off on 9202. The dashboard password was
|
||||||
|
reset with the controller's own escape hatch (`--print-reset-code` → `/claim`), held in `sec` only.
|
||||||
|
Architecture read: `07` §6.3 (the ladder „replay → rollback → hold"), §8; `09` §3 decision 154 (R-638 A), D20 (R-893 A).
|
||||||
|
|
||||||
|
| step (UTC) | act | observed |
|
||||||
|
|---|---|---|
|
||||||
|
| 10:10 | Rallly 4.15.3 + postgres:18 deployed through `POST /api/stacks/rallly/deploy` | running |
|
||||||
|
| 10:11 | marker A (psql in the app's own DB container); „Mentés most" (`POST /api/backup/run`) | unit written 10:12 (`rallly_rallly_postgres_data.tar` 52 MB + `rallly-postgres.sql`) |
|
||||||
|
| 10:13 | **positive control:** marker B, then the household's restore (`POST /backup/restore`, point „helyi") | „1 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult"; markers **A** (B rewound, as designed) |
|
||||||
|
| 10:14:22 | marker C — the household's newest row | live: **A,C** |
|
||||||
|
| 10:14:29 | **fault:** the unit's DB volume tar truncated to half (a torn / damaged copy) | 52 228 608 → 26 114 304 B |
|
||||||
|
| 10:14:29 | the same restore | stop 10.8 s → `volume rm` → `tar: short read` → `[ERROR] volume restore … failed` → definition rewritten → „NOT replaying the database copy" → start → **postgres crash-loops** (`initdb: directory … exists but is not empty`) |
|
||||||
|
| 10:14:44 | what the household reads | „Visszaállítás sikertelen: starting rallly after restore from unit: starting stack rallly: exit code 1 stderr: ...twork rallly_rallly-internal Creating …" — **raw English Docker output** |
|
||||||
|
| 10:15:47 | any copy of the live state (marker C)? | **none** — no undo dump, no live-volume copy for Rallly (`find` of `*restore-undo*`/`pre-restore-*` finds only a September paperless file) |
|
||||||
|
| 10:15 | the dead-app guard | `rallly crossed the crash_loop threshold … while a backup/quiesce/update holds it — NOT stopped` |
|
||||||
|
|
||||||
|
**So:** the household's live data (marker C) was destroyed by the restore itself, the app does not start, and nothing
|
||||||
|
on the box can put the pre-restore state back. The off-site restore got exactly this protection in v0.336.0 (D20 = A:
|
||||||
|
„put back exactly as it was"); the LOCAL path — 07 §3's main customer restore — did not. Logs: `01-run.log`,
|
||||||
|
`02-controller-log-fault-window.txt`. Teardown through the product (`03-teardown.txt`): Rallly removed with its data
|
||||||
|
and backups; no container, no volume, no unit (the `/opt/docker/stacks/rallly` folder is the catalog template every
|
||||||
|
app has).
|
||||||
@@ -0,0 +1,153 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""r1_proof.py — C3 proof of candidate R1/D4 on scratch guest 9202 (data-loss hunt 2026-10-20).
|
||||||
|
|
||||||
|
Claim under test: a LOCAL (Tier-1 unit) restore that fails after it has started replacing the app's volumes
|
||||||
|
restarts the app on whatever is left, and keeps NO copy of the live state it replaced — so the household's newer
|
||||||
|
data is gone and nothing can put it back (unlike the off-site path, which got „put back exactly" in v0.336.0).
|
||||||
|
|
||||||
|
Method (mechanism proof — markers are written with psql inside the app's own database container, as the
|
||||||
|
2026-10-19 R-893 measurement did; the household-facing acts are the dashboard's own endpoints):
|
||||||
|
1 deploy Rallly (throwaway) through POST /api/stacks/rallly/deploy
|
||||||
|
2 marker A; „Mentés most" = POST /api/backup/run (whole box, scratch); wait
|
||||||
|
3 POSITIVE CONTROL: marker B (live), unit restore (POST /backup/restore) -> A present, B gone (expected rewind)
|
||||||
|
4 marker C (the household's newest live row); THE FAULT: truncate the unit's DB volume tar to half
|
||||||
|
5 unit restore again -> observe: the answer, the app's state, what the DB holds, any undo/safety copy on disk
|
||||||
|
6 teardown: remove Rallly with its data and backups through the product; verify nothing left
|
||||||
|
Env: CTL_BASE, CTL_DOMAIN, CTL_PW_ENV (see tools/ctl.py). Prints a timestamped log (no secrets).
|
||||||
|
"""
|
||||||
|
import json, os, re, subprocess, sys, time
|
||||||
|
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/rehearsal-2026-10-20/tools")
|
||||||
|
import ctl # noqa: E402
|
||||||
|
|
||||||
|
APP = "rallly"
|
||||||
|
DOMAIN = os.environ["CTL_DOMAIN"]
|
||||||
|
|
||||||
|
|
||||||
|
def say(*a):
|
||||||
|
print(time.strftime("%H:%M:%S", time.gmtime()) + "Z " + " ".join(str(x) for x in a), flush=True)
|
||||||
|
|
||||||
|
|
||||||
|
def guest(script, timeout=600):
|
||||||
|
r = subprocess.run(["ssh", "demo-hp", "export LC_ALL=C; cat > /tmp/r1p.sh; pct push 9202 /tmp/r1p.sh /tmp/r1p.sh;"
|
||||||
|
" pct exec 9202 -- bash /tmp/r1p.sh; pct exec 9202 -- rm -f /tmp/r1p.sh; rm -f /tmp/r1p.sh"],
|
||||||
|
input=script, capture_output=True, text=True, timeout=timeout)
|
||||||
|
return (r.stdout + r.stderr).strip()
|
||||||
|
|
||||||
|
|
||||||
|
def psql(sql):
|
||||||
|
return guest(f"""C=$(docker ps --format '{{{{.Names}}}}' | grep -E '^rallly.*(postgres|db)' | head -1)
|
||||||
|
U=$(docker exec $C printenv POSTGRES_USER); D=$(docker exec $C printenv POSTGRES_DB)
|
||||||
|
docker exec -i $C psql -v ON_ERROR_STOP=1 -U "$U" -d "$D" -tA <<'SQL'
|
||||||
|
{sql}
|
||||||
|
SQL""")
|
||||||
|
|
||||||
|
|
||||||
|
def stack():
|
||||||
|
code, d = ctl.call("GET", f"/api/stacks/{APP}")
|
||||||
|
return (d.get("data") or {}) if isinstance(d, dict) else {}
|
||||||
|
|
||||||
|
|
||||||
|
def wait_backup(t_cap=1200):
|
||||||
|
t0 = time.time()
|
||||||
|
time.sleep(5)
|
||||||
|
while time.time() - t0 < t_cap:
|
||||||
|
code, d = ctl.call("GET", "/api/backup/status")
|
||||||
|
dd = (d.get("data") or {}) if isinstance(d, dict) else {}
|
||||||
|
if not dd.get("running"):
|
||||||
|
return dd
|
||||||
|
time.sleep(5)
|
||||||
|
return {"timeout": True}
|
||||||
|
|
||||||
|
|
||||||
|
def restore(tag):
|
||||||
|
code, d = ctl.call("GET", f"/api/backup/snapshots?stack={APP}")
|
||||||
|
data = d.get("data") if isinstance(d, dict) else None
|
||||||
|
snaps = data if isinstance(data, list) else next((data[k] for k in ("snapshots", "items", "restore_points")
|
||||||
|
if isinstance(data, dict) and isinstance(data.get(k), list)), [])
|
||||||
|
say(f"[{tag}] restore points offered: {json.dumps(snaps, ensure_ascii=False)[:400]}")
|
||||||
|
sid = snaps[0].get("id") or snaps[0].get("snapshot_id") or snaps[0].get("short_id")
|
||||||
|
out = subprocess.run(["curl", "-sk", "-D", "-", "-o", "/dev/null", "-H", ctl.HOST, "-H", "Cookie: " + ctl._SESS,
|
||||||
|
"-X", "POST", "--data-urlencode", "_csrf=" + ctl._CSRF, "--data-urlencode", "stack_name=" + APP,
|
||||||
|
"--data-urlencode", "snapshot_id=" + sid, ctl.BASE + "/backup/restore"],
|
||||||
|
capture_output=True, text=True).stdout
|
||||||
|
say(f"[{tag}] POST /backup/restore -> {out.splitlines()[0] if out else '?'} "
|
||||||
|
f"{[l for l in out.splitlines() if l.lower().startswith('location')][:1]}")
|
||||||
|
t0, last = time.time(), None
|
||||||
|
while time.time() - t0 < 900:
|
||||||
|
code, d = ctl.call("GET", "/api/backup/restore-status")
|
||||||
|
dd = (d.get("data") or {}) if isinstance(d, dict) else {}
|
||||||
|
cur = json.dumps(dd, ensure_ascii=False)[:600]
|
||||||
|
if cur != last:
|
||||||
|
say(f"[{tag}] restore-status: {cur}")
|
||||||
|
last = cur
|
||||||
|
if not dd.get("running") and time.time() - t0 > 8:
|
||||||
|
break
|
||||||
|
time.sleep(3)
|
||||||
|
time.sleep(20)
|
||||||
|
st = stack()
|
||||||
|
say(f"[{tag}] after: state={st.get('state')} hold={st.get('hold_reason')!r}")
|
||||||
|
|
||||||
|
|
||||||
|
def markers(tag):
|
||||||
|
say(f"[{tag}] DB markers: {psql('SELECT string_agg(m, \',\' ORDER BY m) FROM r1_marker;')!r}")
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
say("== 1 deploy")
|
||||||
|
if not stack().get("deployed"):
|
||||||
|
code, d = ctl.call("GET", f"/api/stacks/{APP}/deploy-fields")
|
||||||
|
fields = (((d.get("data") or {}).get("metadata") or {}).get("deploy_fields")) or []
|
||||||
|
vals = {"DOMAIN": DOMAIN, "SUBDOMAIN": "r1poll"}
|
||||||
|
for f in fields:
|
||||||
|
if f.get("env_var") in vals:
|
||||||
|
continue
|
||||||
|
if f.get("type") == "password":
|
||||||
|
import secrets
|
||||||
|
vals[f["env_var"]] = "R1-" + secrets.token_hex(10)
|
||||||
|
elif f.get("required") and f.get("default"):
|
||||||
|
vals[f["env_var"]] = f["default"]
|
||||||
|
code, d = ctl.call("POST", f"/api/stacks/{APP}/deploy", {"values": vals})
|
||||||
|
say("deploy ->", code, str(d)[:160])
|
||||||
|
for _ in range(90):
|
||||||
|
time.sleep(5)
|
||||||
|
st = stack()
|
||||||
|
if st.get("deployed") and (st.get("app_config") or {}).get("pinned_images") and st.get("state") in ("running", "unhealthy", "degraded"):
|
||||||
|
break
|
||||||
|
say("state:", stack().get("state"))
|
||||||
|
time.sleep(30)
|
||||||
|
say("== 2 marker A + backup")
|
||||||
|
say(psql("CREATE TABLE IF NOT EXISTS r1_marker(m text primary key); INSERT INTO r1_marker VALUES ('A') ON CONFLICT DO NOTHING;"))
|
||||||
|
markers("2")
|
||||||
|
code, d = ctl.call("POST", "/api/backup/run")
|
||||||
|
say("POST /api/backup/run ->", code, str(d)[:200])
|
||||||
|
say("backup finished:", json.dumps(wait_backup(), ensure_ascii=False)[:500])
|
||||||
|
unit = guest(f"for d in /mnt/felhom-drives/*/backups/primary/{APP} /mnt/sys_drive/felhom-data/backups/primary/{APP}; do "
|
||||||
|
f"[ -d $d ] && echo UNIT $d && find $d -maxdepth 3 -type f -printf '%TY-%Tm-%Td %TH:%TM %s %p\\n'; done")
|
||||||
|
say("unit on disk:\n" + unit)
|
||||||
|
say("== 3 POSITIVE CONTROL: marker B (live) then a normal unit restore")
|
||||||
|
say(psql("INSERT INTO r1_marker VALUES ('B');"))
|
||||||
|
markers("3-before")
|
||||||
|
restore("3")
|
||||||
|
time.sleep(15)
|
||||||
|
markers("3-after")
|
||||||
|
say("== 4 marker C (the newest live row), then THE FAULT: the unit's DB volume tar truncated to half")
|
||||||
|
say(psql("INSERT INTO r1_marker VALUES ('C');"))
|
||||||
|
markers("4-live")
|
||||||
|
say(guest(f"for d in /mnt/felhom-drives/*/backups/primary/{APP} /mnt/sys_drive/felhom-data/backups/primary/{APP}; do "
|
||||||
|
f"[ -d $d ] || continue; f=$(find $d -name '*.tar' -path '*volume*' | grep -iE 'postgres|db' | head -1); "
|
||||||
|
f"[ -n \"$f\" ] || f=$(find $d -name '*.tar' | head -1); s=$(stat -c %s \"$f\"); "
|
||||||
|
f"truncate -s $((s/2)) \"$f\"; echo \"TRUNCATED $f $s -> $(stat -c %s \"$f\")\"; done"))
|
||||||
|
say("== 5 the unit restore over the damaged copy")
|
||||||
|
restore("5")
|
||||||
|
time.sleep(30)
|
||||||
|
markers("5-after")
|
||||||
|
say("containers:\n" + guest("docker ps -a --format '{{.Names}} {{.Status}}' | grep -i rallly"))
|
||||||
|
say("db container log tail:\n" + guest("C=$(docker ps -a --format '{{.Names}}' | grep -E '^rallly.*(postgres|db)' | head -1); docker logs --tail 15 $C 2>&1"))
|
||||||
|
say("any undo / safety copy of the live state on disk?\n" + guest(
|
||||||
|
"find / -xdev \\( -path /proc -o -path /sys \\) -prune -o \\( -iname '*restore-undo*' -o -iname 'pre-restore-*' \\) -print 2>/dev/null | head; "
|
||||||
|
"for d in /mnt/felhom-drives/*/backups /mnt/sys_drive/felhom-data/backups; do find $d -maxdepth 4 -iname '*rallly*' -newer /etc/hostname 2>/dev/null | head; done"))
|
||||||
|
say("controller log (restore lines):\n" + guest("docker logs --since 30m felhom-controller 2>&1 | grep -iE 'restore|rallly' | grep -vi 'GET /api' | tail -40"))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
10:48:52Z == 1 deploy glance
|
||||||
|
10:48:52Z deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||||
|
10:49:12Z state: running family_gate in app_config: True
|
||||||
|
10:49:34Z [1 POSITIVE CONTROL after install] gate file: /opt/docker/stacks/traefik/dynamic/family-gate-glance.yml
|
||||||
|
10:49:34Z [1 POSITIVE CONTROL after install] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 401
|
||||||
|
10:49:34Z == 2 export to the data drive
|
||||||
|
10:49:34Z export start -> 200 {'ok': True}
|
||||||
|
10:49:34Z /api/export/status {"display_name": "Glance", "done": false, "error": "", "job_type": "export", "ok": true, "output_path": "", "output_size": "", "running": true, "stack_name": "glance", "steps": [{"label": "Előkészítés", "status": "running"}, {"label": "Konfiguráció mentése", "status": "pending"}, {"label": "Adatbázis mentése", "status": "pending"}, {"label": "Felhasználói adatok", "status": "pending"}, {"label": "
|
||||||
|
10:49:37Z /api/export/status {"display_name": "Glance", "done": true, "error": "", "job_type": "export", "ok": true, "output_path": "/mnt/felhom-drives/scratch_hdd/exports/glance_20261020-104935.fab", "output_size": "5.6 KB", "running": false, "stack_name": "glance", "steps": [{"label": "Előkészítés", "status": "done"}, {"label": "Konfiguráció mentése", "status": "done"}, {"label": "Adatbázis mentése", "status": "done"}, {"la
|
||||||
|
10:49:37Z bundles for glance: []
|
||||||
|
10:49:37Z == 3 remove glance (keep nothing)
|
||||||
|
10:50:09Z remove -> 200 {'ok': True, 'data': {'removed': 'glance', 'volumes_removed': ['glance_glance_config'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alka
|
||||||
|
10:50:21Z [3 after remove] gate file: /opt/docker/stacks/traefik/dynamic/family-gate-glance.yml
|
||||||
|
10:50:21Z [3 after remove] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 404
|
||||||
|
10:50:21Z == 4 import the bundle
|
||||||
|
10:50:21Z import start -> 400 {'error': 'Missing path', 'ok': False}
|
||||||
|
10:50:21Z /api/export/import/status {"display_name": "Glance", "done": true, "error": "", "job_type": "export", "ok": true, "output_path": "/mnt/felhom-drives/scratch_hdd/exports/glance_20261020-104935.fab", "output_size": "5.6 KB", "running": false, "stack_name": "glance", "steps": [{"label": "Előkészítés", "status": "done"}, {"label": "Konfiguráció mentése", "status": "done"}, {"label": "Adatbázis mentése", "status": "done"}, {"la
|
||||||
|
10:53:42Z state after import: not_deployed family_gate in app_config: False app_config keys: []
|
||||||
|
10:54:14Z [4 FAULT after import] gate file: (none)
|
||||||
|
10:54:14Z [4 FAULT after import] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 404
|
||||||
|
10:54:15Z controller log: 2026/10/20 10:48:52 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/20 10:48:58 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/20 10:49:34 family_gate.go:325: [DEBUG] [web] family gate glance: GET / without a pass — 401 (visitor 192.168.0.180)
|
||||||
|
2026/10/20 10:49:35 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/20 10:49:35 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/20 10:49:37 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/20 10:49:39 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/20 10:49:43 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/20 10:50:09 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
|
||||||
|
2026/10/20 10:50:09 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
|
||||||
|
2026/10/20 10:50:21 auth.go:144: [DEBUG] [web] auth: valid session for POST /api/export/import
|
||||||
|
2026/10/20 10:50:21 handler_export.go:328: [DEBUG] [web] apiImportStart: path="" hasPassword=false
|
||||||
|
2026/10/20 10:50:29 family_gate.go:190: [INFO] [stacks] glance: removed the family-gate file of an app that is not installed
|
||||||
|
2026/10/20 10:52:09 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
|
||||||
|
2026/10/20 10:54:09 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
|
||||||
|
exit 0
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
10:54:26Z == 4 import the bundle /mnt/felhom-drives/scratch_hdd/exports/glance_20261020-104935.fab
|
||||||
|
10:54:26Z import start -> 200 {'ok': True}
|
||||||
|
10:54:26Z /api/export/import/status {"display_name": "Glance", "done": false, "error": "", "job_type": "import", "ok": true, "output_path": "", "output_size": "", "running": true, "stack_name": "glance", "steps": [{"label": "Csomag megnyitása", "status": "done"}, {"label": "Alkalmazás előkészítése", "status": "done"}, {"label": "Konfiguráció visszaállítása", "status": "running"}, {"label": "Felhasználói adatok visszaállítása", "stat
|
||||||
|
10:54:29Z /api/export/import/status {"display_name": "Glance", "done": false, "error": "", "job_type": "import", "ok": true, "output_path": "", "output_size": "", "running": true, "stack_name": "glance", "steps": [{"label": "Csomag megnyitása", "status": "done"}, {"label": "Alkalmazás előkészítése", "status": "done"}, {"label": "Konfiguráció visszaállítása", "status": "done"}, {"label": "Felhasználói adatok visszaállítása", "status"
|
||||||
|
10:54:32Z /api/export/import/status {"display_name": "Glance", "done": true, "error": "", "job_type": "import", "ok": true, "output_path": "", "output_size": "", "running": false, "stack_name": "glance", "steps": [{"label": "Csomag megnyitása", "status": "done"}, {"label": "Alkalmazás előkészítése", "status": "done"}, {"label": "Konfiguráció visszaállítása", "status": "done"}, {"label": "Felhasználói adatok visszaállítása", "status"
|
||||||
|
10:56:13Z state after import: running family_gate in app_config: False app_config keys: ['deployed', 'deployed_at', 'desired_state', 'env', 'installed_images', 'locked_fields']
|
||||||
|
10:56:55Z [4 FAULT after import] gate file: (none)
|
||||||
|
10:56:55Z [4 FAULT after import] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 200
|
||||||
|
10:56:56Z controller log: 2026/10/20 10:54:33 manager.go:1814: [INFO] [stacks] glance glanceapp/glance:v0.8.5 running Up 3 seconds (health: starting)
|
||||||
|
2026/10/20 10:54:39 healthprobe.go:187: [DEBUG] Health probe glance: HTTP GET :8080/ → 200 (1ms)
|
||||||
|
2026/10/20 10:54:49 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 10s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:54:59 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 20s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:55:09 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 30s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:55:19 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 40s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:55:29 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 50s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:55:39 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m0s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:55:49 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m10s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:55:59 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m20s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:56:09 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m30s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:56:19 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m40s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:56:29 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 1m50s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:56:39 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 2m0s ago, effective interval 5m0s, healthy=true
|
||||||
|
2026/10/20 10:56:49 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping glance — last check 2m10s ago, effective interval 5m0s, healthy=true
|
||||||
|
exit 0
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
200
|
||||||
|
{
|
||||||
|
200
|
||||||
|
{
|
||||||
|
"ok": true,
|
||||||
|
Exception in thread Thread-1 (pump):
|
||||||
|
Traceback (most recent call last):
|
||||||
|
File "/usr/lib/python3.13/threading.py", line 1043, in _bootstrap_inner
|
||||||
|
self.run()
|
||||||
|
~~~~~~~~^^
|
||||||
|
File "/usr/lib/python3.13/threading.py", line 994, in run
|
||||||
|
self._target(*self._args, **self._kwargs)
|
||||||
|
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||||
|
File "/mnt/5_hdd/felhom.eu/git/cc-secrets/ccsec/cli.py", line 153, in pump
|
||||||
|
dst.write(m.close()); dst.flush()
|
||||||
|
~~~~~~~~~^^
|
||||||
|
BrokenPipeError: [Errno 32] Broken pipe
|
||||||
|
Exception ignored on flushing sys.stdout:
|
||||||
|
BrokenPipeError: [Errno 32] Broken pipe
|
||||||
|
removed '/mnt/felhom-drives/scratch_hdd/exports/glance_20261020-104935.fab'
|
||||||
|
0
|
||||||
|
0
|
||||||
|
0
|
||||||
|
stranger GET after teardown: 404
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# C3 — proven live: a `.fab` export → import makes a family-only app public
|
||||||
|
|
||||||
|
Scratch guest 9202, controller 0.331.0 (the `.fab` import code — `cmd/controller/main.go:3517-3530`
|
||||||
|
`SaveEncryptedAppConfig`, `stacks/family_gate.go:173-190` — read the same at 0.336.0, `7ca59e2`). Architecture: `09` §3
|
||||||
|
decisions 63/64 (the family gate), 221 (Glance only behind it), `07` §6.6 („never open by default").
|
||||||
|
|
||||||
|
| UTC | act (the dashboard's own endpoints) | observed (a stranger's GET through traefik; the gate file on disk) |
|
||||||
|
|---|---|---|
|
||||||
|
| 10:48:52 | `POST /api/stacks/glance/deploy` | running, `app_config.family_gate` set |
|
||||||
|
| 10:49:34 | **control** | gate file `traefik/dynamic/family-gate-glance.yml` present; stranger → **401** |
|
||||||
|
| 10:49:35 | `POST /api/export/start` (stop ticked, no password) | `glance_20261020-104935.fab`, 5.6 KB |
|
||||||
|
| 10:50:09 | stop + `POST /api/stacks/glance/remove` (data + backups) | removed; 10:50:29 „removed the family-gate file of an app that is not installed" |
|
||||||
|
| 10:54:26 | `POST /api/export/import` (the household's „Importálás") | import done in 6 s; app running |
|
||||||
|
| 10:56:13 | the app's record | `app_config` keys: deployed, deployed_at, desired_state, env, installed_images, locked_fields — **no family_gate** |
|
||||||
|
| 10:56:55 | **fault** | gate file **none**; stranger → **200** (Glance answers anyone who reaches the address) |
|
||||||
|
|
||||||
|
The first run's bundle-list parse was a harness slip (`01-run.log`, step 4 sent an empty path → 400, nothing done);
|
||||||
|
step 4 re-ran alone with the path the export's own status gave (`02-import.log`, `step4.py`). Teardown
|
||||||
|
(`03-teardown.txt`): Glance removed through the product, the bundle deleted (own file), no container, volume or gate
|
||||||
|
file left; the address answers 404.
|
||||||
|
|
||||||
|
On a household box the address is reachable from the internet through the tunnel. Family-gated apps in the catalog:
|
||||||
|
Glance, Grimmory, MeTube (the last two hold the family's books and videos).
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""fab_gate_proof.py — C3 for R-1187 (e) on scratch 9202: does a `.fab` export → remove → import drop the family gate?
|
||||||
|
|
||||||
|
Glance has no login of its own; the catalog puts it ONLY behind the family gate (decision 221). Observables, each
|
||||||
|
read from a DIFFERENT channel than the act: (1) the gate file `family-gate-glance.yml` on the guest's disk; (2) what a
|
||||||
|
stranger's request to the app's host gets back from traefik (302 to the family page vs 200 from Glance).
|
||||||
|
Acts are the dashboard's own endpoints: deploy, /api/export/start, stop+remove, /api/export/import.
|
||||||
|
Env: CTL_BASE, CTL_DOMAIN, CTL_PW_ENV (tools/ctl.py).
|
||||||
|
"""
|
||||||
|
import json, os, subprocess, sys, time
|
||||||
|
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/audits/rehearsal-2026-10-20/tools")
|
||||||
|
import ctl # noqa: E402
|
||||||
|
|
||||||
|
APP, SUB = "glance", "r1187glance"
|
||||||
|
DOM = os.environ["CTL_DOMAIN"]
|
||||||
|
|
||||||
|
|
||||||
|
def say(*a):
|
||||||
|
print(time.strftime("%H:%M:%SZ", time.gmtime()), *a, flush=True)
|
||||||
|
|
||||||
|
|
||||||
|
def guest(cmd):
|
||||||
|
r = subprocess.run(["ssh", "demo-hp", f"export LC_ALL=C; pct exec 9202 -- bash -c {json.dumps(cmd)}"],
|
||||||
|
capture_output=True, text=True, timeout=300)
|
||||||
|
return (r.stdout + r.stderr).strip()
|
||||||
|
|
||||||
|
|
||||||
|
def observe(tag):
|
||||||
|
say(f"[{tag}] gate file:", guest("find / -xdev -name 'family-gate-glance.yml' 2>/dev/null | head -3; true") or "(none)")
|
||||||
|
out = subprocess.run(["curl", "-sk", "-o", "/dev/null", "-w", "%{http_code} %{redirect_url}",
|
||||||
|
"-H", f"Host: {SUB}.{DOM}", ctl.BASE + "/"], capture_output=True, text=True).stdout
|
||||||
|
say(f"[{tag}] a stranger's GET https://{SUB}.{DOM}/ ->", out)
|
||||||
|
|
||||||
|
|
||||||
|
def wait_stack(pred, n=60):
|
||||||
|
for _ in range(n):
|
||||||
|
code, d = ctl.call("GET", f"/api/stacks/{APP}")
|
||||||
|
st = (d.get("data") or {}) if isinstance(d, dict) else {}
|
||||||
|
if pred(st):
|
||||||
|
return st
|
||||||
|
time.sleep(5)
|
||||||
|
return st
|
||||||
|
|
||||||
|
|
||||||
|
def poll(path, n=120):
|
||||||
|
last = None
|
||||||
|
for _ in range(n):
|
||||||
|
code, d = ctl.call("GET", path)
|
||||||
|
dd = (d.get("data") or d) if isinstance(d, dict) else {}
|
||||||
|
s = json.dumps(dd, ensure_ascii=False)[:400]
|
||||||
|
if s != last:
|
||||||
|
say(" ", path, s)
|
||||||
|
last = s
|
||||||
|
if isinstance(dd, dict) and dd.get("status") in ("completed", "failed", "done", "error") or (
|
||||||
|
isinstance(dd, dict) and dd.get("running") is False and dd.get("status") not in ("running", "pending")):
|
||||||
|
return dd
|
||||||
|
time.sleep(3)
|
||||||
|
return dd
|
||||||
|
|
||||||
|
|
||||||
|
say("== 1 deploy glance")
|
||||||
|
code, d = ctl.call("POST", f"/api/stacks/{APP}/deploy", {"values": {"DOMAIN": DOM, "SUBDOMAIN": SUB}})
|
||||||
|
say("deploy ->", code, str(d)[:150])
|
||||||
|
st = wait_stack(lambda s: s.get("deployed") and s.get("state") in ("running", "unhealthy", "degraded"))
|
||||||
|
say("state:", st.get("state"), "family_gate in app_config:", bool((st.get("app_config") or {}).get("family_gate")))
|
||||||
|
time.sleep(20)
|
||||||
|
observe("1 POSITIVE CONTROL after install")
|
||||||
|
|
||||||
|
say("== 2 export to the data drive")
|
||||||
|
drive = "/mnt/felhom-drives/scratch_hdd"
|
||||||
|
code, d = ctl.call("POST", "/api/export/start", {"stack_name": APP, "dest_drive": drive, "password": "", "stop_app": True})
|
||||||
|
say("export start ->", code, str(d)[:200])
|
||||||
|
poll("/api/export/status")
|
||||||
|
code, d = ctl.call("GET", "/api/export/bundles")
|
||||||
|
bundles = [b for b in ((d.get("data") or {}).get("bundles") or d.get("data") or []) if isinstance(b, dict) and APP in json.dumps(b)]
|
||||||
|
say("bundles for glance:", json.dumps(bundles, ensure_ascii=False)[:500])
|
||||||
|
path = (bundles[-1].get("path") if bundles else None)
|
||||||
|
|
||||||
|
say("== 3 remove glance (keep nothing)")
|
||||||
|
ctl.call("POST", f"/api/stacks/{APP}/stop")
|
||||||
|
time.sleep(5)
|
||||||
|
code, d = ctl.call("POST", f"/api/stacks/{APP}/remove", {"remove_hdd_data": True, "remove_backups": True})
|
||||||
|
say("remove ->", code, str(d)[:160])
|
||||||
|
time.sleep(10)
|
||||||
|
observe("3 after remove")
|
||||||
|
|
||||||
|
say("== 4 import the bundle")
|
||||||
|
code, d = ctl.call("POST", "/api/export/import", {"path": path, "password": ""})
|
||||||
|
say("import start ->", code, str(d)[:200])
|
||||||
|
poll("/api/export/import/status")
|
||||||
|
st = wait_stack(lambda s: s.get("deployed") and s.get("state") in ("running", "unhealthy", "degraded"), n=40)
|
||||||
|
say("state after import:", st.get("state"), "family_gate in app_config:", bool((st.get("app_config") or {}).get("family_gate")),
|
||||||
|
"app_config keys:", sorted((st.get("app_config") or {}).keys()))
|
||||||
|
time.sleep(30)
|
||||||
|
observe("4 FAULT after import")
|
||||||
|
say("controller log:", guest("docker logs --since 15m felhom-controller 2>&1 | grep -iE 'family|import' | grep -v 'GET /api' | tail -15"))
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Step 4 of fab_gate_proof.py re-run alone: the first run's bundle-list parse found nothing (harness slip); the
|
||||||
|
export's own status gave the path."""
|
||||||
|
import sys
|
||||||
|
sys.argv = ["x"]
|
||||||
|
exec(open("fab_gate_proof.py").read().split('say("== 1 deploy glance")')[0])
|
||||||
|
path = "/mnt/felhom-drives/scratch_hdd/exports/glance_20261020-104935.fab"
|
||||||
|
say("== 4 import the bundle", path)
|
||||||
|
code, d = ctl.call("POST", "/api/export/import", {"path": path, "password": ""})
|
||||||
|
say("import start ->", code, str(d)[:200])
|
||||||
|
poll("/api/export/import/status")
|
||||||
|
st = wait_stack(lambda s: s.get("deployed") and s.get("state") in ("running", "unhealthy", "degraded"), n=40)
|
||||||
|
say("state after import:", st.get("state"), "family_gate in app_config:", bool((st.get("app_config") or {}).get("family_gate")),
|
||||||
|
"app_config keys:", sorted((st.get("app_config") or {}).keys()))
|
||||||
|
time.sleep(40)
|
||||||
|
observe("4 FAULT after import")
|
||||||
|
say("controller log:", guest("docker logs --since 6m felhom-controller 2>&1 | grep -iE 'family|import|glance' | grep -vE 'GET /api|ScanStacks|Env vars' | tail -15"))
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
# go test at controller 7ca59e2 (v0.336.0), scratch clone; 2026-10-20T11:11:31Z
|
||||||
|
=== RUN TestRehearsalR12_LeftoverTempBlocksEveryRetry
|
||||||
|
zz_rehearsal_r12_test.go:21: retry 1: err=fájlmásolás sikertelen: restoring appdata/app/p1.jpg: open /tmp/TestRehearsalR12_LeftoverTempBlocksEveryRetry3138873193/001/usb/appdata/app/p1.jpg.felhom-restoring: file exists; p1.jpg back=false; app restarted=true
|
||||||
|
zz_rehearsal_r12_test.go:21: retry 2: err=fájlmásolás sikertelen: restoring appdata/app/p1.jpg: open /tmp/TestRehearsalR12_LeftoverTempBlocksEveryRetry3138873193/001/usb/appdata/app/p1.jpg.felhom-restoring: file exists; p1.jpg back=false; app restarted=true
|
||||||
|
--- PASS: TestRehearsalR12_LeftoverTempBlocksEveryRetry (0.01s)
|
||||||
|
PASS
|
||||||
|
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.020s
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
// THROWAWAY proof for the 2026-10-20 data-loss hunt (R-1188 c). Never committed. A crash during the whole restore's
|
||||||
|
// file copy leaves `<file>.felhom-restoring` behind (the copy writes it with O_EXCL); the retry then meets it.
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRehearsalR12_LeftoverTempBlocksEveryRetry(t *testing.T) {
|
||||||
|
f, _, liveLeg := wholeFixture(t)
|
||||||
|
// what a power cut in the middle of copying p1.jpg leaves on the live drive
|
||||||
|
if err := os.MkdirAll(liveLeg, 0o755); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
os.WriteFile(filepath.Join(liveLeg, "p1.jpg.felhom-restoring"), []byte("half"), 0o644)
|
||||||
|
for i := 1; i <= 2; i++ {
|
||||||
|
_, err := f.m.RestoreTier2Whole("app")
|
||||||
|
_, statErr := os.Stat(filepath.Join(liveLeg, "p1.jpg"))
|
||||||
|
t.Logf("retry %d: err=%v; p1.jpg back=%v; app restarted=%v", i, err, statErr == nil, f.fake.fullStarted)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
# go test at controller 7ca59e2 (v0.336.0), scratch clone, fake provider seams; 2026-10-20T10:13:05Z
|
||||||
|
=== RUN TestRehearsalR5_WholeRestoreUnitRefusalAfterFileHalf
|
||||||
|
zz_rehearsal_r5_test.go:22: live files BEFORE: map[p2.jpg:P2 live older]
|
||||||
|
zz_rehearsal_r5_test.go:25: RestoreTier2Whole err = A(z) app adatbázis-másolata csonka: nem ér véget, ahogy egy teljes másolat. A visszaállítás biztonsági okból nem indult el — az alkalmazás és az adatai érintetlenek.
|
||||||
|
zz_rehearsal_r5_test.go:26: file counts: {Restored:1 Replaced:1 KeptNewer:0 Unchanged:0 BytesTotal:17}
|
||||||
|
zz_rehearsal_r5_test.go:27: live files AFTER: map[p1.jpg:P1 p2.jpg:P2 mirror NEWER p2.jpg.felhom-20261020T101306Z:P2 live older]
|
||||||
|
zz_rehearsal_r5_test.go:28: app stopped=true, full start happened=false, provider calls=[stop]
|
||||||
|
--- PASS: TestRehearsalR5_WholeRestoreUnitRefusalAfterFileHalf (0.01s)
|
||||||
|
PASS
|
||||||
|
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.019s
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate R5). Never committed.
|
||||||
|
import (
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestRehearsalR5_WholeRestoreUnitRefusalAfterFileHalf(t *testing.T) {
|
||||||
|
cut := strings.Replace(pgDump(1), "PostgreSQL database dump complete", "", 1) // a cut-off copy (R-640)
|
||||||
|
f := r102Tier2Fixture(t, []string{"app_data.tar"}, cut)
|
||||||
|
f.m.stackProvider = fileAppProvider{f.fake}
|
||||||
|
mid := time.Date(2026, 9, 22, 10, 0, 0, 0, time.UTC)
|
||||||
|
mirrorLeg := filepath.Join(f.destBase, "hdd", "appdata", "app")
|
||||||
|
liveLeg := filepath.Join(f.liveDrive, "appdata", "app")
|
||||||
|
writeAt(t, filepath.Join(mirrorLeg, "p1.jpg"), "P1", mid)
|
||||||
|
writeAt(t, filepath.Join(mirrorLeg, "p2.jpg"), "P2 mirror NEWER", mid.Add(72*time.Hour))
|
||||||
|
writeAt(t, filepath.Join(liveLeg, "p2.jpg"), "P2 live older", mid.Add(48*time.Hour))
|
||||||
|
before := fingerprint(t, liveLeg)
|
||||||
|
t.Logf("live files BEFORE: %v", before)
|
||||||
|
|
||||||
|
res, err := f.m.RestoreTier2Whole("app")
|
||||||
|
t.Logf("RestoreTier2Whole err = %v", err)
|
||||||
|
t.Logf("file counts: %+v", res.Files)
|
||||||
|
t.Logf("live files AFTER: %v", fingerprint(t, liveLeg))
|
||||||
|
t.Logf("app stopped=%v, full start happened=%v, provider calls=%v", f.fake.stopped, f.fake.fullStarted, f.fake.calls)
|
||||||
|
}
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
# go test at controller 7ca59e2 (v0.336.0), scratch clone; the PRODUCTION rsyncMirror (rsync -a --delete) on local temp dirs; 2026-10-20T11:02:49Z
|
||||||
|
=== RUN TestRehearsalT1_NightlyMirrorRunsDuringARestore
|
||||||
|
zz_rehearsal_t1_test.go:26: POSITIVE CONTROL: mirror after the night = [a.jpg b.jpg c.jpg d.jpg]
|
||||||
|
zz_rehearsal_t1_test.go:34: restore in progress: backup flag held = true
|
||||||
|
zz_rehearsal_t1_test.go:37: FAULT: mirror DURING the restore = [a.jpg b.jpg] (c.jpg, d.jpg — the files the restore is bringing back — are gone from the only copy)
|
||||||
|
--- PASS: TestRehearsalT1_NightlyMirrorRunsDuringARestore (0.22s)
|
||||||
|
PASS
|
||||||
|
ok gitea.dooplex.hu/admin/felhom-controller/internal/backup 0.230s
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
package backup
|
||||||
|
|
||||||
|
// THROWAWAY proof for the 2026-10-20 data-loss hunt (candidate T1 / R-1180). Never committed.
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
type t1Provider struct{ *t2v2Provider }
|
||||||
|
|
||||||
|
func (p t1Provider) ListDeployedStacks() []StackSummary { return []StackSummary{{Name: "app"}} }
|
||||||
|
|
||||||
|
func TestRehearsalT1_NightlyMirrorRunsDuringARestore(t *testing.T) {
|
||||||
|
m, src, target, prov := newTier2V2(t, "app")
|
||||||
|
prov.has["app"] = true
|
||||||
|
prov.binds["app"] = []ClassifiedBind{mHDD("appdata/app/photos")}
|
||||||
|
m.stackProvider = t1Provider{prov}
|
||||||
|
m.tier2Mirror = rsyncMirror // the PRODUCTION mirror (rsync -a --delete), local temp dirs only
|
||||||
|
for _, n := range []string{"a.jpg", "b.jpg", "c.jpg", "d.jpg"} {
|
||||||
|
mustWrite(t, filepath.Join(src, "appdata", "app", "photos", n), n)
|
||||||
|
}
|
||||||
|
m.RunAllTier2()
|
||||||
|
mir := filepath.Join(target, "backups", "secondary", "app", "hdd", "appdata", "app", "photos")
|
||||||
|
ls := func(d string) []string { e, _ := os.ReadDir(d); var o []string; for _, x := range e { o = append(o, x.Name()) }; return o }
|
||||||
|
t.Logf("POSITIVE CONTROL: mirror after the night = %v", ls(mir))
|
||||||
|
|
||||||
|
// The household deleted half the photos and starts the second drive's restore; the restore holds the backup flag.
|
||||||
|
os.Remove(filepath.Join(src, "appdata", "app", "photos", "c.jpg"))
|
||||||
|
os.Remove(filepath.Join(src, "appdata", "app", "photos", "d.jpg"))
|
||||||
|
if err := m.acquireRunning(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
t.Logf("restore in progress: backup flag held = %v", m.acquireRunning() != nil)
|
||||||
|
// 03:30 — the nightly second-drive leg fires while the restore is still copying.
|
||||||
|
m.RunAllTier2()
|
||||||
|
t.Logf("FAULT: mirror DURING the restore = %v (c.jpg, d.jpg — the files the restore is bringing back — are gone from the only copy)", ls(mir))
|
||||||
|
m.releaseRunning()
|
||||||
|
}
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
// Replays hub cmd/hub/main.go scheduleDaily's arithmetic (felhom.eu 5c48731, lines 1057-1062) over
|
||||||
|
// 2026-10-23..27 with a simulated clock, for each hub daily job. Prints every fire time.
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"fmt"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
bud, _ := time.LoadLocation("Europe/Budapest")
|
||||||
|
jobs := []struct{ name string; h, m int }{{"db-snapshot", 2, 0}, {"prune", 4, 30}, {"deadline-check", 5, 0}, {"offsite-key-audit", 7, 10}}
|
||||||
|
for _, j := range jobs {
|
||||||
|
now := time.Date(2026, 10, 23, 12, 0, 0, 0, bud)
|
||||||
|
end := time.Date(2026, 10, 27, 12, 0, 0, 0, bud)
|
||||||
|
fmt.Printf("%s (%02d:%02d):\n", j.name, j.h, j.m)
|
||||||
|
for now.Before(end) {
|
||||||
|
n := now.In(bud)
|
||||||
|
next := time.Date(n.Year(), n.Month(), n.Day(), j.h, j.m, 0, 0, bud)
|
||||||
|
if !next.After(n) {
|
||||||
|
next = next.Add(24 * time.Hour)
|
||||||
|
}
|
||||||
|
fmt.Printf(" fires %s (%s)\n", next.In(bud).Format("2006-01-02 15:04 MST"), next.UTC().Format("15:04Z"))
|
||||||
|
now = next // fn() runs instantly; loop re-reads the clock at the fire instant
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
--- go run, go version go1.26.0 linux/amd64
|
||||||
|
db-snapshot (02:00):
|
||||||
|
fires 2026-10-24 02:00 CEST (00:00Z)
|
||||||
|
fires 2026-10-25 02:00 CEST (00:00Z)
|
||||||
|
fires 2026-10-25 02:00 CET (01:00Z)
|
||||||
|
fires 2026-10-26 02:00 CET (01:00Z)
|
||||||
|
fires 2026-10-27 02:00 CET (01:00Z)
|
||||||
|
fires 2026-10-28 02:00 CET (01:00Z)
|
||||||
|
prune (04:30):
|
||||||
|
fires 2026-10-24 04:30 CEST (02:30Z)
|
||||||
|
fires 2026-10-25 03:30 CET (02:30Z)
|
||||||
|
fires 2026-10-25 04:30 CET (03:30Z)
|
||||||
|
fires 2026-10-26 04:30 CET (03:30Z)
|
||||||
|
fires 2026-10-27 04:30 CET (03:30Z)
|
||||||
|
fires 2026-10-28 04:30 CET (03:30Z)
|
||||||
|
deadline-check (05:00):
|
||||||
|
fires 2026-10-24 05:00 CEST (03:00Z)
|
||||||
|
fires 2026-10-25 04:00 CET (03:00Z)
|
||||||
|
fires 2026-10-25 05:00 CET (04:00Z)
|
||||||
|
fires 2026-10-26 05:00 CET (04:00Z)
|
||||||
|
fires 2026-10-27 05:00 CET (04:00Z)
|
||||||
|
fires 2026-10-28 05:00 CET (04:00Z)
|
||||||
|
offsite-key-audit (07:10):
|
||||||
|
fires 2026-10-24 07:10 CEST (05:10Z)
|
||||||
|
fires 2026-10-25 06:10 CET (05:10Z)
|
||||||
|
fires 2026-10-25 07:10 CET (06:10Z)
|
||||||
|
fires 2026-10-26 07:10 CET (06:10Z)
|
||||||
|
fires 2026-10-27 07:10 CET (06:10Z)
|
||||||
|
fires 2026-10-28 07:10 CET (06:10Z)
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
# hub customer_notifications BEFORE hub 0.165.0 (live, with WAL) and in the 2026-10-22T00:00Z snapshot; no address, only whether one is set
|
||||||
|
live Tester-2 addr 2026-10-04 16:30:00 2026-10-04 16:30:00 6 9 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone', 'backup_failed', 'disk_critical', 'host_disk_critical', 'node_down', 'offbox_repo_orphaned', 'storage_fill_critical']
|
||||||
|
live Tester4 addr 2026-10-18 18:18:34 2026-10-18 18:18:34 6 9 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone', 'backup_failed', 'disk_critical', 'host_disk_critical', 'node_down', 'offbox_repo_orphaned', 'storage_fill_critical']
|
||||||
|
live _resend-rotation-test noaddr 2026-06-29 06:02:43 2026-09-24 09:39:35 6 3 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone']
|
||||||
|
live demo-felhom addr 2026-02-16 19:22:40 2026-10-22 07:54:55 6 17 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone', 'backup_failed', 'backup_integrity_failed', 'crossdrive_failed', 'db_dump_failed', 'disk_critical', 'disk_warning', 'expected_backup_missed', 'expected_dbdump_missed', 'health_critical', 'health_recovered', 'node_down', 'offbox_enlarge_blocked', 'storage_disconnected', 'storage_reconnected']
|
||||||
|
live demo-hp addr 2026-08-23 09:54:03 2026-09-24 09:39:35 6 3 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone']
|
||||||
|
live tester-1 addr 2026-10-09 06:09:57 2026-10-10 07:53:24 6 0 []
|
||||||
|
snapshot 00:00Z Tester-2 addr 2026-10-04 16:30:00 2026-10-04 16:30:00 6 9 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone', 'backup_failed', 'disk_critical', 'host_disk_critical', 'node_down', 'offbox_repo_orphaned', 'storage_fill_critical']
|
||||||
|
snapshot 00:00Z Tester4 addr 2026-10-18 18:18:34 2026-10-18 18:18:34 6 9 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone', 'backup_failed', 'disk_critical', 'host_disk_critical', 'node_down', 'offbox_repo_orphaned', 'storage_fill_critical']
|
||||||
|
snapshot 00:00Z _resend-rotation-test noaddr 2026-06-29 06:02:43 2026-09-24 09:39:35 6 3 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone']
|
||||||
|
snapshot 00:00Z demo-felhom addr 2026-02-16 19:22:40 2026-09-24 09:39:35 6 13 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone', 'backup_failed', 'db_dump_failed', 'disk_critical', 'disk_warning', 'expected_backup_missed', 'expected_dbdump_missed', 'health_critical', 'node_down', 'offbox_enlarge_blocked', 'storage_disconnected']
|
||||||
|
snapshot 00:00Z demo-hp addr 2026-08-23 09:54:03 2026-09-24 09:39:35 6 3 ['app_stopped_unhealthy', 'app_update_held', 'app_update_undone']
|
||||||
|
snapshot 00:00Z tester-1 addr 2026-10-09 06:09:57 2026-10-10 07:53:24 6 0 []
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
{"status":"ok"} HTTP 200
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
# after hub 0.165.0 + the demo-felhom restore (live, with WAL); compared with the 00:00Z snapshot
|
||||||
|
Tester-2 2026-10-04 16:30:00 cooldown 6 household_saved 0 9 events | events = snapshot: True | address = snapshot: True
|
||||||
|
Tester4 2026-10-18 18:18:34 cooldown 6 household_saved 0 9 events | events = snapshot: True | address = snapshot: True
|
||||||
|
_resend-rotation-test 2026-09-24 09:39:35 cooldown 6 household_saved 1 3 events | events = snapshot: True | address = snapshot: True
|
||||||
|
demo-felhom 2026-10-22 10:56:44 cooldown 6 household_saved 1 13 events | events = snapshot: True | address = snapshot: True
|
||||||
|
demo-hp 2026-09-24 09:39:35 cooldown 6 household_saved 1 3 events | events = snapshot: True | address = snapshot: True
|
||||||
|
tester-1 2026-10-10 07:53:24 cooldown 6 household_saved 1 0 events | events = snapshot: True | address = snapshot: True
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
### root@192.168.0.154
|
||||||
|
containers=10 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.339.0 Up 35 seconds (healthy)
|
||||||
|
errors=0
|
||||||
|
### demo-hp
|
||||||
|
containers=24 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.339.0 Up 36 seconds (healthy)
|
||||||
|
errors=0
|
||||||
|
### felhom-pve
|
||||||
|
containers=9 unhealthy=0 restarting=0 gitea.dooplex.hu/admin/felhom-controller:0.339.0 Up 39 seconds (healthy)
|
||||||
|
errors=0
|
||||||
|
2026/10/22 11:12:30 [INFO] notification prefs: adopted the household's saved list from the hub (13 events, address set=true) — nothing pushed (R-1199)
|
||||||
|
[('min_controller_version', '0.339.0', '2026-10-22 11:14:21'), ('min_controller_version_declared_min_agent', '0.339.0=0.158.1', '2026-10-22 11:14:21')]
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
control NEGATIVE (a stub answering 200 over plain http): convicted, as it must (200 -)
|
||||||
|
== tester-1 (enkicsifelhom.hu): 6 public name(s)
|
||||||
|
OK control POSITIVE http://felhom.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkicsifelhom.hu/r1033-check?q=1
|
||||||
|
OK http://files.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://files.enkicsifelhom.hu/r1033-check?q=1
|
||||||
|
OK http://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkicsifelhom.hu/r1033-check?q=1
|
||||||
|
OK http://paperless.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paperless.enkicsifelhom.hu/r1033-check?q=1
|
||||||
|
OK http://paste.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://paste.enkicsifelhom.hu/r1033-check?q=1
|
||||||
|
OK http://wiki.enkicsifelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkicsifelhom.hu/r1033-check?q=1
|
||||||
|
== demo-hp (enkisfelhom.hu): 14 public name(s)
|
||||||
|
OK http://arcade.enkisfelhom.hu/r1033-check?q=1 -> 301 https://arcade.enkisfelhom.hu/r1033-check?q=1
|
||||||
|
OK http://books.enkisfelhom.hu/r1033-check?q=1 -> 301 https://books.enkisfelhom.hu/r1033-check?q=1
|
||||||
|
OK http://docs.enkisfelhom.hu/r1033-check?q=1 -> 301 https://docs.enkisfelhom.hu/r1033-check?q=1
|
||||||
|
OK http://felhom.enkisfelhom.hu/r1033-check?q=1 -> 301 https://felhom.enkisfelhom.hu/r1033-check?q=1
|
||||||
|
OK http://files.enkisfelhom.hu/r1033-check?q=1 -> 301 https://files.enkisfelhom.hu/r1033-check?q=1
|
||||||
|
OK http://gist.enkisfelhom.hu/r1033-check?q=1 -> 301 https://gist.enkisfelhom.hu/r1033-check?q=1
|
||||||
|
OK http://ha.enkisfelhom.hu/r1033-check?q=1 -> 301 https://ha.enkisfelhom.hu/r1033-check?q=1
|
||||||
|
OK http://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1 -> 301 https://no-such-name-r1033.enkisfelhom.hu/r1033-check?q=1
|
||||||
|
OK http://papir.enkisfelhom.hu/r1033-check?q=1 -> 301 https://papir.enkisfelhom.hu/r1033-check?q=1
|
||||||
|
OK http://paste.enkisfelhom.hu/r1033-check?q=1 -> 301 https://paste.enkisfelhom.hu/r1033-check?q=1
|
||||||
|
OK http://pdf.enkisfelhom.hu/r1033-check?q=1 -> 301 https://pdf.enkisfelhom.hu/r1033-check?q=1
|
||||||
|
OK http://plex.enkisfelhom.hu/r1033-check?q=1 -> 301 https://plex.enkisfelhom.hu/r1033-check?q=1
|
||||||
|
OK http://travel.enkisfelhom.hu/r1033-check?q=1 -> 301 https://travel.enkisfelhom.hu/r1033-check?q=1
|
||||||
|
OK http://wiki.enkisfelhom.hu/r1033-check?q=1 -> 301 https://wiki.enkisfelhom.hu/r1033-check?q=1
|
||||||
|
== demo-felhom (demo-felhom.eu): 8 public name(s)
|
||||||
|
OK http://felhom.demo-felhom.eu/r1033-check?q=1 -> 301 https://felhom.demo-felhom.eu/r1033-check?q=1
|
||||||
|
OK http://files.demo-felhom.eu/r1033-check?q=1 -> 301 https://files.demo-felhom.eu/r1033-check?q=1
|
||||||
|
OK http://gist.demo-felhom.eu/r1033-check?q=1 -> 301 https://gist.demo-felhom.eu/r1033-check?q=1
|
||||||
|
OK http://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1 -> 301 https://no-such-name-r1033.demo-felhom.eu/r1033-check?q=1
|
||||||
|
OK http://prowlarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://prowlarr.demo-felhom.eu/r1033-check?q=1
|
||||||
|
OK http://radarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://radarr.demo-felhom.eu/r1033-check?q=1
|
||||||
|
OK http://sonarr.demo-felhom.eu/r1033-check?q=1 -> 301 https://sonarr.demo-felhom.eu/r1033-check?q=1
|
||||||
|
OK http://torrent.demo-felhom.eu/r1033-check?q=1 -> 301 https://torrent.demo-felhom.eu/r1033-check?q=1
|
||||||
|
== every public name redirects http -> https
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
# after controller 0.339.0 on the three test boxes (live, with WAL), against the 00:00Z snapshot
|
||||||
|
Tester-2 updated 2026-10-04 16:30:00 household_saved 0 9 events | events = snapshot: True | address = snapshot: True
|
||||||
|
Tester4 updated 2026-10-18 18:18:34 household_saved 0 9 events | events = snapshot: True | address = snapshot: True
|
||||||
|
_resend-rotation-test updated 2026-09-24 09:39:35 household_saved 1 3 events | events = snapshot: True | address = snapshot: True
|
||||||
|
demo-felhom updated 2026-10-22 10:56:44 household_saved 1 13 events | events = snapshot: True | address = snapshot: True
|
||||||
|
demo-hp updated 2026-09-24 09:39:35 household_saved 1 3 events | events = snapshot: True | address = snapshot: True
|
||||||
|
tester-1 updated 2026-10-10 07:53:24 household_saved 1 0 events | events = snapshot: True | address = snapshot: True
|
||||||
|
# swap events today
|
||||||
|
('2026-10-22 07:57:07', 'demo-hp', 'host_swap_short')
|
||||||
|
('2026-10-22 07:57:07', 'demo-felhom', 'host_swap_short')
|
||||||
|
('2026-10-22 11:04:27', 'demo-hp', 'host_swap_ok')
|
||||||
|
('2026-10-22 11:09:41', 'demo-felhom', 'host_swap_ok')
|
||||||
|
# operator mails for host_swap today
|
||||||
|
('2026-10-22 07:57:07', 'demo-felhom', 'host_swap_short', 'sent', 'operator')
|
||||||
|
('2026-10-22 07:57:07', 'demo-felhom', 'host_swap_short', 'skipped', 'customer')
|
||||||
|
('2026-10-22 07:57:07', 'demo-hp', 'host_swap_short', 'sent', 'operator')
|
||||||
|
('2026-10-22 07:57:07', 'demo-hp', 'host_swap_short', 'skipped', 'customer')
|
||||||
|
demo-felhom box file: {'email': 'set', 'enabled_events': 13, 'cooldown_hours': 6}
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
# R-1019 — demo-felhom's second-drive run 2026-10-21 01:30 UTC (03:30 CEST), read 02:36 UTC by the rehearsal session, read-only
|
||||||
|
# channel 1: the controller's own log (docker logs felhom-controller, 01:25–01:45Z)
|
||||||
|
perl: warning: Setting locale failed.
|
||||||
|
perl: warning: Please check that your locale settings:
|
||||||
|
LANGUAGE = (unset),
|
||||||
|
LC_ALL = (unset),
|
||||||
|
LC_CTYPE = "UTF-8",
|
||||||
|
LC_NUMERIC = (unset),
|
||||||
|
LC_COLLATE = (unset),
|
||||||
|
LC_TIME = (unset),
|
||||||
|
LC_MESSAGES = (unset),
|
||||||
|
LC_MONETARY = (unset),
|
||||||
|
LC_ADDRESS = (unset),
|
||||||
|
LC_IDENTIFICATION = (unset),
|
||||||
|
LC_MEASUREMENT = (unset),
|
||||||
|
LC_PAPER = (unset),
|
||||||
|
LC_TELEPHONE = (unset),
|
||||||
|
LC_NAME = (unset),
|
||||||
|
LANG = "en_US.UTF-8"
|
||||||
|
are supported and installed on your system.
|
||||||
|
perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
|
||||||
|
2026/10/21 01:29:59 [INFO] [stacks] Status refresh: 8 containers across 68 stacks
|
||||||
|
2026/10/21 01:30:00 [INFO] [scheduler] Running job: tier2-backup
|
||||||
|
2026/10/21 01:30:00 [INFO] [scheduler] Running job: fill-watch
|
||||||
|
2026/10/21 01:30:00 [INFO] [fillwatch] checked 3 filesystem(s), 0 unreadable/skipped, 0 notification(s); bands: all ok
|
||||||
|
2026/10/21 01:30:00 [INFO] [scheduler] Job fill-watch completed (took 0s)
|
||||||
|
2026/10/21 01:30:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/opengist/docker-compose.yml
|
||||||
|
2026/10/21 01:30:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/opengist/docker-compose.yml
|
||||||
|
2026/10/21 01:30:01 [INFO] [settings] Settings saved
|
||||||
|
2026/10/21 01:30:01 [INFO] [backup] Tier 2 copied opengist → /mnt/felhom-drives/hdd_1/backups/secondary/opengist (190.8 KB, 0 leg(s), 0s)
|
||||||
|
2026/10/21 01:30:01 [INFO] Event pushed: crossdrive_completed (info) [hu-only] — Másodlagos mentés elkészült: opengist
|
||||||
|
2026/10/21 01:30:03 [INFO] [settings] Settings saved
|
||||||
|
2026/10/21 01:30:03 [INFO] [backup] Tier 2 copied prowlarr → /mnt/felhom-drives/hdd_1/backups/secondary/prowlarr (5.0 MB, 0 leg(s), 0s)
|
||||||
|
2026/10/21 01:30:03 [INFO] Event pushed: crossdrive_completed (info) [hu-only] — Másodlagos mentés elkészült: prowlarr
|
||||||
|
2026/10/21 01:30:05 [INFO] [settings] Settings saved
|
||||||
|
2026/10/21 01:30:05 [INFO] [backup] Tier 2 copied radarr → /mnt/sys_drive/felhom-data/backups/secondary/radarr (4.1 MB, 0 leg(s), 0s) [SSD: state-only]
|
||||||
|
2026/10/21 01:30:05 [INFO] Event pushed: crossdrive_completed (info) [hu-only] — Másodlagos mentés elkészült: radarr
|
||||||
|
2026/10/21 01:30:07 [INFO] [settings] Settings saved
|
||||||
|
2026/10/21 01:30:07 [INFO] [backup] Tier 2 copied sonarr → /mnt/sys_drive/felhom-data/backups/secondary/sonarr (8.8 MB, 0 leg(s), 0s) [SSD: state-only]
|
||||||
|
2026/10/21 01:30:07 [INFO] Event pushed: crossdrive_completed (info) [hu-only] — Másodlagos mentés elkészült: sonarr
|
||||||
|
2026/10/21 01:30:08 [INFO] [settings] Settings saved
|
||||||
|
2026/10/21 01:30:08 [INFO] [backup] Tier 2 copied qbittorrent → /mnt/sys_drive/felhom-data/backups/secondary/qbittorrent (8.3 MB, 0 leg(s), 0s) [SSD: state-only]
|
||||||
|
2026/10/21 01:30:08 [INFO] [backup] Tier 2 run complete: 5 app(s) processed (incl. volume-only — F6)
|
||||||
|
2026/10/21 01:30:08 [INFO] [scheduler] Job tier2-backup completed (took 8.954s)
|
||||||
|
2026/10/21 01:30:08 [INFO] Event pushed: crossdrive_completed (info) [hu-only] — Másodlagos mentés elkészült: qbittorrent
|
||||||
|
2026/10/21 01:30:09 [INFO] [stacks] Status refresh: 8 containers across 68 stacks
|
||||||
|
2026/10/21 01:30:09 [INFO] [scheduler] Running job: agent-channel-health
|
||||||
|
# channel 2: the second-drive folder itself (find + du, and a search for *torrent* under every secondary root)
|
||||||
|
/mnt/sys_drive/felhom-data/backups/secondary/qbittorrent
|
||||||
|
/mnt/sys_drive/felhom-data/backups/secondary/qbittorrent/recovery-unit
|
||||||
|
/mnt/sys_drive/felhom-data/backups/secondary/qbittorrent/recovery-unit/compose
|
||||||
|
/mnt/sys_drive/felhom-data/backups/secondary/qbittorrent/recovery-unit/compose/app.yaml
|
||||||
|
/mnt/sys_drive/felhom-data/backups/secondary/qbittorrent/recovery-unit/compose/docker-compose.yml
|
||||||
|
/mnt/sys_drive/felhom-data/backups/secondary/qbittorrent/recovery-unit/compose/.felhom.yml
|
||||||
|
/mnt/sys_drive/felhom-data/backups/secondary/qbittorrent/recovery-unit/volume-dumps
|
||||||
|
/mnt/sys_drive/felhom-data/backups/secondary/qbittorrent/recovery-unit/volume-dumps/qbittorrent_qbittorrent_config.tar
|
||||||
|
/mnt/sys_drive/felhom-data/backups/secondary/qbittorrent/recovery-unit/manifest.json
|
||||||
|
/mnt/sys_drive/felhom-data/backups/secondary/qbittorrent/recovery-unit/data-stamps.json
|
||||||
|
/mnt/sys_drive/felhom-data/backups/secondary/qbittorrent/.felhom-tier2-layout
|
||||||
|
8.3M /mnt/sys_drive/felhom-data/backups/secondary/qbittorrent
|
||||||
|
search:
|
||||||
|
/mnt/sys_drive/felhom-data/backups/secondary/qbittorrent
|
||||||
|
/mnt/sys_drive/felhom-data/backups/secondary/qbittorrent/recovery-unit/volume-dumps/qbittorrent_qbittorrent_config.tar
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# R-895 — the hub's 07:10 CEST off-site snapshot check, 2026-10-21 (the first verdict day after the baseline). Read 05:27 UTC from the running hub pod's log (sudo kubectl logs, pod/hub-8c44cf5cb-srth9), read-only.
|
||||||
|
2026/10/21 07:10:00 [INFO] offsitekeys: audit Tester-2: 1 line(s), all pinned append-only
|
||||||
|
2026/10/21 07:10:01 [WARN] offsitekeys: snapshots Tester-2: the hub's own listing failed: offsitekeys: list snapshots: "ls -l /home/felhom-repo/snapshots": Process exited with status 2: /usr/bin/ls: cannot access '/home/felhom-repo/snapshots': No such file or directory
|
||||||
|
2026/10/21 07:10:01 [INFO] offsitekeys: audit Tester4: 1 line(s), all pinned append-only
|
||||||
|
2026/10/21 07:10:02 [INFO] offsitekeys: snapshots Tester4: the hub's own listing — 44 snapshot file(s), 0 other line(s), newest 2026-10-21T02:18:00Z, oldest 2026-10-20T16:33:00Z; box's last reported run 2026-10-21T02:18:11Z (R-895)
|
||||||
|
2026/10/21 07:10:02 [INFO] offsitekeys: snapshots Tester4: first listing stored as the baseline (44 file(s), 0 too new to judge) — no verdict
|
||||||
|
2026/10/21 07:10:03 [INFO] offsitekeys: audit demo-felhom: 1 line(s), all pinned append-only
|
||||||
|
2026/10/21 07:10:05 [INFO] offsitekeys: snapshots demo-felhom: the hub's own listing — 38 snapshot file(s), 0 other line(s), newest 2026-10-21T02:15:00Z, oldest 2026-08-12T15:15:00Z; box's last reported run 2026-10-21T02:15:35Z (R-895)
|
||||||
|
2026/10/21 07:10:05 [INFO] offsitekeys: snapshots demo-felhom: 5 new snapshot file(s), each inside a reported box run
|
||||||
|
2026/10/21 07:10:06 [INFO] offsitekeys: audit demo-hp: 1 line(s), all pinned append-only
|
||||||
|
2026/10/21 07:10:07 [INFO] offsitekeys: snapshots demo-hp: the hub's own listing — 289 snapshot file(s), 0 other line(s), newest 2026-10-21T02:18:00Z, oldest 2026-08-31T21:29:00Z; box's last reported run 2026-10-21T02:18:21Z (R-895)
|
||||||
|
2026/10/21 07:10:07 [INFO] offsitekeys: snapshots demo-hp: 14 new snapshot file(s), each inside a reported box run
|
||||||
|
2026/10/21 07:10:08 [INFO] offsitekeys: audit tester-1: 1 line(s), all pinned append-only
|
||||||
|
2026/10/21 07:10:09 [INFO] offsitekeys: snapshots tester-1: the hub's own listing — 58 snapshot file(s), 0 other line(s), newest 2026-10-21T02:15:00Z, oldest 2026-10-06T02:15:00Z; box's last reported run 2026-10-21T02:16:05Z (R-895)
|
||||||
|
2026/10/21 07:10:09 [INFO] offsitekeys: snapshots tester-1: 4 new snapshot file(s), each inside a reported box run
|
||||||
|
2026/10/21 07:10:09 [INFO] off-site key check: 5 sub-account(s) read in 9.268s
|
||||||
|
2026/10/21 07:10:09 [INFO] offsite-key-audit: next run at 2026-10-22 07:10 CEST (in 23h59m51s)
|
||||||
|
2026/10/21 07:10:43 [INFO] osupdates: guest candidate 5d6cfda60716db81 (272 packages, first seen 2026-10-21T03:19:43Z): waiting — healthy for 1h51m0s of 24h0m0s
|
||||||
|
2026/10/21 07:10:43 [INFO] osupdates: host candidate 8989bb243244a4f4 (607 packages, first seen 2026-10-21T03:19:43Z): waiting — healthy for 1h51m0s of 24h0m0s
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
GET 200
|
||||||
|
POST 200
|
||||||
|
answer: Felhom — Doboz összekötése Felhom doboz összekötése Kösd össze a most telepített Felhom dobozodat a fiókoddal. Add meg a doboz képernyőjén látható párosító kódot és a tulajdonosi jelmondatodat. A megadott adatok nem megfelelőek. Ellenőrizd a párosító kódot és a tulajdonosi jelmondatot, majd próbáld újra. Párosító kód A doboz monitorán jelenik meg, a telepítés után. Tulajdonosi jelmondat Az öt szób
|
||||||
|
GET 200
|
||||||
|
POST 200
|
||||||
|
answer: Felhom — Doboz összekötése Felhom doboz összekötése Sikeres összekötés. A doboz kb. egy percen belül folytatja a telepítést. Ezt az oldalt bezárhatod — a beállítás a háttérben befejeződik, és a vezérlőpultod hamarosan elérhető lesz. Felhom.eu
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
GET /claim HTTP/2 200
|
||||||
|
csrf_len 64 cookie_present yes
|
||||||
|
POST /claim HTTP/2 302 location: /
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
=== /backups/remote (21160 bytes)
|
||||||
|
Biztonsági mentés — Távoli mentés — Felhom.eu Indítópult Vezérlőpult Alkalmazások Tárhely Meghajtók Hálózati tárhely Biztonsági mentés Áttekintés Távoli mentés Alkalmazások Visszaállítás Megosztás Hálózati megosztás Rendszermonitor Hibakeresés Beállítások Rendszer Értesítések Biztonság és hozzáférés 0.336.0 Magyar English Kijelentkezés ↗ A korábbi távoli mentéseid megvannak, de ehhez a géphez a helyreállítási kódod szükséges. Megnézem Most nem Ne emlékeztessen újra Biztonsági mentés — Távoli mentés tester4.test A korábbi, házon kívüli mentéseid visszaszerezhetők. Ezt a gépet újratelepítették, és a Felhom központi rendszere őriz hozzá egy lezárt csomagot. A helyreállítási kódoddal feloldhatod a korábbi mentéseidet, és megnézheted, mi van bennük. Adatok visszaszerzése Felhom offsite tárhely kiépítve — a beállítás automatikus, folyamatban. Ha egy napon belül nem áll be, jelezd az üzemeltetőnek. Távoli mentés (3. mentés) — titkosított, offsite Az alkalmazás-mentések titkosított másolata egy távoli tárolóra — saját NAS vagy Felhom offsite tárhely — restic + SFTP kapcsolaton. A tároló csak titkosított adatot lát. Ez a 3-2-1 szabály „1 off-site" lába — független a helyi másodpéldánytól és a teljes rendszermentéstől. – Utolsó távoli mentés – Tároló méret · a pillanatképek száma még ismeretlen u629488-sub5@u629488-sub5.your-storagebox.de /home/felhom-repo Távoli mentési cél beállítása Cél címe (IP vagy hosztnév — NAS vagy SFTP-kiszolgáló) SSH port Felhasználó Tároló útvonala a célgépen Napi automatikus távoli mentés engedélyezve SSH privát kulcs A kulcsot 0600-as fájlba írjuk; sosem naplózzuk és nem tároljuk a beállításokban. A célgép ismert-host sora (known_hosts) A host-kulcs rögzítése (no blind TOFU). Lekérdezhető: ssh-keyscan -p <port> <host> Mentés
|
||||||
|
=== / (4642 bytes)
|
||||||
|
Adatok visszaszerzése — Felhom Adatok visszaszerzése Tester4 Ezt a gépet újratelepítették. A korábbi, házon kívüli mentéseid megvannak — a Felhom központi rendszere őriz hozzájuk egy lezárt csomagot, amelyet 2026. 10. 20. 14:16 zártunk le. A csomagot csak a te helyreállítási kódoddal lehet kinyitni. A helyreállítási kódot senki nem tudja pótolni — sem a Felhom, sem az ügyfélszolgálat, sem az üzemeltető. Ez szándékos: így a mentéseidet rajtad kívül senki nem tudja megnyitni. Ha a kód elveszett, a korábbi mentések nem nyithatók meg többé. Ha megadod a kódot, feloldjuk a mentéseid zárolását és megmutatjuk, mi van bennük — melyik alkalmazás, mikorról, mekkora. Ebben a lépésben semmit nem állítunk vissza és semmi nem változik. A visszaállítást utána, alkalmazásonként külön választhatod. Helyreállítási kód (tíz szó) Mentések feloldása Most nem A „Most nem” csak azt jelenti, hogy nem zavarunk vele többet a kezdőlapon. A mentéseid ettől megmaradnak, és ez az oldal a Biztonsági mentés → Távoli mentés oldalról bármikor újra elérhető. Ha a helyreállítási kódod véglegesen elveszett, és tudomásul veszed, hogy a korábbi mentések így nem nyithatók meg többé: ez a lehetőség akkor válik elérhetővé, ha a gép már újra kapcsolódott a házon kívüli tárhelyhez. Addig a mentéseid érintetlenül megmaradnak. Magyar English
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
=== /recovery (4642 bytes)
|
||||||
|
Adatok visszaszerzése — Felhom Adatok visszaszerzése Tester4 Ezt a gépet újratelepítették. A korábbi, házon kívüli mentéseid megvannak — a Felhom központi rendszere őriz hozzájuk egy lezárt csomagot, amelyet 2026. 10. 20. 14:16 zártunk le. A csomagot csak a te helyreállítási kódoddal lehet kinyitni. A helyreállítási kódot senki nem tudja pótolni — sem a Felhom, sem az ügyfélszolgálat, sem az üzemeltető. Ez szándékos: így a mentéseidet rajtad kívül senki nem tudja megnyitni. Ha a kód elveszett, a korábbi mentések nem nyithatók meg többé. Ha megadod a kódot, feloldjuk a mentéseid zárolását és megmutatjuk, mi van bennük — melyik alkalmazás, mikorról, mekkora. Ebben a lépésben semmit nem állítunk vissza és semmi nem változik. A visszaállítást utána, alkalmazásonként külön választhatod. Helyreállítási kód (tíz szó) Mentések feloldása Most nem A „Most nem” csak azt jelenti, hogy nem zavarunk vele többet a kezdőlapon. A mentéseid ettől megmaradnak, és ez az oldal a Biztonsági mentés → Távoli mentés oldalról bármikor újra elérhető. Ha a helyreállítási kódod véglegesen elveszett, és tudomásul veszed, hogy a korábbi mentések így nem nyithatók meg többé: ez a lehetőség akkor válik elérhetővé, ha a gép már újra kapcsolódott a házon kívüli tárhelyhez. Addig a mentéseid érintetlenül megmaradnak. Magyar English
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
status: ['HTTP/2 200 ']
|
||||||
|
page: Adatok visszaszerzése — Felhom Adatok visszaszerzése Tester4 A megadott helyreállítási kódot nem fogadtuk el. Ellenőrizd, hogy mind a tíz szót pontosan, szóközökkel elválasztva írtad be — a kis- és nagybetűk nem számítanak. Semmi nem változott, nyugodtan próbáld újra. Ezt a gépet újratelepítették. A korábbi, házon kívüli mentéseid megvannak — a Felhom központi rendszere őriz hozzájuk egy lezárt csomagot, amelyet 2026. 10. 20. 14:16 zártunk le. A csomagot csak a te helyreállítási kódoddal lehet kinyitni. A helyreállítási kódot senki nem tudja pótolni — sem a Felhom, sem az ügyfélszolgálat, sem az üzemeltető. Ez szándékos: így a mentéseidet rajtad kívül senki nem tudja megnyitni. Ha a kód elveszett, a korábbi mentések nem nyithatók meg többé. Ha megadod a kódot, feloldjuk a mentéseid zárolását és megmutatjuk, mi van bennük — melyik alkalmazás, mikorról, mekkora. Ebben a lépésben semmit nem állítunk vissza és semmi nem változik. A visszaállítást utána, alkalmazásonként külön választhatod. Helyreállítási kód (tíz szó) Mentések feloldása Most nem A „Most nem” csak azt jelenti, hogy nem zavarunk vele többet a kezdőlapon. A mentéseid ettől megmaradnak, és ez az oldal a Biztonsági mentés → Távoli mentés oldalról bármikor újra elérhető. Ha a helyreállítási kódod véglegesen elveszett, és tudomásul veszed, hogy a korábbi mentések így nem nyithatók meg többé: ez a lehetőség akkor válik elérhetővé, ha a gép már újra kapcsolódott a házon kívüli tárhelyhez. Addig a mentéseid érintetlenül me
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
status: ['HTTP/2 200 ']
|
||||||
|
page: Adatok visszaszerzése — Felhom A mentéseid elérhetők A mentéseid zárolása feloldva. Az alábbiakat találtuk a tárolóban — semmit nem állítottunk vissza. Ezek a te mentéseid, a lent jelzett időpontokból. Nézd át, hogy tényleg azt találod-e itt, amire számítasz — semmit nem állítottunk vissza és semmi nem változott. Alkalmazás Legutóbbi mentés Méret _shares 2026. 10. 28. 07:39 1,8 GB actualbudget 2026. 10. 28. 07:39 524,9 KB bookstack 2026. 10. 28. 07:39 154,1 MB immich 2026. 10. 28. 07:39 1,2 GB mealie 2026. 10. 28. 07:39 1,7 MB nextcloud 2026. 10. 28. 07:39 1,0 GB paperless-ngx 2026. 10. 28. 07:39 74,4 MB radicale 2026. 10. 28. 07:39 34,0 KB vaultwarden 2026. 10. 28. 07:39 391,3 KB A visszaállítás alkalmazásonként történik, a Biztonsági mentés → Visszaállítás oldalon. Ott választhatod ki, melyik alkalmazás mit hozzon vissza. Tovább a visszaállításhoz Vissza a kezdőlapra Magyar English
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
=== /backups/restore (25021 bytes)
|
||||||
|
Biztonsági mentés — Visszaállítás — Felhom.eu Indítópult Vezérlőpult Alkalmazások Tárhely Meghajtók Hálózati tárhely Biztonsági mentés Áttekintés Távoli mentés Alkalmazások Visszaállítás Megosztás Hálózati megosztás Rendszermonitor Hibakeresés Beállítások Rendszer Értesítések Biztonság és hozzáférés 0.336.0 Magyar English Kijelentkezés ↗ Biztonsági mentés — Visszaállítás tester4.test Visszaállítás a távoli tárolóból Válaszd ki az alkalmazást, és a következő oldalon döntsd el, mit szeretnél: ellenőrzést külön mappába, csak a hiányzó fájlok visszahozását, vagy teljes visszaállítást. Egyik sem indul el kérdés nélkül. actualbudget Nincs telepítve — előbb telepítsd újra , utána hozhatod vissza az adatait. Visszaállítás… bookstack Nincs telepítve — előbb telepítsd újra , utána hozhatod vissza az adatait. Visszaállítás… immich Nincs telepítve — előbb telepítsd újra , utána hozhatod vissza az adatait. Visszaállítás… mealie Nincs telepítve — előbb telepítsd újra , utána hozhatod vissza az adatait. Visszaállítás… nextcloud Nincs telepítve — előbb telepítsd újra , utána hozhatod vissza az adatait. Visszaállítás… paperless-ngx Nincs telepítve — előbb telepítsd újra , utána hozhatod vissza az adatait. Visszaállítás… radicale Nincs telepítve — előbb telepítsd újra , utána hozhatod vissza az adatait. Visszaállítás… vaultwarden Nincs telepítve — előbb telepítsd újra , utána hozhatod vissza az adatait. Visszaállítás… Meglévő ellenőrző másolatok Nincs ellenőrző másolat a meghajtón. Hordozható mentéscsomag (.fab) Hordozható pillanatfelvétel — bárhol tárolhatod, és bármikor visszatöltheted egy meghajtóról. A folyamatos védelmet az 1–3. szintű mentés adja. Nincs telepített alkalmazás.
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
202 {"ok": true, "message": "Telepítés elindítva – az állapot a kártyán követhető"}
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
12:12:27Z ('deploying', None)
|
||||||
|
12:13:07Z ('degraded', None)
|
||||||
|
12:13:27Z ('starting', None)
|
||||||
|
12:14:27Z ('running', None)
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
12:15:09Z POST /backup/offbox/restore {'app': 'nextcloud', 'mode': 'full'} -> HTTP/2 302 ['location: /backups/restore/app?name=nextcloud&full_prep=nextcloud&full_size=1.0+GB']
|
||||||
|
12:15:09Z POST /backup/offbox/restore {'app': 'nextcloud', 'mode': 'full', 'confirm': '1'} -> HTTP/2 302 ['location: /backups/restore/app?name=nextcloud&flash=flash.offbox.restore_started']
|
||||||
|
12:15:09Z status: {"running": true, "op": "offbox-restore", "stack": "nextcloud", "started_at": "2026-10-21T12:15:09.946691559Z"}
|
||||||
|
12:15:50Z status: {"running": false, "op": "offbox-restore", "stack": "nextcloud", "started_at": "2026-10-21T12:15:09.946691559Z", "last": {"op": "offbox-restore", "stack": "nextcloud", "ok": true, "message": "A(z) nextcloud teljes mentése visszaállítva ellenőrző mappába: /mnt/felhom-drives/belso/backups/offsite-restore/nextcloud — a saját fájljaiddal együtt. A meglévő adatok változatlanok.", "finished_at": "2026-10-21T12:15:47.152729216Z"}, "last_recent": true}
|
||||||
|
12:15:50Z POST /backup/offbox/reconstitute {'app': 'nextcloud', 'confirm': '1'} -> HTTP/2 302 ['location: /backups/restore/app?name=nextcloud&flash=flash.offbox.full_restore_started']
|
||||||
|
12:15:50Z status: {"running": true, "op": "offbox-reconstitute", "stack": "nextcloud", "started_at": "2026-10-21T12:15:50.091899858Z", "last": {"op": "offbox-restore", "stack": "nextcloud", "ok": true, "message": "A(z) nextcloud teljes mentése visszaállítva ellenőrző mappába: /mnt/felhom-drives/belso/backups/offsite-restore/nextcloud — a saját fájljaiddal együtt. A meglévő adatok változatlanok.", "finished_at": "2026-10-21T12:15:47.152729216Z"}, "last_recent": true}
|
||||||
|
12:16:00Z status: {"running": false, "op": "offbox-reconstitute", "stack": "nextcloud", "started_at": "2026-10-21T12:15:50.091899858Z", "last": {"op": "offbox-reconstitute", "stack": "nextcloud", "ok": false, "message": "A teljes visszaállítás sikertelen: A(z) nextcloud mentése szerint az adatok korábban itt voltak: /mnt/felhom-drives/hdd_1. Most viszont ide állna vissza: /mnt/felhom-drives/belso. Ez lehet szándékos — például ha meghajtót cseréltél —, de magától nem folytatjuk. Ha így jó, erősítsd meg alább, és a visszaállítás az új helyre fut.", "finished_at": "2026-10-21T12:15:54.494276193Z"}, "last_recent":
|
||||||
|
12:16:00Z app state: running
|
||||||
|
12:18:57Z POST /backup/offbox/reconstitute {'app': 'nextcloud', 'confirm': '1', 'ack_placement': '1'} -> HTTP/2 302 ['location: /backups/restore/app?name=nextcloud&flash=flash.offbox.full_restore_started']
|
||||||
|
12:18:57Z status: {"running": true, "op": "offbox-reconstitute", "stack": "nextcloud", "started_at": "2026-10-21T12:18:57.09132257Z", "last": {"op": "offbox-reconstitute", "stack": "nextcloud", "ok": false, "message": "A teljes visszaállítás sikertelen: A(z) nextcloud mentése szerint az adatok korábban itt voltak: /mnt/felhom-drives/hdd_1. Most viszont ide állna vissza: /mnt/felhom-drives/belso. Ez lehet szándékos — például ha meghajtót cseréltél —, de magától nem folytatjuk. Ha így jó, erősítsd meg alább, és a visszaállítás az új helyre fut.", "finished_at": "2026-10-21T12:15:54.494276193Z"}, "last_recent": tr
|
||||||
|
12:20:07Z status: {"running": false, "op": "offbox-reconstitute", "stack": "nextcloud", "started_at": "2026-10-21T12:18:57.09132257Z", "last": {"op": "offbox-reconstitute", "stack": "nextcloud", "ok": true, "message": "A(z) nextcloud: 3113 fájl és 2 adatkötet és az adatbázis visszaállítva (mentés: 2026. 10. 28. 07:37) — az alkalmazás újraindult.", "finished_at": "2026-10-21T12:20:03.882952809Z"}, "last_recent": true}
|
||||||
|
12:20:07Z app state: running
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
nextcloud Csalad: manifest 3033 files; on the box 3035; byte-identical 3031; missing/different 2; extra 4
|
||||||
|
MISSING /Csalad/nagy_fajl_1_5GB.bin
|
||||||
|
MISSING /Csalad/Kulonleges_nevek/link_a_nagy_fajlra
|
||||||
|
EXTRA /Csalad/feltoltes_1.jpg
|
||||||
|
EXTRA /Csalad/feltoltes_2.jpg
|
||||||
|
EXTRA /Csalad/feltoltes_3.jpg
|
||||||
|
EXTRA /Csalad/feltoltes_4.jpg
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
status 200 {"last_duration": "", "last_error": "", "last_run": "", "orphaned": false, "progress": {"active": false, "current_app": "", "percent": 0, "bytes_done": 0, "total_bytes": 0, "done_human": "", "total_human": "", "files_done": 0, "total_files": 0, "current_file": "", "elapsed_sec": 0, "phase": ""}, "repo_size_human": "", "snapshots": 0, "status": ""}
|
||||||
|
nextcloud snapshots: 1
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
crash_seen:Tester4-68d1d8 {'boots': 0, 'tripped_at': '', 'rearmed_at': '', 'oops_boot': ''}
|
||||||
|
crash_seen:Tester4-31fa0e {'boots': 0, 'tripped_at': '', 'rearmed_at': '', 'oops_boot': ''}
|
||||||
|
crash_seen:Tester4-816e21 {'boots': 5, 'tripped_at': '2026-10-20T17:34:12Z', 'rearmed_at': '', 'oops_boot': ''}
|
||||||
|
offsite_snapshot_listing:Tester4 {'listed_at': '2026-10-21T05:10:01.734603053Z', 'entries': {'0035e753344c8f2b7fea0e50081d7a88afb698357a0e8ac4f6b830f85d54e34b': '2026-10-20T19:59:00Z', '0297495c6dfcc5b468c95daa2b79cfacc2c1b78de7214e8e98c4f933f70805e0': '2026-10-20T20:00:00Z', '04f7483b259464e87fcb012fa7e020ab7a2014b3907dfbe960bced33ddee3a58': '2026-10-21T02:18:00Z', '0e4825a4db9343c378ca7adecd99dbf27d82649a135fabd728370fdc2d36e17d': '2026-10-21T02:17:00Z', '20e56eda6bb024f63cd03a7672b29ce80f6100c15dcac7b9c4e15b86ec10fbe5': '2026-10-20T17:24:00Z', '226db1616dd52db34d67d22a9c89c7260e177e417e6f8eeae1ff3d0192042eb9': '2026-10-21T02:17:00Z', '2454ff01c39750f4f3772d266822c6e421665c429b22f7326cde66a8f2a63dfc': '2026-10-20T17:23:00Z', '340361eb807a2721d062a88c9ed7f447997c0e3437615fbf6ecc4d202a482da8': '2026-10-20T17:01:00Z', '34457cb6649bc68a698e97abfaa9c98d7304b179c0b481a2309646da388508f0': '2026-10-20T17:23:00Z', '35c54872377f018f9e28aa4ddfd59c6b9584dc6a66c6f83307aedf3a8a38e67e': '2026-10-20T17:23:00Z', '45a6f196bb71aa168ce7242a0858f66422a0a064ac7003f0b9cae5bd6e634080': '2026-10-20T17:01:00Z', '4cfa40bbe8257f9f2c30f82547b9826e39e9a89ee064f00355591d8a01520eb1': '2026-10-20T17:06:00Z', '4e7265c1d07dd0829e5067b9858569127c4ab11743b0a8f848fe4cd5aec38c30': '2026-10-21T02:17:00Z', '5210812729c887baf4f82b8c6e1903cf979d1f489cba3edcd8c3f992751ed8b1': '2026-10-20T17:23:00Z', '63bd7b2776b77d90e4e0c3cce9ce9fe46a3efb7dc5677b572128e3cb515f2aab': '2026-10-20T17:01:00Z', '6baefdc488918a060a01ddfa34efaa6616af3bbac912e44197a2de6c2a8e9a6b': '2026-10-20T17:01:00Z', '6d851619e3d5e084a773d482ec7a3677cc5b29bdaed02c6d59f9eddc7dc2be28': '2026-10-20T17:07:00Z', '73ddca76a1f43d06e25991415500a512d12c8ee676820bd98f3fe9166d3a032c': '2026-10-20T17:06:00Z', '7d62cf64ceda05866dbc8e11aec64a827bf0e06295eed9b85f474ff06f55a4e7': '2026-10-20T17:23:00Z', '884b7302a0a0f611872b608b9b6e19a41c5c02ec916fa5d031195ed1e2609cdb': '2026-10-21T02:17:00Z', '89672cb92aa1bf747de12605fb08a2e74633ec6f90a3166dcb94acf919285ff0': '2026-10-20T19:59:00Z', '922bd5c5ec4f109586817d69ed5f46c594b0016d45a9586810dbc3d4899e24a0': '2026-10-20T17:06:00Z', '9367210c2eab00267b3a28e1421b983951b83c4f3dd0fe0d45d39cec17dacf3d': '2026-10-20T19:59:00Z', '99abedc1beba19fefcde8d3af5a32456fa1eef7d5735adebeb8c0d5eb7422983': '2026-10-20T17:07:00Z', '9bc25446b167361cc4c2fb6759b2b99379f1a0bf63116b74a2bb08957f1ae74b': '2026-10-20T17:07:00Z', 'a99a2e5a04aca65f6ccdc3788d12dd3aa14e93f8ce9cfa4cbc9ac422cc2e3104': '2026-10-20T17:06:00Z', 'a9c7b799e0fbb225cac25ffcd03524091de13b6901c07b18fa6639750214a03f': '2026-10-20T17:23:00Z', 'ab65ba7c79c70a801aa378dd5cc136ceda948f18a74d71ffb60ce7f3f37109ed': '2026-10-20T19:59:00Z', 'b58be9dc628c6adf7093ca9c6fadcc7ce5c8313ea4e267569627d383373045c0': '2026-10-20T17:07:00Z', 'b60565ceb5bc3f1c79ff340dbfda14e5b8feb4b2a395579630d22bde0ba8530d': '2026-10-21T02:17:00Z', 'ca85e58e6d5a8bacc5aca15592c9f68d16283a8dd92b3225a27e79e34444ba2d': '2026-10-20T16:34:00Z', 'd0175c1e45d6c217ff551562b118a98a3d25dbf38a069f4f5a79b254384679d3': '2026-10-20T19:59:00Z', 'd16a10549d220016df5d48112eb5919752a5fb1c3af623224f874bf49ee80378': '2026-10-20T17:24:00Z', 'd5bc5a881f83c507794f44f3ba33e2922177dfadacad57acf65576135a1da58a': '2026-10-20T17:23:00Z', 'd7386835561d4fcb433e81bc58af2bad909779d9f03c4da81bc43009ed4047a9': '2026-10-20T16:33:00Z', 'd94b45adb044464f1e3df6983c46b4ed649df79ed40cdd49ad9d2d6f79723898': '2026-10-20T19:59:00Z', 'dac78d2c5d1893171b3ddca73e1ee14cfddaa6056546f354f372e7e7b5ea4b75': '2026-10-21T02:17:00Z', 'dd54e2e7e79e895643de0b422cbf6f202f913553b273a221df47562c98a308ce': '2026-10-20T16:48:00Z', 'de64c840545fcd0278d454504353bbc826c619e77a4d406aaf4ec3e1479aef8c': '2026-10-20T19:59:00Z', 'e12cc2d54135129d9bdf2542b54ed8e7ad83efbaaa0081f7edcbac76911d5bc3': '2026-10-21T02:18:00Z', 'ee6c61c7ed1384cbb6950295e03d4c29ebc68a5fa08e689c958bb68d91b50e09': '2026-10-20T16:33:00Z', 'f2956c651cedf6ad5d710e10d3902c20ea0f9ce542586f03a645408c474be2a9': '2026-10-20T19:59:00Z', 'f99f3e44d904886cac06945d6fb8be44bbb6fe7583a37b8fde71133ce779b119': '2026-10-21T02:17:00Z', 'fa2b1dd5f8be7cdfee1723a261d7d7e97e1e8e8047cc807a050959d0b62dea75': '2026-10-20T17:06:00Z'}}
|
||||||
|
crash_seen:Tester4-648b80 {'boots': 0, 'tripped_at': '', 'rearmed_at': '', 'oops_boot': ''}
|
||||||
|
crash_seen:Tester4-f6b67c {'boots': 0, 'tripped_at': '', 'rearmed_at': '', 'oops_boot': ''}
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
12:42:44Z POST /backup/offbox/run -> HTTP/2 302 ['location: /backups/remote?flash=flash.offbox.run_started']
|
||||||
|
12:42:44Z ('running', 0, '', '', 0)
|
||||||
|
12:43:45Z ('ok', 63, '', '', 0)
|
||||||
|
{"last_duration": "52s", "last_error": "", "last_run": "2026-10-21T12:43:42Z", "orphaned": false, "progress": {"active": false, "current_app": "", "percent": 0, "bytes_done": 0, "total_bytes": 0, "done_human": "", "total_human": "", "files_done": 0, "total_files": 0, "current_file": "", "elapsed_sec": 0, "phase": ""}, "repo_size_human": "1.4 GB", "snapshots": 63, "status": "ok"}
|
||||||
|
12:44:02Z run 2 -> HTTP/2 302
|
||||||
|
12:45:02Z {'status': 'ok', 'snapshots': 64, 'repo_size_human': '1.4 GB', 'last_run': '2026-10-21T12:44:47Z', 'orphaned': False, 'last_error': ''}
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
POST 200
|
||||||
|
answer: Felhom — Doboz összekötése Felhom doboz összekötése Kész. Ha ez egy valódi hivatkozás volt, és a dobozod még nincs összekötve, néhány percen belül új e-mailt kapsz a regisztrált címedre. Ha nem jön, szólj az ügyfélszolgálatnak. Felhom.eu
|
||||||
|
--- today's spent link:
|
||||||
|
POST 200
|
||||||
|
answer: Felhom — Doboz összekötése Felhom doboz összekötése A régi dobozod még össze van kötve a fiókoddal. Ezért most nem küldünk új linket. Írj a Felhom ügyfélszolgálatának: leválasztjuk a régi dobozt, és utána küldünk új linket. Az új dobozod addig vár; nem kell újra telepítened. Felhom.eu
|
||||||
|
2026/10/21 14:45:25 [WARN] self-bind: Tester4 asked for a new link but host Tester4-f6b67c is still linked — no link sent; the operator is told
|
||||||
|
After Width: | Height: | Size: 90 KiB |
|
After Width: | Height: | Size: 155 KiB |
|
After Width: | Height: | Size: 136 KiB |
|
After Width: | Height: | Size: 129 KiB |
|
After Width: | Height: | Size: 132 KiB |
|
After Width: | Height: | Size: 133 KiB |
|
After Width: | Height: | Size: 148 KiB |
|
After Width: | Height: | Size: 148 KiB |
|
After Width: | Height: | Size: 131 KiB |
|
After Width: | Height: | Size: 1.3 KiB |
@@ -0,0 +1,8 @@
|
|||||||
|
Install pressed 11:56:12
|
||||||
|
pairing screen seen 11:59:59
|
||||||
|
bind POST 12:01:16
|
||||||
|
bind POST 2 12:02:58
|
||||||
|
claim 12:08:22
|
||||||
|
wrong code 12:09:55
|
||||||
|
right code 12:10:13
|
||||||
|
nextcloud install 12:12:15
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
13:48:37Z == 1 deploy glance
|
||||||
|
13:48:37Z deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||||
|
13:48:52Z state: running family_gate in app_config: True
|
||||||
|
13:49:14Z [1 POSITIVE CONTROL after install] gate file: /opt/docker/stacks/traefik/dynamic/family-gate-glance.yml
|
||||||
|
13:49:14Z [1 POSITIVE CONTROL after install] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 401
|
||||||
|
13:49:14Z == 2 export to the data drive
|
||||||
|
13:49:14Z export start -> 200 {'ok': True}
|
||||||
|
13:49:14Z /api/export/status {"display_name": "Glance", "done": false, "error": "", "job_type": "export", "ok": true, "output_path": "", "output_size": "", "running": true, "stack_name": "glance", "steps": [{"label": "Előkészítés", "status": "running"}, {"label": "Konfiguráció mentése", "status": "pending"}, {"label": "Adatbázis mentése", "status": "pending"}, {"label": "Felhasználói adatok", "status": "pending"}, {"label": "
|
||||||
|
13:49:17Z /api/export/status {"display_name": "Glance", "done": false, "error": "", "job_type": "export", "ok": true, "output_path": "/mnt/felhom-drives/scratch_hdd/exports/glance_20261021-134917.fab", "output_size": "5.6 KB", "running": true, "stack_name": "glance", "steps": [{"label": "Előkészítés", "status": "done"}, {"label": "Konfiguráció mentése", "status": "done"}, {"label": "Adatbázis mentése", "status": "done"}, {"la
|
||||||
|
13:49:20Z /api/export/status {"display_name": "Glance", "done": true, "error": "", "job_type": "export", "ok": true, "output_path": "/mnt/felhom-drives/scratch_hdd/exports/glance_20261021-134917.fab", "output_size": "5.6 KB", "running": false, "stack_name": "glance", "steps": [{"label": "Előkészítés", "status": "done"}, {"label": "Konfiguráció mentése", "status": "done"}, {"label": "Adatbázis mentése", "status": "done"}, {"la
|
||||||
|
13:49:20Z bundles for glance: []
|
||||||
|
13:49:20Z == 3 remove glance (keep nothing)
|
||||||
|
13:49:53Z remove -> 200 {'ok': True, 'data': {'removed': 'glance', 'volumes_removed': ['glance_glance_config'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alka
|
||||||
|
13:50:05Z [3 after remove] gate file: (none)
|
||||||
|
13:50:05Z [3 after remove] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 404
|
||||||
|
13:50:05Z == 4 import the bundle
|
||||||
|
13:50:05Z import start -> 400 {'error': 'Missing path', 'ok': False}
|
||||||
|
13:50:05Z /api/export/import/status {"display_name": "Glance", "done": true, "error": "", "job_type": "export", "ok": true, "output_path": "/mnt/felhom-drives/scratch_hdd/exports/glance_20261021-134917.fab", "output_size": "5.6 KB", "running": false, "stack_name": "glance", "steps": [{"label": "Előkészítés", "status": "done"}, {"label": "Konfiguráció mentése", "status": "done"}, {"label": "Adatbázis mentése", "status": "done"}, {"la
|
||||||
|
13:53:26Z state after import: not_deployed family_gate in app_config: False app_config keys: []
|
||||||
|
13:53:57Z [4 AFTER import (fixed: must be 401, gate file present)] gate file: (none)
|
||||||
|
13:53:57Z [4 AFTER import (fixed: must be 401, gate file present)] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 404
|
||||||
|
13:53:59Z controller log: 2026/10/21 13:48:37 family_gate.go:145: [INFO] [stacks] glance: family gate ON before the first start — only family members reach [r1187glance.enkisfelhom.hu] (exceptions: [])
|
||||||
|
2026/10/21 13:48:37 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/21 13:48:44 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/21 13:49:14 family_gate.go:325: [DEBUG] [web] family gate glance: GET / without a pass — 401 (visitor 192.168.0.180)
|
||||||
|
2026/10/21 13:49:15 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/21 13:49:17 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/21 13:49:21 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/21 13:49:21 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/21 13:49:26 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/21 13:49:53 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
|
||||||
|
2026/10/21 13:50:00 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
|
||||||
|
2026/10/21 13:50:00 family_gate.go:190: [INFO] [stacks] glance: removed the family-gate file of an app that is not installed
|
||||||
|
2026/10/21 13:50:05 auth.go:144: [DEBUG] [web] auth: valid session for POST /api/export/import
|
||||||
|
2026/10/21 13:50:05 handler_export.go:328: [DEBUG] [web] apiImportStart: path="" hasPassword=false
|
||||||
|
2026/10/21 13:52:00 manager.go:675: [DEBUG] [stacks] ScanStacks: found stack "recipe-importer" deployed=false composePath=/opt/docker/stacks/recipe-importer/docker-compose.yml
|
||||||
|
13:53:59Z == 5 teardown: remove glance through the product, delete the bundle
|
||||||
|
13:54:31Z remove -> 200 {'ok': True, 'data': {'removed': 'glance', 'volumes_removed': [], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alkalmazás nem tárolt sajá
|
||||||
|
13:54:43Z [5 after teardown] gate file: (none)
|
||||||
|
13:54:43Z [5 after teardown] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 404
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
13:55:01Z == 1 deploy glance
|
||||||
|
13:55:02Z deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||||
|
13:55:22Z state: running family_gate in app_config: True
|
||||||
|
13:55:44Z [1 POSITIVE CONTROL after install] gate file: /opt/docker/stacks/traefik/dynamic/family-gate-glance.yml
|
||||||
|
13:55:44Z [1 POSITIVE CONTROL after install] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 401
|
||||||
|
13:55:44Z == 2 export to the data drive
|
||||||
|
13:55:44Z export start -> 200 {'ok': True}
|
||||||
|
13:55:44Z /api/export/status {"display_name": "Glance", "done": false, "error": "", "job_type": "export", "ok": true, "output_path": "", "output_size": "", "running": true, "stack_name": "glance", "steps": [{"label": "Előkészítés", "status": "running"}, {"label": "Konfiguráció mentése", "status": "pending"}, {"label": "Adatbázis mentése", "status": "pending"}, {"label": "Felhasználói adatok", "status": "pending"}, {"label": "
|
||||||
|
13:55:47Z /api/export/status {"display_name": "Glance", "done": true, "error": "", "job_type": "export", "ok": true, "output_path": "/mnt/felhom-drives/scratch_hdd/exports/glance_20261021-135546.fab", "output_size": "5.6 KB", "running": false, "stack_name": "glance", "steps": [{"label": "Előkészítés", "status": "done"}, {"label": "Konfiguráció mentése", "status": "done"}, {"label": "Adatbázis mentése", "status": "done"}, {"la
|
||||||
|
13:55:47Z bundles for glance: []
|
||||||
|
13:55:47Z bundle path: /mnt/felhom-drives/scratch_hdd/exports/glance_20261021-135546.fab
|
||||||
|
13:55:47Z == 3 remove glance (keep nothing)
|
||||||
|
13:56:19Z remove -> 200 {'ok': True, 'data': {'removed': 'glance', 'volumes_removed': ['glance_glance_config'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alka
|
||||||
|
13:56:31Z [3 after remove] gate file: (none)
|
||||||
|
13:56:31Z [3 after remove] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 404
|
||||||
|
13:56:31Z == 4 import the bundle
|
||||||
|
13:56:31Z import start -> 200 {'ok': True}
|
||||||
|
13:56:31Z /api/export/import/status {"display_name": "Glance", "done": false, "error": "", "job_type": "import", "ok": true, "output_path": "", "output_size": "", "running": true, "stack_name": "glance", "steps": [{"label": "Csomag megnyitása", "status": "done"}, {"label": "Alkalmazás előkészítése", "status": "done"}, {"label": "Konfiguráció visszaállítása", "status": "done"}, {"label": "Felhasználói adatok visszaállítása", "status"
|
||||||
|
13:56:37Z /api/export/import/status {"display_name": "Glance", "done": true, "error": "", "job_type": "import", "ok": true, "output_path": "", "output_size": "", "running": false, "stack_name": "glance", "steps": [{"label": "Csomag megnyitása", "status": "done"}, {"label": "Alkalmazás előkészítése", "status": "done"}, {"label": "Konfiguráció visszaállítása", "status": "done"}, {"label": "Felhasználói adatok visszaállítása", "status"
|
||||||
|
13:56:52Z state after import: running family_gate in app_config: True app_config keys: ['deployed', 'deployed_at', 'desired_state', 'env', 'family_gate', 'installed_images', 'locked_fields']
|
||||||
|
13:57:24Z [4 AFTER import (fixed: must be 401, gate file present)] gate file: /opt/docker/stacks/traefik/dynamic/family-gate-glance.yml
|
||||||
|
13:57:24Z [4 AFTER import (fixed: must be 401, gate file present)] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 401
|
||||||
|
13:57:26Z controller log: 2026/10/21 13:56:31 handler_export.go:352: [INFO] [web] Import started from /mnt/felhom-drives/scratch_hdd/exports/glance_20261021-135546.fab
|
||||||
|
2026/10/21 13:56:31 restore.go:223: [INFO] [appexport] Import started for glance_20261021-135546.fab
|
||||||
|
2026/10/21 13:56:31 export.go:154: [DEBUG] [appexport] === IMPORT START: path=/mnt/felhom-drives/scratch_hdd/exports/glance_20261021-135546.fab encrypted=false ===
|
||||||
|
2026/10/21 13:56:31 export.go:154: [DEBUG] [appexport] temp dir: /tmp/felhom-import-3760501659
|
||||||
|
2026/10/21 13:56:31 restore.go:330: [INFO] Import: opening bundle for glance (Glance)
|
||||||
|
2026/10/21 13:56:31 export.go:154: [DEBUG] [appexport] restoring config from /tmp/felhom-import-3760501659/config → /opt/docker/stacks/glance
|
||||||
|
2026/10/21 13:56:31 import_record.go:51: [INFO] [stacks] glance: import record saved (fresh=true, family gate carried=false, sign-up lock carried=false)
|
||||||
|
2026/10/21 13:56:31 restore.go:762: [INFO] Import: populating volume glance_glance_config
|
||||||
|
2026/10/21 13:56:31 family_gate.go:145: [INFO] [stacks] glance: family gate ON before the first start — only family members reach [r1187glance.enkisfelhom.hu] (exceptions: [])
|
||||||
|
2026/10/21 13:56:31 import_record.go:95: [INFO] [stacks] glance: import prepared before the start (family gate=true, sign-up lock=false)
|
||||||
|
2026/10/21 13:56:31 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/21 13:56:35 restore.go:485: [INFO] Import completed: glance (Glance) in 4.029790494s
|
||||||
|
2026/10/21 13:56:35 export.go:154: [DEBUG] [appexport] === IMPORT END: elapsed=4.030525583s ===
|
||||||
|
2026/10/21 13:56:38 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SUBDOMAIN, IMPORT_PATH] (8 app + 0 system)
|
||||||
|
2026/10/21 13:57:24 family_gate.go:325: [DEBUG] [web] family gate glance: GET / without a pass — 401 (visitor 192.168.0.180)
|
||||||
|
13:57:26Z == 5 teardown: remove glance through the product, delete the bundle
|
||||||
|
13:57:58Z remove -> 200 {'ok': True, 'data': {'removed': 'glance', 'volumes_removed': ['glance_glance_config'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az alka
|
||||||
|
13:57:59Z bundle delete: deleted /mnt/felhom-drives/scratch_hdd/exports/glance_20261021-135546.fab
|
||||||
|
13:58:11Z [5 after teardown] gate file: (none)
|
||||||
|
13:58:11Z [5 after teardown] a stranger's GET https://r1187glance.enkisfelhom.hu/ -> 404
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
load started 13:26:08
|
||||||
|
exp1 swap 512->4096 at 13:34:06
|
||||||
|
proof started 13:43:45 (swap cap 4096, guest RAM 4096)
|
||||||
|
evidence copied 16:27:04
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
13:10:54Z immich deploy 202 Telepítés elindítva – az állapot a kártyán követhető
|
||||||
|
13:13:25Z immich → running 152 s
|
||||||
|
13:13:25Z paperless-ngx deploy 202 Telepítés elindítva – az állapot a kártyán követhető
|
||||||
|
13:15:46Z paperless-ngx → running 140 s
|
||||||
|
13:15:46Z vaultwarden deploy 202 Telepítés elindítva – az állapot a kártyán követhető
|
||||||
|
13:16:06Z vaultwarden → running 20 s
|
||||||
|
13:16:06Z mealie deploy 202 Telepítés elindítva – az állapot a kártyán követhető
|
||||||
|
13:17:36Z mealie → running 90 s
|
||||||
|
13:17:37Z actualbudget deploy 202 Telepítés elindítva – az állapot a kártyán követhető
|
||||||
|
13:18:07Z actualbudget → running 30 s
|
||||||
|
13:18:07Z radicale deploy 202 Telepítés elindítva – az állapot a kártyán követhető
|
||||||
|
13:18:27Z radicale → running 20 s
|
||||||
|
13:18:27Z bookstack deploy 202 Telepítés elindítva – az állapot a kártyán követhető
|
||||||
|
13:19:37Z bookstack → running 70 s
|
||||||
|
13:19:37Z memory: {"data": {"vmid": 9201, "allocated_mb": 4096, "usage_mb": 2550, "host_total_mb": 7939, "min_mb": 2048, "max_mb": 5891, "floor_mb": 3062, "running": true}, "ok": true}
|
||||||
@@ -0,0 +1,312 @@
|
|||||||
|
13:27:30Z LOAD START
|
||||||
|
13:27:30Z immich admin-sign-up 201
|
||||||
|
/root/load.py:42: DeprecationWarning: datetime.datetime.utcfromtimestamp() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.fromtimestamp(timestamp, datetime.UTC).
|
||||||
|
st = os.stat(f); ts = dt.datetime.utcfromtimestamp(st.st_mtime).isoformat() + "Z"
|
||||||
|
13:27:31Z immich IMG_20090000.jpg 201
|
||||||
|
13:27:31Z immich IMG_20090018.jpg 201
|
||||||
|
13:27:31Z immich IMG_20090036.jpg 201
|
||||||
|
13:27:31Z immich IMG_20090054.jpg 201
|
||||||
|
13:27:31Z immich IMG_20090072.jpg 201
|
||||||
|
13:27:31Z immich IMG_20090090.jpg 201
|
||||||
|
13:27:31Z immich IMG_20090108.jpg 201
|
||||||
|
13:27:32Z immich IMG_20090126.jpg 201
|
||||||
|
13:27:32Z immich IMG_20090144.jpg 201
|
||||||
|
13:27:32Z immich IMG_20090162.jpg 201
|
||||||
|
13:27:32Z immich IMG_20090180.jpg 201
|
||||||
|
13:27:32Z immich IMG_20090198.jpg 201
|
||||||
|
13:27:32Z immich IMG_20090216.jpg 201
|
||||||
|
13:27:32Z immich IMG_20090234.jpg 201
|
||||||
|
13:27:32Z immich IMG_20090252.jpg 201
|
||||||
|
13:27:32Z immich IMG_20090270.jpg 201
|
||||||
|
13:27:32Z immich IMG_20090288.jpg 201
|
||||||
|
13:27:32Z immich IMG_20100001.jpg 201
|
||||||
|
13:27:33Z immich IMG_20100019.jpg 201
|
||||||
|
13:27:33Z immich IMG_20100037.jpg 201
|
||||||
|
13:27:33Z immich IMG_20100055.jpg 201
|
||||||
|
13:27:33Z immich IMG_20100073.jpg 201
|
||||||
|
13:27:33Z immich IMG_20100091.jpg 201
|
||||||
|
13:27:33Z immich IMG_20100109.jpg 201
|
||||||
|
13:27:33Z immich IMG_20100127.jpg 201
|
||||||
|
13:27:33Z immich IMG_20100145.jpg 201
|
||||||
|
13:27:34Z immich IMG_20100163.jpg 201
|
||||||
|
13:27:34Z immich IMG_20100181.jpg 201
|
||||||
|
13:27:34Z immich IMG_20100199.jpg 201
|
||||||
|
13:27:34Z immich IMG_20100217.jpg 201
|
||||||
|
13:27:34Z immich IMG_20100235.jpg 201
|
||||||
|
13:27:35Z immich IMG_20100253.jpg 201
|
||||||
|
13:27:35Z immich IMG_20100271.jpg 201
|
||||||
|
13:27:35Z immich IMG_20100289.jpg 201
|
||||||
|
13:27:35Z immich IMG_20110002.jpg 201
|
||||||
|
13:27:35Z immich IMG_20110020.jpg 201
|
||||||
|
13:27:35Z immich IMG_20110038.jpg 201
|
||||||
|
13:27:35Z immich IMG_20110056.jpg 201
|
||||||
|
13:27:36Z immich IMG_20110074.jpg 201
|
||||||
|
13:27:36Z immich IMG_20110092.jpg 201
|
||||||
|
13:27:37Z immich IMG_20110110.jpg 201
|
||||||
|
13:27:37Z immich IMG_20110128.jpg 201
|
||||||
|
13:27:37Z immich IMG_20110146.jpg 201
|
||||||
|
13:27:37Z immich IMG_20110164.jpg 201
|
||||||
|
13:27:38Z immich IMG_20110182.jpg 201
|
||||||
|
13:27:38Z immich IMG_20110200.jpg 201
|
||||||
|
13:27:38Z immich IMG_20110218.jpg 201
|
||||||
|
13:27:38Z immich IMG_20110236.jpg 201
|
||||||
|
13:27:38Z immich IMG_20110254.jpg 201
|
||||||
|
13:27:38Z immich IMG_20110272.jpg 201
|
||||||
|
13:27:38Z immich IMG_20110290.jpg 201
|
||||||
|
13:27:38Z immich IMG_20120003.jpg 201
|
||||||
|
13:27:39Z immich IMG_20120021.jpg 201
|
||||||
|
13:27:39Z immich IMG_20120039.jpg 201
|
||||||
|
13:27:39Z immich IMG_20120057.jpg 201
|
||||||
|
13:27:40Z immich IMG_20120075.jpg 201
|
||||||
|
13:27:40Z immich IMG_20120093.jpg 201
|
||||||
|
13:27:40Z immich IMG_20120111.jpg 201
|
||||||
|
13:27:41Z immich IMG_20120129.jpg 201
|
||||||
|
13:27:41Z immich IMG_20120147.jpg 201
|
||||||
|
13:27:42Z immich IMG_20120165.jpg 201
|
||||||
|
13:27:42Z immich IMG_20120183.jpg 201
|
||||||
|
13:27:42Z immich IMG_20120201.jpg 201
|
||||||
|
13:27:42Z immich IMG_20120219.jpg 201
|
||||||
|
13:27:42Z immich IMG_20120237.jpg 201
|
||||||
|
13:27:42Z immich IMG_20120255.jpg 201
|
||||||
|
13:27:42Z immich IMG_20120273.jpg 201
|
||||||
|
13:27:43Z immich IMG_20120291.jpg 201
|
||||||
|
13:27:43Z immich IMG_20130004.jpg 201
|
||||||
|
13:27:43Z immich IMG_20130022.jpg 201
|
||||||
|
13:27:43Z immich IMG_20130040.jpg 201
|
||||||
|
13:27:43Z immich IMG_20130058.jpg 201
|
||||||
|
13:27:43Z immich IMG_20130076.jpg 201
|
||||||
|
13:27:44Z immich IMG_20130094.jpg 201
|
||||||
|
13:27:44Z immich IMG_20130112.jpg 201
|
||||||
|
13:27:44Z immich IMG_20130130.jpg 201
|
||||||
|
13:27:44Z immich IMG_20130148.jpg 201
|
||||||
|
13:27:44Z immich IMG_20130166.jpg 201
|
||||||
|
13:27:44Z immich IMG_20130184.jpg 201
|
||||||
|
13:27:45Z immich IMG_20130202.jpg 201
|
||||||
|
13:27:45Z immich IMG_20130220.jpg 201
|
||||||
|
13:27:45Z immich IMG_20130238.jpg 201
|
||||||
|
13:27:45Z immich IMG_20130256.jpg 201
|
||||||
|
13:27:46Z immich IMG_20130274.jpg 201
|
||||||
|
13:27:46Z immich IMG_20130292.jpg 201
|
||||||
|
13:27:46Z immich IMG_20140005.jpg 201
|
||||||
|
13:27:55Z immich IMG_20140023.jpg 201
|
||||||
|
13:27:55Z immich IMG_20140041.jpg 201
|
||||||
|
13:27:55Z immich IMG_20140059.jpg 201
|
||||||
|
13:27:56Z immich IMG_20140077.jpg 201
|
||||||
|
13:27:58Z immich IMG_20140095.jpg 201
|
||||||
|
13:27:58Z immich IMG_20140113.jpg 201
|
||||||
|
13:27:58Z immich IMG_20140131.jpg 201
|
||||||
|
13:27:58Z immich IMG_20140149.jpg 201
|
||||||
|
13:27:58Z immich IMG_20140167.jpg 201
|
||||||
|
13:27:58Z immich IMG_20140185.jpg 201
|
||||||
|
13:27:59Z immich IMG_20140203.jpg 201
|
||||||
|
13:27:59Z immich IMG_20140221.jpg 201
|
||||||
|
13:27:59Z immich IMG_20140239.jpg 201
|
||||||
|
13:28:00Z immich IMG_20140257.jpg 201
|
||||||
|
13:28:00Z immich IMG_20140275.jpg 201
|
||||||
|
13:28:00Z immich IMG_20140293.jpg 201
|
||||||
|
13:28:00Z immich IMG_20150006.jpg 201
|
||||||
|
13:28:00Z immich IMG_20150024.jpg 201
|
||||||
|
13:28:00Z immich IMG_20150042.jpg 201
|
||||||
|
13:28:00Z immich IMG_20150060.jpg 201
|
||||||
|
13:28:01Z immich IMG_20150078.jpg 201
|
||||||
|
13:28:01Z immich IMG_20150096.jpg 201
|
||||||
|
13:28:01Z immich IMG_20150114.jpg 201
|
||||||
|
13:28:01Z immich IMG_20150132.jpg 201
|
||||||
|
13:28:01Z immich IMG_20150150.jpg 201
|
||||||
|
13:28:01Z immich IMG_20150168.jpg 201
|
||||||
|
13:28:01Z immich IMG_20150186.jpg 201
|
||||||
|
13:28:02Z immich IMG_20150204.jpg 201
|
||||||
|
13:28:02Z immich IMG_20150222.jpg 201
|
||||||
|
13:28:02Z immich IMG_20150240.jpg 201
|
||||||
|
13:28:02Z immich IMG_20150258.jpg 201
|
||||||
|
13:28:02Z immich IMG_20150276.jpg 201
|
||||||
|
13:28:03Z immich IMG_20150294.jpg 201
|
||||||
|
13:28:03Z immich IMG_20160007.jpg 201
|
||||||
|
13:28:03Z immich IMG_20160025.jpg 201
|
||||||
|
13:28:03Z immich IMG_20160043.jpg 201
|
||||||
|
13:28:04Z immich IMG_20160061.jpg 201
|
||||||
|
13:28:04Z immich IMG_20160079.jpg 201
|
||||||
|
13:28:04Z immich IMG_20160097.jpg 201
|
||||||
|
13:28:04Z immich IMG_20160115.jpg 201
|
||||||
|
13:28:05Z immich IMG_20160133.jpg 201
|
||||||
|
13:28:05Z immich IMG_20160151.jpg 201
|
||||||
|
13:28:05Z immich IMG_20160169.jpg 201
|
||||||
|
13:28:05Z immich IMG_20160187.jpg 201
|
||||||
|
13:28:06Z immich IMG_20160205.jpg 201
|
||||||
|
13:28:07Z immich IMG_20160223.jpg 201
|
||||||
|
13:28:07Z immich IMG_20160241.jpg 201
|
||||||
|
13:28:08Z immich IMG_20160259.jpg 201
|
||||||
|
13:28:08Z immich IMG_20160277.jpg 201
|
||||||
|
13:28:08Z immich IMG_20160295.jpg 201
|
||||||
|
13:28:09Z immich IMG_20170008.jpg 201
|
||||||
|
13:28:09Z immich IMG_20170026.jpg 201
|
||||||
|
13:28:09Z immich IMG_20170044.jpg 201
|
||||||
|
13:28:10Z immich IMG_20170062.jpg 201
|
||||||
|
13:28:10Z immich IMG_20170080.jpg 201
|
||||||
|
13:28:10Z immich IMG_20170098.jpg 201
|
||||||
|
13:28:10Z immich IMG_20170116.jpg 201
|
||||||
|
13:28:11Z immich IMG_20170134.jpg 201
|
||||||
|
13:28:11Z immich IMG_20170152.jpg 201
|
||||||
|
13:28:11Z immich IMG_20170170.jpg 201
|
||||||
|
13:34:32Z immich IMG_20170188.jpg 201
|
||||||
|
13:34:37Z immich IMG_20170206.jpg 502
|
||||||
|
13:34:55Z immich IMG_20170224.jpg 201
|
||||||
|
13:34:56Z immich IMG_20170242.jpg 201
|
||||||
|
13:34:56Z immich IMG_20170260.jpg 201
|
||||||
|
13:34:56Z immich IMG_20170278.jpg 201
|
||||||
|
13:34:57Z immich IMG_20170296.jpg 201
|
||||||
|
13:34:57Z immich IMG_20180009.jpg 201
|
||||||
|
13:34:57Z immich IMG_20180027.jpg 201
|
||||||
|
13:34:57Z immich IMG_20180045.jpg 201
|
||||||
|
13:34:58Z immich IMG_20180063.jpg 201
|
||||||
|
13:34:58Z immich IMG_20180081.jpg 201
|
||||||
|
13:34:59Z immich IMG_20180099.jpg 201
|
||||||
|
13:34:59Z immich IMG_20180117.jpg 201
|
||||||
|
13:35:00Z immich IMG_20180135.jpg 201
|
||||||
|
13:35:00Z immich IMG_20180153.jpg 201
|
||||||
|
13:35:00Z immich IMG_20180171.jpg 201
|
||||||
|
13:35:00Z immich IMG_20180189.jpg 201
|
||||||
|
13:35:01Z immich IMG_20180207.jpg 201
|
||||||
|
13:35:01Z immich IMG_20180225.jpg 201
|
||||||
|
13:35:01Z immich IMG_20180243.jpg 201
|
||||||
|
13:35:02Z immich IMG_20180261.jpg 201
|
||||||
|
13:35:02Z immich IMG_20180279.jpg 201
|
||||||
|
13:35:02Z immich IMG_20180297.jpg 201
|
||||||
|
13:35:02Z immich IMG_20190010.jpg 201
|
||||||
|
13:35:03Z immich IMG_20190028.jpg 201
|
||||||
|
13:35:03Z immich IMG_20190046.jpg 201
|
||||||
|
13:35:03Z immich IMG_20190064.jpg 201
|
||||||
|
13:35:03Z immich IMG_20190082.jpg 201
|
||||||
|
13:35:03Z immich IMG_20190100.jpg 201
|
||||||
|
13:35:03Z immich IMG_20190118.jpg 201
|
||||||
|
13:35:03Z immich IMG_20190136.jpg 201
|
||||||
|
13:35:03Z immich IMG_20190154.jpg 201
|
||||||
|
13:35:04Z immich IMG_20190172.jpg 201
|
||||||
|
13:35:04Z immich IMG_20190190.jpg 201
|
||||||
|
13:35:04Z immich IMG_20190208.jpg 201
|
||||||
|
13:35:04Z immich IMG_20190226.jpg 201
|
||||||
|
13:35:05Z immich IMG_20190244.jpg 201
|
||||||
|
13:35:05Z immich IMG_20190262.jpg 201
|
||||||
|
13:35:05Z immich IMG_20190280.jpg 201
|
||||||
|
13:35:05Z immich IMG_20190298.jpg 201
|
||||||
|
13:35:05Z immich IMG_20200011.jpg 201
|
||||||
|
13:35:05Z immich IMG_20200029.jpg 201
|
||||||
|
13:35:06Z immich IMG_20200047.jpg 201
|
||||||
|
13:35:06Z immich IMG_20200065.jpg 201
|
||||||
|
13:35:06Z immich IMG_20200083.jpg 201
|
||||||
|
13:35:06Z immich IMG_20200101.jpg 201
|
||||||
|
13:35:06Z immich IMG_20200119.jpg 201
|
||||||
|
13:35:06Z immich IMG_20200137.jpg 201
|
||||||
|
13:35:07Z immich IMG_20200155.jpg 201
|
||||||
|
13:35:07Z immich IMG_20200173.jpg 201
|
||||||
|
13:35:07Z immich IMG_20200191.jpg 201
|
||||||
|
13:35:07Z immich IMG_20200209.jpg 201
|
||||||
|
13:35:07Z immich IMG_20200227.jpg 201
|
||||||
|
13:35:07Z immich IMG_20200245.jpg 201
|
||||||
|
13:35:08Z immich IMG_20200263.jpg 201
|
||||||
|
13:35:08Z immich IMG_20200281.jpg 201
|
||||||
|
13:35:08Z immich IMG_20200299.jpg 201
|
||||||
|
13:35:08Z immich IMG_20210012.jpg 201
|
||||||
|
13:35:08Z immich IMG_20210030.jpg 201
|
||||||
|
13:35:09Z immich IMG_20210048.jpg 201
|
||||||
|
13:35:09Z immich IMG_20210066.jpg 201
|
||||||
|
13:35:09Z immich IMG_20210084.jpg 201
|
||||||
|
13:35:09Z immich IMG_20210102.jpg 201
|
||||||
|
13:35:09Z immich IMG_20210120.jpg 201
|
||||||
|
13:35:09Z immich IMG_20210138.jpg 201
|
||||||
|
13:35:10Z immich IMG_20210156.jpg 201
|
||||||
|
13:35:10Z immich IMG_20210174.jpg 201
|
||||||
|
13:35:10Z immich IMG_20210192.jpg 201
|
||||||
|
13:35:11Z immich IMG_20210210.jpg 201
|
||||||
|
13:35:11Z immich IMG_20210228.jpg 201
|
||||||
|
13:35:11Z immich IMG_20210246.jpg 201
|
||||||
|
13:35:11Z immich IMG_20210264.jpg 201
|
||||||
|
13:35:11Z immich IMG_20210282.jpg 201
|
||||||
|
13:35:11Z immich IMG_20220013.jpg 201
|
||||||
|
13:35:11Z immich IMG_20220031.jpg 201
|
||||||
|
13:35:11Z immich IMG_20220049.jpg 201
|
||||||
|
13:35:11Z immich IMG_20220067.jpg 201
|
||||||
|
13:35:11Z immich IMG_20220085.jpg 201
|
||||||
|
13:35:12Z immich IMG_20220103.jpg 201
|
||||||
|
13:35:12Z immich IMG_20220121.jpg 201
|
||||||
|
13:35:13Z immich IMG_20220139.jpg 201
|
||||||
|
13:35:13Z immich IMG_20220157.jpg 201
|
||||||
|
13:35:13Z immich IMG_20220175.jpg 201
|
||||||
|
13:35:13Z immich IMG_20220193.jpg 201
|
||||||
|
13:35:13Z immich IMG_20220211.jpg 201
|
||||||
|
13:35:14Z immich IMG_20220229.jpg 201
|
||||||
|
13:35:14Z immich IMG_20220247.jpg 201
|
||||||
|
13:35:14Z immich IMG_20220265.jpg 201
|
||||||
|
13:35:15Z immich IMG_20220283.jpg 201
|
||||||
|
13:35:15Z immich IMG_20230014.jpg 201
|
||||||
|
13:35:16Z immich IMG_20230032.jpg 201
|
||||||
|
13:35:16Z immich IMG_20230050.jpg 201
|
||||||
|
13:35:16Z immich IMG_20230068.jpg 201
|
||||||
|
13:35:17Z immich IMG_20230086.jpg 201
|
||||||
|
13:35:17Z immich IMG_20230104.jpg 201
|
||||||
|
13:35:17Z immich IMG_20230122.jpg 201
|
||||||
|
13:35:18Z immich IMG_20230140.jpg 201
|
||||||
|
13:35:18Z immich IMG_20230158.jpg 201
|
||||||
|
13:35:18Z immich IMG_20230176.jpg 201
|
||||||
|
13:35:18Z immich IMG_20230194.jpg 201
|
||||||
|
13:35:19Z immich IMG_20230212.jpg 201
|
||||||
|
13:35:19Z immich IMG_20230230.jpg 201
|
||||||
|
13:35:19Z immich IMG_20230248.jpg 201
|
||||||
|
13:35:19Z immich IMG_20230266.jpg 201
|
||||||
|
13:35:19Z immich IMG_20230284.jpg 201
|
||||||
|
13:35:20Z immich IMG_20240015.jpg 201
|
||||||
|
13:35:20Z immich IMG_20240033.jpg 201
|
||||||
|
13:35:20Z immich IMG_20240051.jpg 201
|
||||||
|
13:35:20Z immich IMG_20240069.jpg 201
|
||||||
|
13:35:20Z immich IMG_20240087.jpg 201
|
||||||
|
13:35:20Z immich IMG_20240105.jpg 201
|
||||||
|
13:35:20Z immich IMG_20240123.jpg 201
|
||||||
|
13:35:21Z immich IMG_20240141.jpg 201
|
||||||
|
13:35:21Z immich IMG_20240159.jpg 201
|
||||||
|
13:35:22Z immich IMG_20240177.jpg 201
|
||||||
|
13:35:23Z immich IMG_20240195.jpg 201
|
||||||
|
13:35:23Z immich IMG_20240213.jpg 201
|
||||||
|
13:35:23Z immich IMG_20240231.jpg 201
|
||||||
|
13:35:24Z immich IMG_20240249.jpg 201
|
||||||
|
13:35:24Z immich IMG_20240267.jpg 201
|
||||||
|
13:35:25Z immich IMG_20240285.jpg 201
|
||||||
|
13:35:25Z immich IMG_20250016.jpg 201
|
||||||
|
13:35:25Z immich IMG_20250034.jpg 201
|
||||||
|
13:35:25Z immich IMG_20250052.jpg 201
|
||||||
|
13:35:25Z immich IMG_20250070.jpg 201
|
||||||
|
13:35:26Z immich IMG_20250088.jpg 201
|
||||||
|
13:35:26Z immich IMG_20250106.jpg 201
|
||||||
|
13:35:26Z immich IMG_20250124.jpg 201
|
||||||
|
13:35:27Z immich IMG_20250142.jpg 201
|
||||||
|
13:35:27Z immich IMG_20250160.jpg 201
|
||||||
|
13:35:27Z immich IMG_20250178.jpg 201
|
||||||
|
13:35:27Z immich IMG_20250196.jpg 201
|
||||||
|
13:35:27Z immich IMG_20250214.jpg 201
|
||||||
|
13:35:27Z immich IMG_20250232.jpg 201
|
||||||
|
13:35:27Z immich IMG_20250250.jpg 201
|
||||||
|
13:35:28Z immich IMG_20250268.jpg 201
|
||||||
|
13:35:28Z immich IMG_20250286.jpg 201
|
||||||
|
13:35:28Z immich IMG_20260017.jpg 201
|
||||||
|
13:35:28Z immich IMG_20260035.jpg 201
|
||||||
|
13:35:28Z immich IMG_20260053.jpg 201
|
||||||
|
13:35:28Z immich IMG_20260071.jpg 201
|
||||||
|
13:35:28Z immich IMG_20260089.jpg 201
|
||||||
|
13:35:29Z immich IMG_20260107.jpg 201
|
||||||
|
13:35:29Z immich IMG_20260125.jpg 201
|
||||||
|
13:35:29Z immich IMG_20260143.jpg 201
|
||||||
|
13:35:29Z immich IMG_20260161.jpg 201
|
||||||
|
13:35:29Z immich IMG_20260179.jpg 201
|
||||||
|
13:35:30Z immich IMG_20260197.jpg 201
|
||||||
|
13:35:30Z immich IMG_20260215.jpg 201
|
||||||
|
13:35:30Z immich IMG_20260233.jpg 201
|
||||||
|
13:35:30Z immich IMG_20260251.jpg 201
|
||||||
|
13:35:31Z immich IMG_20260269.jpg 201
|
||||||
|
13:35:31Z immich IMG_20260287.jpg 201
|
||||||
|
13:35:31Z immich IMG_20160007_masolat.jpg 200
|
||||||
|
13:35:31Z immich duplikatum_IMG_20160007.jpg 200
|
||||||
|
13:35:39Z immich nyaralas_2025.mp4 201
|
||||||
|
13:35:39Z immich rajz_Kata.png 201
|
||||||
|
13:35:39Z immich torott_kep.jpg 201
|
||||||
|
13:35:39Z immich ures.jpg 400
|
||||||
|
13:35:39Z immich DONE ok 304 bad 2
|
||||||
|
13:35:39Z LOAD END
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
13:26:05Z LOAD START
|
||||||
|
13:26:05Z immich admin-sign-up 401
|
||||||
|
13:26:05Z immich login 401
|
||||||
|
13:26:05Z nextcloud rclone START
|
||||||
|
13:26:06Z paperless jelszavas_kivonat.pdf 200
|
||||||
|
13:26:06Z paperless kezikonyv_100_oldal.pdf 200
|
||||||
|
13:26:06Z paperless szamla_2024_000.pdf 200
|
||||||
|
13:26:07Z paperless szamla_2024_003.pdf 200
|
||||||
|
13:26:07Z paperless szamla_2024_006.pdf 200
|
||||||
|
13:26:08Z paperless szamla_2024_009.pdf 200
|
||||||
|
13:26:08Z paperless szamla_2024_012.pdf 200
|
||||||
|
13:26:09Z paperless szamla_2024_015.pdf 200
|
||||||
|
13:26:09Z paperless szamla_2024_018.pdf 200
|
||||||
|
13:26:10Z paperless szamla_2024_021.pdf 200
|
||||||
|
13:26:11Z paperless szamla_2024_024.pdf 200
|
||||||
|
13:26:11Z paperless szamla_2024_027.pdf 200
|
||||||
|
13:26:12Z paperless szamla_2024_030.pdf 200
|
||||||
|
13:26:12Z paperless szamla_2024_033.pdf 200
|
||||||
|
13:26:12Z paperless szamla_2024_036.pdf 200
|
||||||
|
13:26:13Z paperless szamla_2024_039.pdf 200
|
||||||
|
13:26:14Z paperless szamla_2024_042.pdf 200
|
||||||
|
13:26:14Z paperless szamla_2024_045.pdf 200
|
||||||
|
13:26:15Z paperless szamla_2024_048.pdf 200
|
||||||
|
13:26:15Z paperless szamla_2025_001.pdf 200
|
||||||
|
13:26:16Z paperless szamla_2025_004.pdf 200
|
||||||
|
13:26:16Z paperless szamla_2025_007.pdf 200
|
||||||
|
13:26:17Z paperless szamla_2025_010.pdf 200
|
||||||
|
13:26:18Z paperless szamla_2025_013.pdf 200
|
||||||
|
13:26:18Z paperless szamla_2025_016.pdf 200
|
||||||
|
13:26:19Z paperless szamla_2025_019.pdf 200
|
||||||
|
13:26:19Z paperless szamla_2025_022.pdf 200
|
||||||
|
13:26:20Z paperless szamla_2025_025.pdf 200
|
||||||
|
13:26:21Z paperless szamla_2025_028.pdf 200
|
||||||
|
13:26:21Z paperless szamla_2025_031.pdf 200
|
||||||
|
13:26:22Z paperless szamla_2025_034.pdf 200
|
||||||
|
13:26:23Z paperless szamla_2025_037.pdf 200
|
||||||
|
13:26:23Z paperless szamla_2025_040.pdf 200
|
||||||
|
13:26:24Z paperless szamla_2025_043.pdf 200
|
||||||
|
13:26:25Z paperless szamla_2025_046.pdf 200
|
||||||
|
13:26:25Z paperless szamla_2025_049.pdf 200
|
||||||
|
13:26:25Z paperless szamla_2026_002.pdf 200
|
||||||
|
13:26:26Z paperless szamla_2026_005.pdf 200
|
||||||
|
13:26:26Z paperless szamla_2026_008.pdf 200
|
||||||
|
13:26:27Z paperless szamla_2026_011.pdf 200
|
||||||
|
13:26:27Z paperless szamla_2026_014.pdf 200
|
||||||
|
13:26:28Z paperless szamla_2026_017.pdf 200
|
||||||
|
13:26:29Z paperless szamla_2026_020.pdf 200
|
||||||
|
13:26:29Z paperless szamla_2026_023.pdf 200
|
||||||
|
13:26:30Z paperless szamla_2026_026.pdf 200
|
||||||
|
13:26:30Z paperless szamla_2026_029.pdf 200
|
||||||
|
13:26:31Z paperless szamla_2026_032.pdf 200
|
||||||
|
13:26:31Z paperless szamla_2026_035.pdf 200
|
||||||
|
13:26:32Z paperless szamla_2026_038.pdf 200
|
||||||
|
13:26:32Z paperless szamla_2026_041.pdf 200
|
||||||
|
13:26:33Z paperless szamla_2026_044.pdf 200
|
||||||
|
13:26:33Z paperless szamla_2026_047.pdf 200
|
||||||
|
13:26:34Z paperless szkennelt_nyilatkozat.pdf 200
|
||||||
|
13:26:34Z paperless DONE ok 53 bad 0
|
||||||
|
16:25:27Z nextcloud DONE rc 0
|
||||||
|
16:25:27Z LOAD END
|
||||||
@@ -0,0 +1,555 @@
|
|||||||
|
13:43:40Z PROOF START 120 min — each stream repeats on its own until the end
|
||||||
|
13:43:42Z c1 immich new user 201
|
||||||
|
13:43:47Z c1 paperless delete 52 200
|
||||||
|
13:43:48Z c1 paperless empty trash 200
|
||||||
|
13:44:47Z c1 immich DONE ok 305 bad 1
|
||||||
|
13:44:48Z c1 paperless DONE ok 53 bad 0
|
||||||
|
13:44:48Z c2 immich new user 201
|
||||||
|
13:44:50Z c2 paperless delete 3 200
|
||||||
|
13:44:51Z c2 paperless empty trash 200
|
||||||
|
13:45:55Z c2 immich DONE ok 305 bad 1
|
||||||
|
13:45:57Z c3 immich new user 201
|
||||||
|
13:46:01Z c2 paperless DONE ok 53 bad 0
|
||||||
|
13:46:06Z c3 paperless delete 5 200
|
||||||
|
13:46:07Z c3 paperless empty trash 200
|
||||||
|
13:47:01Z c3 immich DONE ok 305 bad 1
|
||||||
|
13:47:03Z c4 immich new user 201
|
||||||
|
13:47:05Z c3 paperless DONE ok 53 bad 0
|
||||||
|
13:47:06Z c4 paperless delete 5 200
|
||||||
|
13:47:08Z c4 paperless empty trash 200
|
||||||
|
13:48:04Z c4 immich DONE ok 305 bad 1
|
||||||
|
13:48:06Z c5 immich new user 201
|
||||||
|
13:48:06Z c4 paperless DONE ok 53 bad 0
|
||||||
|
13:48:08Z c5 paperless delete 4 200
|
||||||
|
13:48:10Z c5 paperless empty trash 200
|
||||||
|
13:49:06Z c5 immich DONE ok 305 bad 1
|
||||||
|
13:49:07Z c5 paperless DONE ok 53 bad 0
|
||||||
|
13:49:08Z c6 immich new user 201
|
||||||
|
13:49:12Z c6 paperless delete 4 200
|
||||||
|
13:49:14Z c6 paperless empty trash 200
|
||||||
|
13:50:17Z c6 paperless DONE ok 53 bad 0
|
||||||
|
13:50:30Z c7 paperless delete 5 200
|
||||||
|
13:50:31Z c6 immich DONE ok 305 bad 1
|
||||||
|
13:50:31Z c7 paperless empty trash 200
|
||||||
|
13:50:32Z c7 immich new user 201
|
||||||
|
13:51:26Z c7 paperless DONE ok 53 bad 0
|
||||||
|
13:51:28Z c8 paperless delete 5 200
|
||||||
|
13:51:29Z c8 paperless empty trash 200
|
||||||
|
13:51:33Z c7 immich DONE ok 305 bad 1
|
||||||
|
13:51:34Z c8 immich new user 201
|
||||||
|
13:52:29Z c8 paperless DONE ok 53 bad 0
|
||||||
|
13:52:31Z c9 paperless delete 4 200
|
||||||
|
13:52:31Z c9 paperless empty trash 200
|
||||||
|
13:52:32Z c8 immich DONE ok 305 bad 1
|
||||||
|
13:52:33Z c9 immich new user 201
|
||||||
|
13:53:24Z c9 paperless DONE ok 53 bad 0
|
||||||
|
13:53:26Z c10 paperless delete 4 200
|
||||||
|
13:53:28Z c10 paperless empty trash 200
|
||||||
|
13:53:29Z c9 immich DONE ok 305 bad 1
|
||||||
|
13:53:31Z c10 immich new user 201
|
||||||
|
13:54:21Z c10 paperless DONE ok 53 bad 0
|
||||||
|
13:54:24Z c11 paperless delete 3 200
|
||||||
|
13:54:26Z c11 paperless empty trash 200
|
||||||
|
13:54:32Z c10 immich DONE ok 305 bad 1
|
||||||
|
13:54:34Z c11 immich new user 201
|
||||||
|
13:55:18Z c11 paperless DONE ok 53 bad 0
|
||||||
|
13:55:20Z c12 paperless delete 4 200
|
||||||
|
13:55:22Z c12 paperless empty trash 200
|
||||||
|
13:55:31Z c11 immich DONE ok 305 bad 1
|
||||||
|
13:55:33Z c12 immich new user 201
|
||||||
|
13:56:23Z c12 paperless DONE ok 53 bad 0
|
||||||
|
13:56:25Z c13 paperless delete 5 200
|
||||||
|
13:56:26Z c13 paperless empty trash 200
|
||||||
|
13:56:31Z c12 immich DONE ok 305 bad 1
|
||||||
|
13:56:33Z c13 immich new user 201
|
||||||
|
13:57:23Z c13 paperless DONE ok 53 bad 0
|
||||||
|
13:57:26Z c14 paperless delete 3 200
|
||||||
|
13:57:27Z c14 paperless empty trash 200
|
||||||
|
13:57:30Z c13 immich DONE ok 305 bad 1
|
||||||
|
13:57:32Z c14 immich new user 201
|
||||||
|
13:58:26Z c14 paperless DONE ok 53 bad 0
|
||||||
|
13:58:30Z c15 paperless delete 3 200
|
||||||
|
13:58:32Z c14 immich DONE ok 305 bad 1
|
||||||
|
13:58:32Z c15 paperless empty trash 200
|
||||||
|
13:58:33Z c15 immich new user 201
|
||||||
|
13:59:32Z c15 immich DONE ok 305 bad 1
|
||||||
|
13:59:32Z c15 paperless DONE ok 53 bad 0
|
||||||
|
13:59:33Z c16 immich new user 201
|
||||||
|
13:59:34Z c16 paperless delete 6 200
|
||||||
|
13:59:35Z c16 paperless empty trash 200
|
||||||
|
14:00:48Z c16 immich DONE ok 305 bad 1
|
||||||
|
14:00:49Z c17 immich new user 201
|
||||||
|
14:00:50Z c16 paperless DONE ok 53 bad 0
|
||||||
|
14:00:52Z c17 paperless delete 5 200
|
||||||
|
14:00:53Z c17 paperless empty trash 200
|
||||||
|
14:01:47Z c17 immich DONE ok 305 bad 1
|
||||||
|
14:01:49Z c18 immich new user 201
|
||||||
|
14:01:56Z c17 paperless DONE ok 53 bad 0
|
||||||
|
14:01:58Z c18 paperless delete 4 200
|
||||||
|
14:01:58Z c18 paperless empty trash 200
|
||||||
|
14:02:52Z c18 paperless DONE ok 53 bad 0
|
||||||
|
14:02:53Z c18 immich DONE ok 305 bad 1
|
||||||
|
14:02:55Z c19 immich new user 201
|
||||||
|
14:02:55Z c19 paperless delete 4 200
|
||||||
|
14:02:56Z c19 paperless empty trash 200
|
||||||
|
14:03:50Z c19 paperless DONE ok 53 bad 0
|
||||||
|
14:03:53Z c20 paperless delete 4 200
|
||||||
|
14:03:54Z c20 paperless empty trash 200
|
||||||
|
14:03:55Z c19 immich DONE ok 305 bad 1
|
||||||
|
14:03:57Z c20 immich new user 201
|
||||||
|
14:04:53Z c20 paperless DONE ok 53 bad 0
|
||||||
|
14:04:55Z c21 paperless delete 5 200
|
||||||
|
14:04:56Z c21 paperless empty trash 200
|
||||||
|
14:05:02Z c20 immich DONE ok 305 bad 1
|
||||||
|
14:05:04Z c21 immich new user 201
|
||||||
|
14:05:55Z c21 paperless DONE ok 53 bad 0
|
||||||
|
14:05:58Z c22 paperless delete 4 200
|
||||||
|
14:06:00Z c22 paperless empty trash 200
|
||||||
|
14:06:10Z c21 immich DONE ok 305 bad 1
|
||||||
|
14:06:12Z c22 immich new user 201
|
||||||
|
14:07:05Z c22 paperless DONE ok 53 bad 0
|
||||||
|
14:07:07Z c23 paperless delete 5 200
|
||||||
|
14:07:08Z c23 paperless empty trash 200
|
||||||
|
14:07:14Z c22 immich DONE ok 305 bad 1
|
||||||
|
14:07:15Z c23 immich new user 201
|
||||||
|
14:08:05Z c23 paperless DONE ok 53 bad 0
|
||||||
|
14:08:08Z c24 paperless delete 4 200
|
||||||
|
14:08:10Z c24 paperless empty trash 200
|
||||||
|
14:08:12Z c23 immich DONE ok 305 bad 1
|
||||||
|
14:08:13Z c24 immich new user 201
|
||||||
|
14:09:08Z c24 immich DONE ok 305 bad 1
|
||||||
|
14:09:10Z c25 immich new user 201
|
||||||
|
14:09:12Z c24 paperless DONE ok 53 bad 0
|
||||||
|
14:09:14Z c25 paperless delete 6 200
|
||||||
|
14:09:15Z c25 paperless empty trash 200
|
||||||
|
14:10:14Z c25 immich DONE ok 305 bad 1
|
||||||
|
14:10:15Z c26 immich new user 201
|
||||||
|
14:10:18Z c25 paperless DONE ok 53 bad 0
|
||||||
|
14:10:22Z c26 paperless delete 3 200
|
||||||
|
14:10:24Z c26 paperless empty trash 200
|
||||||
|
14:11:11Z c26 immich DONE ok 305 bad 1
|
||||||
|
14:11:13Z c27 immich new user 201
|
||||||
|
14:11:21Z c26 paperless DONE ok 53 bad 0
|
||||||
|
14:11:25Z c27 paperless delete 4 200
|
||||||
|
14:11:27Z c27 paperless empty trash 200
|
||||||
|
14:12:17Z c27 immich DONE ok 305 bad 1
|
||||||
|
14:12:19Z c28 immich new user 201
|
||||||
|
14:12:29Z c27 paperless DONE ok 53 bad 0
|
||||||
|
14:12:33Z c28 paperless delete 5 200
|
||||||
|
14:12:34Z c28 paperless empty trash 200
|
||||||
|
14:13:16Z c28 immich DONE ok 305 bad 1
|
||||||
|
14:13:19Z c29 immich new user 201
|
||||||
|
14:13:31Z c28 paperless DONE ok 53 bad 0
|
||||||
|
14:13:34Z c29 paperless delete 4 200
|
||||||
|
14:13:35Z c29 paperless empty trash 200
|
||||||
|
14:14:14Z c29 immich DONE ok 305 bad 1
|
||||||
|
14:14:17Z c30 immich new user 201
|
||||||
|
14:14:43Z c29 paperless DONE ok 53 bad 0
|
||||||
|
14:14:46Z c30 paperless delete 5 200
|
||||||
|
14:14:47Z c30 paperless empty trash 200
|
||||||
|
14:15:26Z c30 immich DONE ok 305 bad 1
|
||||||
|
14:15:28Z c31 immich new user 201
|
||||||
|
14:15:48Z c30 paperless DONE ok 53 bad 0
|
||||||
|
14:15:50Z c31 paperless delete 6 200
|
||||||
|
14:15:50Z c31 paperless empty trash 200
|
||||||
|
14:16:27Z c31 immich DONE ok 305 bad 1
|
||||||
|
14:16:28Z c32 immich new user 201
|
||||||
|
14:16:53Z c31 paperless DONE ok 53 bad 0
|
||||||
|
14:16:56Z c32 paperless delete 4 200
|
||||||
|
14:16:57Z c32 paperless empty trash 200
|
||||||
|
14:17:27Z c32 immich DONE ok 305 bad 1
|
||||||
|
14:17:29Z c33 immich new user 201
|
||||||
|
14:17:58Z c32 paperless DONE ok 53 bad 0
|
||||||
|
14:18:01Z c33 paperless delete 5 200
|
||||||
|
14:18:03Z c33 paperless empty trash 200
|
||||||
|
14:18:25Z c33 immich DONE ok 305 bad 1
|
||||||
|
14:18:26Z c34 immich new user 201
|
||||||
|
14:19:00Z c33 paperless DONE ok 53 bad 0
|
||||||
|
14:19:04Z c34 paperless delete 4 200
|
||||||
|
14:19:05Z c34 paperless empty trash 200
|
||||||
|
14:19:24Z c34 immich DONE ok 305 bad 1
|
||||||
|
14:19:25Z c35 immich new user 201
|
||||||
|
14:20:07Z c34 paperless DONE ok 53 bad 0
|
||||||
|
14:20:10Z c35 paperless delete 5 200
|
||||||
|
14:20:11Z c35 paperless empty trash 200
|
||||||
|
14:20:21Z c35 immich DONE ok 305 bad 1
|
||||||
|
14:20:24Z c36 immich new user 201
|
||||||
|
14:21:15Z c35 paperless DONE ok 53 bad 0
|
||||||
|
14:21:21Z c36 paperless delete 5 200
|
||||||
|
14:21:22Z c36 paperless empty trash 200
|
||||||
|
14:21:32Z c36 immich DONE ok 305 bad 1
|
||||||
|
14:21:35Z c37 immich new user 201
|
||||||
|
14:22:22Z c36 paperless DONE ok 53 bad 0
|
||||||
|
14:22:26Z c37 paperless delete 4 200
|
||||||
|
14:22:29Z c37 paperless empty trash 200
|
||||||
|
14:22:31Z c37 immich DONE ok 305 bad 1
|
||||||
|
14:22:34Z c38 immich new user 201
|
||||||
|
14:23:28Z c37 paperless DONE ok 53 bad 0
|
||||||
|
14:23:32Z c38 paperless delete 3 200
|
||||||
|
14:23:33Z c38 immich DONE ok 305 bad 1
|
||||||
|
14:23:33Z c38 paperless empty trash 200
|
||||||
|
14:23:35Z c39 immich new user 201
|
||||||
|
14:24:26Z c39 immich DONE ok 305 bad 1
|
||||||
|
14:24:28Z c40 immich new user 201
|
||||||
|
14:24:32Z c38 paperless DONE ok 53 bad 0
|
||||||
|
14:24:35Z c39 paperless delete 5 200
|
||||||
|
14:24:36Z c39 paperless empty trash 200
|
||||||
|
14:25:34Z c40 immich DONE ok 305 bad 1
|
||||||
|
14:25:35Z c41 immich new user 201
|
||||||
|
14:25:37Z c39 paperless DONE ok 53 bad 0
|
||||||
|
14:25:39Z c40 paperless delete 5 200
|
||||||
|
14:25:40Z c40 paperless empty trash 200
|
||||||
|
14:26:39Z c40 paperless DONE ok 53 bad 0
|
||||||
|
14:26:41Z c41 immich DONE ok 305 bad 1
|
||||||
|
14:26:41Z c41 paperless delete 4 200
|
||||||
|
14:26:43Z c41 paperless empty trash 200
|
||||||
|
14:26:44Z c42 immich new user 201
|
||||||
|
14:27:41Z c41 paperless DONE ok 53 bad 0
|
||||||
|
14:27:43Z c42 paperless delete 4 200
|
||||||
|
14:27:46Z c42 paperless empty trash 200
|
||||||
|
14:27:47Z c42 immich DONE ok 305 bad 1
|
||||||
|
14:27:48Z c43 immich new user 201
|
||||||
|
14:28:41Z c43 immich DONE ok 305 bad 1
|
||||||
|
14:28:42Z c44 immich new user 201
|
||||||
|
14:28:42Z c42 paperless DONE ok 53 bad 0
|
||||||
|
14:28:45Z c43 paperless delete 4 200
|
||||||
|
14:28:47Z c43 paperless empty trash 200
|
||||||
|
14:29:39Z c44 immich DONE ok 305 bad 1
|
||||||
|
14:29:40Z c45 immich new user 201
|
||||||
|
14:29:49Z c43 paperless DONE ok 53 bad 0
|
||||||
|
14:29:52Z c44 paperless delete 5 200
|
||||||
|
14:29:54Z c44 paperless empty trash 200
|
||||||
|
14:30:55Z c45 immich DONE ok 305 bad 1
|
||||||
|
14:30:57Z c46 immich new user 201
|
||||||
|
14:30:58Z c44 paperless DONE ok 53 bad 0
|
||||||
|
14:31:02Z c45 paperless delete 4 200
|
||||||
|
14:31:03Z c45 paperless empty trash 200
|
||||||
|
14:32:02Z c45 paperless DONE ok 53 bad 0
|
||||||
|
14:32:07Z c46 paperless delete 4 200
|
||||||
|
14:32:09Z c46 paperless empty trash 200
|
||||||
|
14:32:09Z c46 immich DONE ok 305 bad 1
|
||||||
|
14:32:11Z c47 immich new user 201
|
||||||
|
14:33:13Z c46 paperless DONE ok 53 bad 0
|
||||||
|
14:33:15Z c47 paperless delete 6 200
|
||||||
|
14:33:17Z c47 paperless empty trash 200
|
||||||
|
14:33:29Z c47 immich DONE ok 305 bad 1
|
||||||
|
14:33:31Z c48 immich new user 201
|
||||||
|
14:34:18Z c47 paperless DONE ok 53 bad 0
|
||||||
|
14:34:21Z c48 paperless delete 4 200
|
||||||
|
14:34:22Z c48 paperless empty trash 200
|
||||||
|
14:34:43Z c48 immich DONE ok 305 bad 1
|
||||||
|
14:34:45Z c49 immich new user 201
|
||||||
|
14:35:34Z c48 paperless DONE ok 53 bad 0
|
||||||
|
14:35:38Z c49 paperless delete 5 200
|
||||||
|
14:35:40Z c49 paperless empty trash 200
|
||||||
|
14:35:48Z c49 immich DONE ok 305 bad 1
|
||||||
|
14:35:51Z c50 immich new user 201
|
||||||
|
14:36:50Z c49 paperless DONE ok 53 bad 0
|
||||||
|
14:36:56Z c50 paperless delete 4 200
|
||||||
|
14:36:58Z c50 paperless empty trash 200
|
||||||
|
14:37:01Z c50 immich DONE ok 305 bad 1
|
||||||
|
14:37:03Z c51 immich new user 201
|
||||||
|
14:38:01Z c51 immich DONE ok 305 bad 1
|
||||||
|
14:38:02Z c50 paperless DONE ok 53 bad 0
|
||||||
|
14:38:03Z c52 immich new user 201
|
||||||
|
14:38:08Z c51 paperless delete 5 200
|
||||||
|
14:38:09Z c51 paperless empty trash 200
|
||||||
|
14:39:11Z c51 paperless DONE ok 53 bad 0
|
||||||
|
14:39:12Z c52 paperless delete 5 200
|
||||||
|
14:39:13Z c52 immich DONE ok 305 bad 1
|
||||||
|
14:39:13Z c52 paperless empty trash 200
|
||||||
|
14:39:14Z c53 immich new user 201
|
||||||
|
14:40:14Z c53 immich DONE ok 305 bad 1
|
||||||
|
14:40:16Z c54 immich new user 201
|
||||||
|
14:40:17Z c52 paperless DONE ok 53 bad 0
|
||||||
|
14:40:19Z c53 paperless delete 5 200
|
||||||
|
14:40:21Z c53 paperless empty trash 200
|
||||||
|
14:41:15Z c53 paperless DONE ok 53 bad 0
|
||||||
|
14:41:19Z c54 paperless delete 3 200
|
||||||
|
14:41:20Z c54 paperless empty trash 200
|
||||||
|
14:41:28Z c54 immich DONE ok 305 bad 1
|
||||||
|
14:41:30Z c55 immich new user 201
|
||||||
|
14:42:23Z c54 paperless DONE ok 53 bad 0
|
||||||
|
14:42:28Z c55 paperless delete 5 200
|
||||||
|
14:42:29Z c55 paperless empty trash 200
|
||||||
|
14:42:38Z c55 immich DONE ok 305 bad 1
|
||||||
|
14:42:39Z c56 immich new user 201
|
||||||
|
14:43:31Z c55 paperless DONE ok 53 bad 0
|
||||||
|
14:43:34Z c56 paperless delete 5 200
|
||||||
|
14:43:36Z c56 paperless empty trash 200
|
||||||
|
14:43:37Z c56 immich DONE ok 305 bad 1
|
||||||
|
14:43:40Z c57 immich new user 201
|
||||||
|
14:44:36Z c56 paperless DONE ok 53 bad 0
|
||||||
|
14:44:38Z c57 paperless delete 5 200
|
||||||
|
14:44:39Z c57 paperless empty trash 200
|
||||||
|
14:44:50Z c57 immich DONE ok 305 bad 1
|
||||||
|
14:44:51Z c58 immich new user 201
|
||||||
|
14:45:44Z c57 paperless DONE ok 53 bad 0
|
||||||
|
14:45:47Z c58 immich DONE ok 305 bad 1
|
||||||
|
14:45:48Z c58 paperless delete 5 200
|
||||||
|
14:45:49Z c59 immich new user 201
|
||||||
|
14:45:49Z c58 paperless empty trash 200
|
||||||
|
14:46:49Z c58 paperless DONE ok 53 bad 0
|
||||||
|
14:46:53Z c59 paperless delete 4 200
|
||||||
|
14:46:57Z c59 paperless empty trash 200
|
||||||
|
14:46:59Z c59 immich DONE ok 305 bad 1
|
||||||
|
14:47:01Z c60 immich new user 201
|
||||||
|
14:47:58Z c59 paperless DONE ok 53 bad 0
|
||||||
|
14:47:59Z c60 paperless delete 5 200
|
||||||
|
14:48:00Z c60 paperless empty trash 200
|
||||||
|
14:48:01Z c60 immich DONE ok 305 bad 1
|
||||||
|
14:48:02Z c61 immich new user 201
|
||||||
|
14:48:57Z c60 paperless DONE ok 53 bad 0
|
||||||
|
14:49:01Z c61 paperless delete 3 200
|
||||||
|
14:49:05Z c61 paperless empty trash 200
|
||||||
|
14:49:06Z c61 immich DONE ok 305 bad 1
|
||||||
|
14:49:09Z c62 immich new user 201
|
||||||
|
14:50:07Z c61 paperless DONE ok 53 bad 0
|
||||||
|
14:50:10Z c62 paperless delete 4 200
|
||||||
|
14:50:11Z c62 paperless empty trash 200
|
||||||
|
14:50:15Z c62 immich DONE ok 305 bad 1
|
||||||
|
14:50:17Z c63 immich new user 201
|
||||||
|
14:51:13Z c62 paperless DONE ok 53 bad 0
|
||||||
|
14:51:16Z c63 paperless delete 4 200
|
||||||
|
14:51:17Z c63 paperless empty trash 200
|
||||||
|
14:51:27Z c63 immich DONE ok 305 bad 1
|
||||||
|
14:51:30Z c64 immich new user 201
|
||||||
|
14:52:18Z c63 paperless DONE ok 53 bad 0
|
||||||
|
14:52:20Z c64 paperless delete 5 200
|
||||||
|
14:52:21Z c64 paperless empty trash 200
|
||||||
|
14:52:45Z c64 immich DONE ok 305 bad 1
|
||||||
|
14:52:47Z c65 immich new user 201
|
||||||
|
14:53:19Z c64 paperless DONE ok 53 bad 0
|
||||||
|
14:53:23Z c65 paperless delete 4 200
|
||||||
|
14:53:25Z c65 paperless empty trash 200
|
||||||
|
14:53:41Z c65 immich DONE ok 305 bad 1
|
||||||
|
14:53:43Z c66 immich new user 201
|
||||||
|
14:54:25Z c65 paperless DONE ok 53 bad 0
|
||||||
|
14:54:27Z c66 paperless delete 4 200
|
||||||
|
14:54:28Z c66 paperless empty trash 200
|
||||||
|
14:54:44Z c66 immich DONE ok 305 bad 1
|
||||||
|
14:54:46Z c67 immich new user 201
|
||||||
|
14:55:36Z c66 paperless DONE ok 53 bad 0
|
||||||
|
14:55:40Z c67 paperless delete 5 200
|
||||||
|
14:55:42Z c67 paperless empty trash 200
|
||||||
|
14:55:42Z c67 immich DONE ok 305 bad 1
|
||||||
|
14:55:45Z c68 immich new user 201
|
||||||
|
14:56:42Z c67 paperless DONE ok 53 bad 0
|
||||||
|
14:56:44Z c68 paperless delete 5 200
|
||||||
|
14:56:44Z c68 paperless empty trash 200
|
||||||
|
14:56:52Z c68 immich DONE ok 305 bad 1
|
||||||
|
14:56:54Z c69 immich new user 201
|
||||||
|
14:57:43Z c68 paperless DONE ok 53 bad 0
|
||||||
|
14:57:46Z c69 paperless delete 4 200
|
||||||
|
14:57:47Z c69 paperless empty trash 200
|
||||||
|
14:58:00Z c69 immich DONE ok 305 bad 1
|
||||||
|
14:58:02Z c70 immich new user 201
|
||||||
|
14:58:45Z c69 paperless DONE ok 53 bad 0
|
||||||
|
14:58:47Z c70 paperless delete 4 200
|
||||||
|
14:58:48Z c70 paperless empty trash 200
|
||||||
|
14:59:12Z c70 immich DONE ok 305 bad 1
|
||||||
|
14:59:13Z c71 immich new user 201
|
||||||
|
14:59:56Z c70 paperless DONE ok 53 bad 0
|
||||||
|
14:59:58Z c71 paperless delete 5 200
|
||||||
|
14:59:59Z c71 paperless empty trash 200
|
||||||
|
15:00:13Z c71 immich DONE ok 305 bad 1
|
||||||
|
15:00:15Z c72 immich new user 201
|
||||||
|
15:01:03Z c71 paperless DONE ok 53 bad 0
|
||||||
|
15:01:07Z c72 paperless delete 5 200
|
||||||
|
15:01:08Z c72 paperless empty trash 200
|
||||||
|
15:01:23Z c72 immich DONE ok 305 bad 1
|
||||||
|
15:01:25Z c73 immich new user 201
|
||||||
|
15:02:16Z c72 paperless DONE ok 53 bad 0
|
||||||
|
15:02:18Z c73 paperless delete 4 200
|
||||||
|
15:02:19Z c73 paperless empty trash 200
|
||||||
|
15:02:29Z c73 immich DONE ok 305 bad 1
|
||||||
|
15:02:31Z c74 immich new user 201
|
||||||
|
15:03:22Z c73 paperless DONE ok 53 bad 0
|
||||||
|
15:03:25Z c74 paperless delete 3 200
|
||||||
|
15:03:26Z c74 paperless empty trash 200
|
||||||
|
15:03:33Z c74 immich DONE ok 305 bad 1
|
||||||
|
15:03:35Z c75 immich new user 201
|
||||||
|
15:04:21Z c74 paperless DONE ok 53 bad 0
|
||||||
|
15:04:25Z c75 paperless delete 4 200
|
||||||
|
15:04:28Z c75 paperless empty trash 200
|
||||||
|
15:04:31Z c75 immich DONE ok 305 bad 1
|
||||||
|
15:04:32Z c76 immich new user 201
|
||||||
|
15:05:35Z c75 paperless DONE ok 53 bad 0
|
||||||
|
15:05:37Z c76 immich DONE ok 305 bad 1
|
||||||
|
15:05:39Z c76 paperless delete 5 200
|
||||||
|
15:05:39Z c77 immich new user 201
|
||||||
|
15:05:40Z c76 paperless empty trash 200
|
||||||
|
15:06:43Z c76 paperless DONE ok 53 bad 0
|
||||||
|
15:06:46Z c77 paperless delete 5 200
|
||||||
|
15:06:47Z c77 paperless empty trash 200
|
||||||
|
15:06:53Z c77 immich DONE ok 305 bad 1
|
||||||
|
15:06:55Z c78 immich new user 201
|
||||||
|
15:07:49Z c77 paperless DONE ok 53 bad 0
|
||||||
|
15:07:50Z c78 paperless delete 4 200
|
||||||
|
15:07:51Z c78 paperless empty trash 200
|
||||||
|
15:07:58Z c78 immich DONE ok 305 bad 1
|
||||||
|
15:08:00Z c79 immich new user 201
|
||||||
|
15:08:54Z c78 paperless DONE ok 53 bad 0
|
||||||
|
15:08:58Z c79 paperless delete 4 200
|
||||||
|
15:08:59Z c79 paperless empty trash 200
|
||||||
|
15:09:09Z c79 immich DONE ok 305 bad 1
|
||||||
|
15:09:11Z c80 immich new user 201
|
||||||
|
15:10:00Z c79 paperless DONE ok 53 bad 0
|
||||||
|
15:10:03Z c80 paperless delete 5 200
|
||||||
|
15:10:06Z c80 paperless empty trash 200
|
||||||
|
15:10:15Z c80 immich DONE ok 305 bad 1
|
||||||
|
15:10:16Z c81 immich new user 201
|
||||||
|
15:11:08Z c80 paperless DONE ok 53 bad 0
|
||||||
|
15:11:11Z c81 paperless delete 5 200
|
||||||
|
15:11:15Z c81 paperless empty trash 200
|
||||||
|
15:11:15Z c81 immich DONE ok 305 bad 1
|
||||||
|
15:11:18Z c82 immich new user 201
|
||||||
|
15:12:18Z c81 paperless DONE ok 53 bad 0
|
||||||
|
15:12:20Z c82 paperless delete 5 200
|
||||||
|
15:12:21Z c82 paperless empty trash 200
|
||||||
|
15:12:27Z c82 immich DONE ok 305 bad 1
|
||||||
|
15:12:28Z c83 immich new user 201
|
||||||
|
15:13:19Z c82 paperless DONE ok 53 bad 0
|
||||||
|
15:13:21Z c83 paperless delete 4 200
|
||||||
|
15:13:22Z c83 paperless empty trash 200
|
||||||
|
15:13:35Z c83 immich DONE ok 305 bad 1
|
||||||
|
15:13:37Z c84 immich new user 201
|
||||||
|
15:14:20Z c83 paperless DONE ok 53 bad 0
|
||||||
|
15:14:22Z c84 paperless delete 4 200
|
||||||
|
15:14:23Z c84 paperless empty trash 200
|
||||||
|
15:14:41Z c84 immich DONE ok 305 bad 1
|
||||||
|
15:14:42Z c85 immich new user 201
|
||||||
|
15:15:25Z c84 paperless DONE ok 53 bad 0
|
||||||
|
15:15:28Z c85 paperless delete 4 200
|
||||||
|
15:15:29Z c85 paperless empty trash 200
|
||||||
|
15:15:44Z c85 immich DONE ok 305 bad 1
|
||||||
|
15:15:46Z c86 immich new user 201
|
||||||
|
15:16:31Z c85 paperless DONE ok 53 bad 0
|
||||||
|
15:16:34Z c86 paperless delete 3 200
|
||||||
|
15:16:36Z c86 paperless empty trash 200
|
||||||
|
15:16:51Z c86 immich DONE ok 305 bad 1
|
||||||
|
15:16:54Z c87 immich new user 201
|
||||||
|
15:17:38Z c86 paperless DONE ok 53 bad 0
|
||||||
|
15:17:41Z c87 paperless delete 4 200
|
||||||
|
15:17:42Z c87 paperless empty trash 200
|
||||||
|
15:18:01Z c87 immich DONE ok 305 bad 1
|
||||||
|
15:18:02Z c88 immich new user 201
|
||||||
|
15:18:42Z c87 paperless DONE ok 53 bad 0
|
||||||
|
15:18:45Z c88 paperless delete 4 200
|
||||||
|
15:18:46Z c88 paperless empty trash 200
|
||||||
|
15:19:01Z c88 immich DONE ok 305 bad 1
|
||||||
|
15:19:03Z c89 immich new user 201
|
||||||
|
15:19:54Z c88 paperless DONE ok 53 bad 0
|
||||||
|
15:19:57Z c89 paperless delete 5 200
|
||||||
|
15:19:59Z c89 paperless empty trash 200
|
||||||
|
15:20:04Z c89 immich DONE ok 305 bad 1
|
||||||
|
15:20:06Z c90 immich new user 201
|
||||||
|
15:20:59Z c89 paperless DONE ok 53 bad 0
|
||||||
|
15:21:02Z c90 paperless delete 4 200
|
||||||
|
15:21:04Z c90 paperless empty trash 200
|
||||||
|
15:21:05Z c90 immich DONE ok 305 bad 1
|
||||||
|
15:21:08Z c91 immich new user 201
|
||||||
|
15:22:09Z c90 paperless DONE ok 53 bad 0
|
||||||
|
15:22:12Z c91 paperless delete 5 200
|
||||||
|
15:22:12Z c91 immich DONE ok 305 bad 1
|
||||||
|
15:22:13Z c91 paperless empty trash 200
|
||||||
|
15:22:14Z c92 immich new user 201
|
||||||
|
15:23:10Z c91 paperless DONE ok 53 bad 0
|
||||||
|
15:23:14Z c92 paperless delete 4 200
|
||||||
|
15:23:15Z c92 immich DONE ok 305 bad 1
|
||||||
|
15:23:15Z c92 paperless empty trash 200
|
||||||
|
15:23:17Z c93 immich new user 201
|
||||||
|
15:24:14Z c93 immich DONE ok 305 bad 1
|
||||||
|
15:24:15Z c92 paperless DONE ok 53 bad 0
|
||||||
|
15:24:16Z c94 immich new user 201
|
||||||
|
15:24:19Z c93 paperless delete 4 200
|
||||||
|
15:24:20Z c93 paperless empty trash 200
|
||||||
|
15:25:16Z c94 immich DONE ok 305 bad 1
|
||||||
|
15:25:18Z c95 immich new user 201
|
||||||
|
15:25:20Z c93 paperless DONE ok 53 bad 0
|
||||||
|
15:25:23Z c94 paperless delete 4 200
|
||||||
|
15:25:24Z c94 paperless empty trash 200
|
||||||
|
15:26:17Z c95 immich DONE ok 305 bad 1
|
||||||
|
15:26:20Z c96 immich new user 201
|
||||||
|
15:26:29Z c94 paperless DONE ok 53 bad 0
|
||||||
|
15:26:33Z c95 paperless delete 5 200
|
||||||
|
15:26:34Z c95 paperless empty trash 200
|
||||||
|
15:27:35Z c95 paperless DONE ok 53 bad 0
|
||||||
|
15:27:38Z c96 paperless delete 4 200
|
||||||
|
15:27:40Z c96 paperless empty trash 200
|
||||||
|
15:27:40Z c96 immich DONE ok 305 bad 1
|
||||||
|
15:27:41Z c97 immich new user 201
|
||||||
|
15:28:39Z c97 immich DONE ok 305 bad 1
|
||||||
|
15:28:40Z c98 immich new user 201
|
||||||
|
15:28:43Z c96 paperless DONE ok 53 bad 0
|
||||||
|
15:28:47Z c97 paperless delete 4 200
|
||||||
|
15:28:48Z c97 paperless empty trash 200
|
||||||
|
15:29:49Z c97 paperless DONE ok 53 bad 0
|
||||||
|
15:29:49Z c98 immich DONE ok 305 bad 1
|
||||||
|
15:29:52Z c99 immich new user 201
|
||||||
|
15:29:53Z c98 paperless delete 3 200
|
||||||
|
15:29:54Z c98 paperless empty trash 200
|
||||||
|
15:30:56Z c99 immich DONE ok 305 bad 1
|
||||||
|
15:30:58Z c100 immich new user 201
|
||||||
|
15:31:02Z c98 paperless DONE ok 53 bad 0
|
||||||
|
15:31:06Z c99 paperless delete 4 200
|
||||||
|
15:31:08Z c99 paperless empty trash 200
|
||||||
|
15:32:03Z c100 immich DONE ok 305 bad 1
|
||||||
|
15:32:05Z c101 immich new user 201
|
||||||
|
15:32:12Z c99 paperless DONE ok 53 bad 0
|
||||||
|
15:32:16Z c100 paperless delete 4 200
|
||||||
|
15:32:17Z c100 paperless empty trash 200
|
||||||
|
15:33:06Z c101 immich DONE ok 305 bad 1
|
||||||
|
15:33:07Z c102 immich new user 201
|
||||||
|
15:33:12Z c100 paperless DONE ok 53 bad 0
|
||||||
|
15:33:14Z c101 paperless delete 4 200
|
||||||
|
15:33:15Z c101 paperless empty trash 200
|
||||||
|
15:34:10Z c101 paperless DONE ok 53 bad 0
|
||||||
|
15:34:16Z c102 paperless delete 4 200
|
||||||
|
15:34:17Z c102 immich DONE ok 305 bad 1
|
||||||
|
15:34:17Z c102 paperless empty trash 200
|
||||||
|
15:34:19Z c103 immich new user 201
|
||||||
|
15:35:18Z c102 paperless DONE ok 53 bad 0
|
||||||
|
15:35:23Z c103 paperless delete 4 200
|
||||||
|
15:35:24Z c103 paperless empty trash 200
|
||||||
|
15:35:38Z c103 immich DONE ok 305 bad 1
|
||||||
|
15:35:40Z c104 immich new user 201
|
||||||
|
15:36:27Z c103 paperless DONE ok 53 bad 0
|
||||||
|
15:36:30Z c104 paperless delete 4 200
|
||||||
|
15:36:31Z c104 paperless empty trash 200
|
||||||
|
15:37:03Z c104 immich DONE ok 305 bad 1
|
||||||
|
15:37:05Z c105 immich new user 201
|
||||||
|
15:37:41Z c104 paperless DONE ok 53 bad 0
|
||||||
|
15:37:45Z c105 paperless delete 5 200
|
||||||
|
15:37:45Z c105 paperless empty trash 200
|
||||||
|
15:38:10Z c105 immich DONE ok 305 bad 1
|
||||||
|
15:38:11Z c106 immich new user 201
|
||||||
|
15:38:46Z c105 paperless DONE ok 53 bad 0
|
||||||
|
15:38:49Z c106 paperless delete 4 200
|
||||||
|
15:38:50Z c106 paperless empty trash 200
|
||||||
|
15:39:14Z c106 immich DONE ok 305 bad 1
|
||||||
|
15:39:15Z c107 immich new user 201
|
||||||
|
15:39:47Z c106 paperless DONE ok 53 bad 0
|
||||||
|
15:39:51Z c107 paperless delete 4 200
|
||||||
|
15:39:52Z c107 paperless empty trash 200
|
||||||
|
15:40:25Z c107 immich DONE ok 305 bad 1
|
||||||
|
15:40:26Z c108 immich new user 201
|
||||||
|
15:40:53Z c107 paperless DONE ok 53 bad 0
|
||||||
|
15:40:57Z c108 paperless delete 5 200
|
||||||
|
15:40:58Z c108 paperless empty trash 200
|
||||||
|
15:41:28Z c108 immich DONE ok 305 bad 1
|
||||||
|
15:41:30Z c109 immich new user 201
|
||||||
|
15:42:01Z c108 paperless DONE ok 53 bad 0
|
||||||
|
15:42:05Z c109 paperless delete 4 200
|
||||||
|
15:42:06Z c109 paperless empty trash 200
|
||||||
|
15:42:32Z c109 immich DONE ok 305 bad 1
|
||||||
|
15:42:34Z c110 immich new user 201
|
||||||
|
15:43:08Z c109 paperless DONE ok 53 bad 0
|
||||||
|
15:43:10Z c110 paperless delete 4 200
|
||||||
|
15:43:11Z c110 paperless empty trash 200
|
||||||
|
15:43:35Z c110 immich DONE ok 305 bad 1
|
||||||
|
15:43:38Z c111 immich new user 201
|
||||||
|
15:44:15Z c110 paperless DONE ok 53 bad 0
|
||||||
|
15:44:15Z paperless cycles 110
|
||||||
|
15:44:38Z c111 immich DONE ok 305 bad 1
|
||||||
|
15:44:38Z immich cycles 111
|
||||||