Night 2026-10-19: register 161 -> 161 (closed R-921, R-953, R-986; opened R-1168..R-1170; R-1001/R-1011/R-1010/R-1070/R-1167 READY, R-893 READY with the measured design note -> D20, R-1083 gap); morning list in STATUS; 03/05/07/11 updated; R-893 evidence
gates / gates (push) Successful in 13m5s
gates / gates (push) Successful in 13m5s
This commit is contained in:
+12
@@ -16,6 +16,18 @@
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
|
||||
> **2026-10-19 (night) — controller 0.335.0 is the floor (MinAgent 0.158.1, 18:39 UTC); hub 0.162.0 BUILT, not deployed;
|
||||
> agent `56aabc7` and installer 1.35.0 committed, not released/published.** R-921 (operator answer „add one field"): the
|
||||
> agent serves its host-wide heavy-operation gate as `host_busy` on `GET /backup/status`; the controller reads it before
|
||||
> any stop (held → contention, no stop). R-1070's cause: Docker 29 prints „error: no such object" in lower case, so a
|
||||
> guest without cloudflared read `unknown` and failed the host OS-update rule; new state `absent` (passes when absent
|
||||
> before the step too); `11` §8.2 had claimed `unknown` does not fail — corrected. R-1010: the janitor's forced snapshot
|
||||
> delete is the API's `force=1`. Hub 0.162.0: R-1001 (restart seeding from the DB), R-1011 (exact-file sha). R-953 is
|
||||
> gone: ArgoCD tracks none of the three Secrets now. **R-986: `scripts/security_scan_gate.py`, shared, in every repo's
|
||||
> fast set and CI, WARN-FIRST**; govulncheck pin v1.1.4 → v1.8.0 (v1.1.4 cannot read Go 1.27); CI has no Go, so
|
||||
> govulncheck runs only in the hook; a depth-1 CI clone scans the whole tree (the root commit), all accepted. R-893
|
||||
> measured on 9202 (`audits/night-2026-10-19/P3/`) → D20. 9202 is on DHCP (.121 now).
|
||||
|
||||
> **2026-10-19 (late afternoon) — controller 0.334.0 is the floor (MinAgent 0.158.1, 16:31 local); hub 0.161.0 DEPLOYED;
|
||||
> tenantsync v1.3.0 on ep0 (operator yes).** R-1159 (D19 „box key first, then lock"): the passphrase is used AFTER enroll
|
||||
> (installer `/config` for its git login, the agent bakes it into `bootstrap.json`, the guest's first-boot pull, and —
|
||||
|
||||
@@ -0,0 +1,146 @@
|
||||
# REPORT — evening/night brief 2026-10-19 (one stop less per night, no false tunnel error, R-893 measured, hub and agent fixes ready, a scanner on every push, ep0's first weekly clean-up)
|
||||
|
||||
## In plain words
|
||||
|
||||
- **Morning list:** deploy hub 0.162.0? · release agent 0.161.0? · publish installer 1.35.0? · decide D20 (R-893).
|
||||
- **One app stop less per night** (R-921): the controller (0.335.0, the floor) now asks the agent „are you busy?" before
|
||||
it stops the apps. The agent's answer is written but not released, so it starts working with the agent release.
|
||||
- **No false error for a box without a tunnel** (R-1070): the real cause was Docker 29 writing its „not found" message
|
||||
in small letters. Fixed in the agent (not released).
|
||||
- **A failed off-site restore, measured** (R-893): the box puts the newer rows over the backup's older volume; a table
|
||||
only the backup had stays. The app is held stopped. Two options written; pick A (save the app's volumes first).
|
||||
- **Hub fixes built, not installed:** no repeat mail after a restart; the agent list offers only real agent packages.
|
||||
The ArgoCD danger to three hub secrets is already gone (ArgoCD no longer tracks them).
|
||||
- **Installer message written, not published:** a re-run on a live box says „delete the old host first", EN and HU.
|
||||
- **A scanner on every push, in all four repos** (warn-first). First scan: no real secret, no called weakness.
|
||||
- **The scratch box cannot get its agent key without an agent change** (R-1083): the exact gap is written down.
|
||||
- **ep0's first weekly clean-up:** read after 05:30 — see Part 10.
|
||||
|
||||
Decisions CC took: none beyond the brief. The operator answered one question (R-921, 19:2x): „add one field".
|
||||
|
||||
## Part 1 — R-921 (`07` §6.4; `03` local API) — DONE (live once the agent ships)
|
||||
|
||||
- No existing agent answer served the host-wide gate (read: `/backup/due`, `/backup/status`, the 409 shape). Operator:
|
||||
add a read-only field to the existing route, no new route.
|
||||
- Agent `56aabc7`: `host_busy` `{busy, what}` on `GET /backup/status`. Controller 0.335.0: `precheckSkips` → held →
|
||||
no stop, contention for each due tier (15-minute skip, one „BLOCKED" alarm after 3 h); free / older agent / failed
|
||||
read → the old order; the window's own tier in flight → attach as before.
|
||||
- **The race** (free at the read, busy at the start): one short stop, resumed at once — today's behaviour, pinned.
|
||||
- Tests, red-proved: agent `TestBackupStatus_R921_HostBusy` (assignment removed → `free gate: host_busy = <nil>`);
|
||||
controller `TestR921_NoStopWhileTheHostIsBusy` (branch removed → 3 apps stopped, the start refused),
|
||||
`…SkipsStillAlarmAfterTheCeiling`, `…OtherAnswersKeepTheOldOrder` (4 rows), `TestHostBusy_ReadsTheAgentsGate`.
|
||||
- Live: demo-hp's agent answers no `host_busy` (read inside 9201) — the old order, as designed. A live busy state was
|
||||
not made (it needs the new agent on the box). R-921 closed per the brief.
|
||||
|
||||
## Part 2 — R-1070 (`11` §8.2) — DONE in the agent (not released)
|
||||
|
||||
- Where: `felhom-agent/internal/osupdate/leg.go` `HostHealthVerdict` (the tunnel check) and
|
||||
`internal/hub/cloudflared.go` `ClassifyTunnel`. Measured on demo-hp: `docker inspect` of a missing name prints
|
||||
`error: no such object: …` (Docker 29); the match wanted `No such object` → `unknown` → the host rule failed.
|
||||
- Fix: case-insensitive match → new state `absent`; the host rule passes `absent` only when the reading from before
|
||||
the step (Docker answered) also had no cloudflared; `not_running` and `unknown` still fail. Hub 0.162.0 shows
|
||||
`absent` as „none set up". `11` §8.2 had said `unknown` does not fail — corrected (the code always failed it).
|
||||
- Tests, red-proved: `TestHostHealthVerdict_R1070_NoTunnelSetUp` (7 rows), three `TestClassifyTunnel` rows,
|
||||
`TestSystemRows_R1070_AbsentTunnelIsNotAWarning`.
|
||||
|
||||
## Part 3 — R-893: measured and designed, no restore code changed
|
||||
|
||||
Helper on 9202 (fences carried in full). Evidence and the note: `documentation/audits/night-2026-10-19/P3/`.
|
||||
|
||||
- Rallly 4.15.3 + postgres 18; a scratch SFTP off-site on 9202's own disk; one failing SQL line in the prepared copy
|
||||
of the dump, so the replay failed AFTER the volume was replaced.
|
||||
- **What it left:** the newest rows (the undo dump) over the backup's volume; a table only the backup had stayed
|
||||
(37 tables against 36); definition unchanged (same version); app held (`restore_mixed`), a start → 409.
|
||||
- The restore page's sentence is false on both halves for this app → R-1169.
|
||||
- **Options:** A — copy the app's live volumes before the replay, put them back on failure, start the app (one copy's
|
||||
disk for minutes; Rallly 69 MB in ~1 s; a fit check before the stop). B — empty the DB volume before the undo load
|
||||
(no disk; a bad undo copy leaves an empty database; usually still a support call). **Pick: A** (the operator's D8
|
||||
direction). Not measured: the version-change branch (needs a drill-catalog commit).
|
||||
- Teardown: 9202 back on 0.331.0 as found, every throwaway removed (checked again by CC); host: `/tmp` only; hub: nothing.
|
||||
|
||||
## Part 4 — hub 0.162.0, built, NOT deployed (`05`)
|
||||
|
||||
- R-1001: start-up seeding from the DB for `host_stale`/`host_down`; `restore_test_stale` edge from the DB. Tests
|
||||
red-proved (each restart = a new checker over the same DB).
|
||||
- R-1011: exact-file sha (`ErrNoFile`); step bundles leave the list; a vouch of one → `artifact_file_missing`. Registry
|
||||
read: 6 step-bundle versions, every release holds `felhom-agent`. Tests red-proved.
|
||||
- R-953: **no change needed** — read back 18:21 UTC: ArgoCD tracks none of the three Secrets (no tracking mark), so the
|
||||
next deploy does nothing with them. `P4/r953-readback.txt`.
|
||||
- Image `sha256:69285e640a9c5e4ce59b1fbb5cd94623fa47a71798e9d86fa0767ee6c69a6ac8` from a clean clone at `d92a712e`.
|
||||
|
||||
## Part 5 — agent, committed, NOT released (`03`)
|
||||
|
||||
R-1010 (`56aabc7`): on „Failed to find logical volume …_vzdump" one retry with the API's `force=1` (`pvesh usage`
|
||||
lists `--force`; the same token privilege; no sudoers line, no root command); never a second force. Red-proved. Plus
|
||||
R-921's field and R-1070. CHANGELOG „Unreleased". Not exercised live (it needs a half-removed snapshot on `pve/data`).
|
||||
|
||||
## Part 6 — installer 1.35.0, written, NOT published
|
||||
|
||||
R-1167: `die_live_box` in EN and HU at the preflight, host-enroll and the controller.yaml fetch. Harness: a static case
|
||||
and two behaviour cases (the real functions against a local fake hub), red-proved on a mutated copy. **CI red twice**
|
||||
(2106, 2108): the CI runner has no curl; the behaviour cases now SKIP out loud there (`5d0dfeac`).
|
||||
|
||||
## Part 7 — the scanner (R-986), warn-first — DONE
|
||||
|
||||
- Shared `scripts/security_scan_gate.py` in all four runners' fast sets (hook + CI); `.security-accepted` per repo;
|
||||
8 decoys in scratch repos, red-proved; CI fetches the pinned gitleaks (both sha256 checked — seen working in CI 2108).
|
||||
- govulncheck pin v1.1.4 → **v1.8.0** (v1.1.4 cannot read the Go 1.27 code). CI has no Go: govulncheck NOT CHECKED
|
||||
there, out loud; the hook runs it.
|
||||
- **First scan of today's `main`** (`P7/triage.md`): govulncheck 0 called in all three modules. gitleaks history:
|
||||
controller 12, agent 15, catalog 1 (all as on 2026-10-12), felhom.eu 50 (+7: the usenet bench 5, a boot journal 1,
|
||||
a test 1). Today's tree: controller 4, felhom.eu 42, agent 15, catalog 0 — every one triaged: **0 real**; accepted =
|
||||
throwaway demo/bench app values (the bench LXC 9401 stopped, the apps gone from 9202); false = test constants, public
|
||||
keys, an SSH fingerprint, a tunnel ID, hashes.
|
||||
|
||||
## Part 8 — R-1083: the exact gap (no agent change)
|
||||
|
||||
The agent mints a per-guest token only inside `--selftest=provision`, for a guest it brings up from a golden in the same
|
||||
call. No product way exists for an existing guest. Row stays OPEN with the gap and two options.
|
||||
|
||||
## Part 9 — controller 0.335.0, the floor
|
||||
|
||||
rc.1 on demo-hp 18:08 UTC (0 errors) → `0.335.0` `sha256:262831f9…` → per-box floors 18:22 → all three moved by
|
||||
18:37 → box walk green (0 unhealthy, 0 refusals, catalog in sync; https-only 28 OK, 0 FAIL; „Fájlok ↗" one sign-in on
|
||||
demo-hp and Tester 1) → **global floor 0.335.0 / MinAgent 0.158.1 at 18:39:26 UTC**, read back; per-box floors cleared.
|
||||
i18n: no page changed, no fixture changed. Evidence `P9/`.
|
||||
|
||||
## Part 10 — ep0's first weekly clean-up
|
||||
|
||||
PENDING at the time of writing (runs 03:30 UTC).
|
||||
|
||||
## Versions, commits
|
||||
|
||||
- Controller 0.335.0 (`15ea002`, REPORT `c4b3b0a`, scanner `a03c25a`). Hub 0.162.0 built (`5f31da04`). Agent
|
||||
`56aabc7`, `84ef581` (not released). Installer 1.35.0 (`d92a712e`, `5d0dfeac`, not published). felhom.eu scanner
|
||||
`15f74288`, docs `f92fa5a8`. Catalog `87f5588`, `57ddb20`.
|
||||
- **System poster:** no listed fact changed tonight — none of the three.
|
||||
- **Instruction/doc corrections:** `11-os-updates.md` §8.2 (the `unknown` tunnel sentence was false); `nodes.md` and
|
||||
`nightly-rotation.md` (9202 is DHCP, .121 now).
|
||||
|
||||
## Rows
|
||||
|
||||
Register **161 before, 161 after** (before Part 10). **Closed 3:** R-921, R-953, R-986. **Opened 3:** R-1168 (warn →
|
||||
fail), R-1169 (the false restore sentence), R-1170 (a vanished tunnel raises no alarm). Re-stated: R-1001, R-1011,
|
||||
R-1010, R-1070, R-1167 (READY, waiting on the release/deploy/publish), R-893 (READY, D20), R-1083 (the gap).
|
||||
|
||||
## Observations
|
||||
|
||||
1. The restore page's held-mixed sentence is false for the measured app. FILED: R-1169
|
||||
2. A tunnel container that disappears on a box with a tunnel configured raises no alarm. FILED: R-1170
|
||||
3. The scanner warns only. FILED: R-1168
|
||||
4. `11` §8.2 said an `unknown` tunnel does not fail the host rule; the code always failed it. NOT-A-FINDING: corrected in the session (the size rule); it is R-1070's cause.
|
||||
5. govulncheck v1.1.4 (the 2026-10-12 pin) cannot read Go 1.27 code. NOT-A-FINDING: the pin moved to v1.8.0 tonight, recorded in `SCANNERS.md` §5.
|
||||
6. The CI runner has no curl; two new harness cases failed CI 2106 and 2108. NOT-A-FINDING: fixed the same night (`5d0dfeac`), they skip out loud there.
|
||||
7. R-953's tracking marks went after 2026-10-12 with no record of who removed them. NOT-A-FINDING: the danger the row named is gone, nothing to fix.
|
||||
8. 9202's DHCP address moved (.114 → .121) and the walk named the old one. NOT-A-FINDING: fixed (`box_walk.py` reads it; two docs corrected).
|
||||
9. The brief said the installer already prints HU and EN; it printed English only. NOT-A-FINDING: the HU lines are new, named in the scripts CHANGELOG.
|
||||
10. The agent's backup cap (R-973) can also refuse a POST after the stop. NOT-A-FINDING: it answers 409, so the apps resume at once (the BUSY path), and it fires only after a flood of presses that each stopped the apps anyway.
|
||||
11. The manual off-site run dumped and restarted every app, not only the one switched on. NOT-A-FINDING: by design, the coherent app-data pair (`07`).
|
||||
12. On CI's depth-1 clone, gitleaks scans the whole tree (the root commit), ~100 s for felhom.eu. NOT-A-FINDING: a stronger check than the diff, all current hits accepted; recorded in `CONTEXT.md`.
|
||||
|
||||
## Teardown
|
||||
|
||||
- Machine: 9202 back as found (0.331.0, no throwaway left — checked); 9201 on the release 0.335.0; probe scripts removed.
|
||||
- Host: `/tmp` copies only, removed. Nothing on `pve/data`.
|
||||
- Hub: per-box floors cleared; nothing else created.
|
||||
- ep0: read only.
|
||||
@@ -232,7 +232,11 @@ The controller (in its LXC) reaches the agent (on the host) over the local bridg
|
||||
- `POST /rollback` — roll *this* guest back to a named snapshot (post-deploy failure recovery).
|
||||
- `POST /backup` — request a backup-now of *this* guest (enqueued; non-destructive).
|
||||
- `GET /backup/due` — whether a policy-scheduled backup is due for *this* guest, so the controller can quiesce then call `POST /backup` (the app-consistent path, §8).
|
||||
- `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI.
|
||||
- `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI. **`host_busy`
|
||||
(R-921, agent Unreleased `56aabc7`):** `GET /backup/status` also says whether the host-wide heavy-operation gate is
|
||||
held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off), the
|
||||
state in which `POST /backup` answers 409; absent when the gate is not wired. The controller (≥ 0.335.0) reads it
|
||||
before it stops any app. A read, never a reservation.
|
||||
- **Crash-boot fact (R-856, agent v0.149.0):** `GET /host/crash-guard` — the host crash guard's last-boot record
|
||||
(`present`, `last_boot_at`, `last_boot_unclean`, `tripped`) read from `/var/lib/felhom-crash-guard/state.json`; a missing
|
||||
or garbled file answers 200 `present:false`, an older agent 404 — both read as a normal boot. The controller waits
|
||||
@@ -399,7 +403,10 @@ per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a ba
|
||||
> space on …" — so the hub raises `restore_test_failed`; it is never a pass and never dropped.
|
||||
> - **Leftovers on a timer.** A failed scratch teardown and the stale-lock sweep (R-673) run every 10
|
||||
> minutes, not only at agent start; the sweep holds the one-heavy-operation gate; after 3 failed
|
||||
> teardown tries the operator is told.
|
||||
> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent Unreleased
|
||||
> `56aabc7`):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's
|
||||
> snapshot already gone), it retries once with Proxmox's own `force=1` on the same API call (no sudoers line, no
|
||||
> root command); any other error is returned unchanged, and a failed forced delete is never forced again.
|
||||
> - **A thin pool ≥ 90 % requests an immediate report**; the hub judges a thin pool on the worse of data
|
||||
> and metadata, critical at 90 %, one alarm per pool per 6 hours (`08` §6.2).
|
||||
>
|
||||
|
||||
@@ -90,7 +90,11 @@ Evolves the existing staleness checker (60s **cadence**, a **configured** thresh
|
||||
2026-09-17, **45 m** since operator ruling A on R-549; OK under it, down at 2× = 90 m; today: controller-report recency → `node_stale`/`down`/`recovered`):
|
||||
|
||||
- **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's
|
||||
liveness signal; a silent agent = the box is gone (the critical alert).
|
||||
liveness signal; a silent agent = the box is gone (the critical alert). **A hub restart does not re-mail a known
|
||||
silence (R-1001, hub 0.162.0 — built 2026-10-19, not yet deployed):** a host whose newest `host_stale`/`host_down`
|
||||
event is newer than its last report is seeded with that state at start, silently; `restore_test_stale` reads its edge
|
||||
from the DB on the first sweep (a signal for the tier inside the read window and newer than its last proof). A failed
|
||||
lookup lets the alarm through.
|
||||
- **Guest up/down comes from the host report's per-guest status** — authoritative, every poll, faster
|
||||
than waiting for a guest report to go stale.
|
||||
- **Guest-report recency = secondary** app-level signal.
|
||||
@@ -134,7 +138,11 @@ walk is green; the golden is baked weekly and vouched by the operator; a new box
|
||||
itself to the floor (R-1052: not yet proven end to end). Setting the global floor today needs the operator page with the
|
||||
operator's password — there is no scoped route for CC (R-1051). **An agent can be vouched mid-week; the golden stays
|
||||
weekly** (hub v0.158.0): the R-120 golden guard judges only a CHANGED golden selection, so a Day-0 save that keeps the
|
||||
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet.
|
||||
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. **The agent list
|
||||
offers real agent packages only (R-1011, hub 0.162.0, not yet deployed):** the sha lookup asks for exactly the
|
||||
`felhom-agent` file (the golden: `golden.tar.zst`) and never falls back to another file, so a step bundle
|
||||
(`0.156.1-step1`, only the config bundle) is left out of the list, and a vouch of one is refused
|
||||
(`artifact_file_missing`).
|
||||
|
||||
**The controller floor and its agent requirement (hub v0.112.0, R-472).** The managed controller floor
|
||||
rides the report ACK. `store.ResolveManagedFloor` decides per box whether to serve it, from three
|
||||
|
||||
@@ -728,7 +728,13 @@ definition is written or a named volume replaced — a failed placement, volume
|
||||
the live definition back and HOLDS the app stopped (`restore_mixed`, operator-cleared); the household reads that it
|
||||
needs our help; the operator gets a `backup_run_failures` line. The hold is persisted before the stop. Placed files
|
||||
ALONE (no version change, no volume replaced) still restart after a successful rollback (`TestR379_ScenarioA`): that
|
||||
mix, and the undo of everything, is option A — „put back exactly as it was", the next slice on R-893.
|
||||
mix, and the undo of everything, is option A — „put back exactly as it was", the next slice on R-893. **`[FACT]`
|
||||
2026-10-19 (measured on 9202, controller 0.335.0, `audits/night-2026-10-19/P3/`):** a forced replay failure after the
|
||||
volume was replaced left the NEWEST rows (the undo dump) on top of the backup's volume — a table only the backup had
|
||||
stayed (37 tables, the live app had 36); the definition did not move (same version); the app was held and a start
|
||||
answered 409. The restore page's „the database is back as it was" is false for it. Design note with the two options and
|
||||
the pick (A: a copy of the app's live volumes, put back on failure — one copy's disk for the restore's length, a fit
|
||||
check before the stop): `audits/night-2026-10-19/P3/design-R-893.md`; the decision is on `STATUS.md`.
|
||||
|
||||
**[DESIGN] 2026-08-22 — the failure ladder of a database restore: replay → rollback → hold.**
|
||||
Recorded here rather than only in a closed register row, because a decision that survives only inside
|
||||
@@ -826,8 +832,12 @@ parts above, not a measured press).
|
||||
demo-hp: the off-site tier stopped every app for ~1 min and the agent then refused it (BUSY: the night OS step held the
|
||||
agent's host-wide lock). Before stopping anything the controller now asks the agent which tiers have a job in flight
|
||||
(`GET /backup/status`); another tier's job running or snapshotted → no stop, the tier stays due. A refusal after the
|
||||
stop still resumes the apps at once (pinned by test). **Not covered:** the host-wide busy lock (OS step, restore test,
|
||||
fstrim) — the agent serves it on no endpoint, so the measured case needs an agent change too (R-921, LEFT).
|
||||
stop still resumes the apps at once (pinned by test). **The host-wide busy lock (2026-10-19, operator answer „add one
|
||||
field"):** the agent serves it as `host_busy` on `GET /backup/status` (agent Unreleased `56aabc7`); controller 0.335.0
|
||||
(the floor) reads it before any stop — held → no app stopped, each due tier recorded as contention (retry in 15 min,
|
||||
one „BLOCKED" alarm after 3 h), exactly the verdict a 409 gets. An agent without the field keeps the old order (stop,
|
||||
ask, resume at once). The check is a read, not a reservation: a gate taken between the read and the start costs one
|
||||
short stop, resumed at once. The measured case is closed on a box once it runs the next agent release.
|
||||
**Measured 2026-10-05 on demo-hp (9 apps, controller v0.295.0):** „Mentés most" stopped the apps at 09:19:08Z, the
|
||||
local tier ran 09:19:29–09:24:09, the PBS tier was busy (the controller logged a retry in 15 min; no second stop was seen in the next 55 min), the last app was back at 09:24:55Z — the
|
||||
longest stop **5 min 47 s**, for the local tier alone. The button text and its confirm (v0.296.0) give both
|
||||
|
||||
@@ -724,7 +724,11 @@ Evidence: `audits/os-host-lane-2026-10-04/` (parts A–G).
|
||||
skips the host step.
|
||||
- **The host health rule** (`HostHealthVerdict`, pinned in `internal/osupdate`): `felhom-agent`, `pveproxy`,
|
||||
`pvedaemon`, `pvestatd` and `pve-cluster` are `active`; the customer guest runs; the guest health rule (§8.1)
|
||||
passes; the tunnel is `running` — an `unknown` tunnel does not fail it, `not_running` does. Same 5-minute wait.
|
||||
passes; the tunnel is `running` — `not_running` and `unknown` fail it (CORRECTED 2026-10-19: this line said
|
||||
„an `unknown` tunnel does not fail it"; the code always failed it, which is what Tester4 hit, R-1070). **Since agent
|
||||
Unreleased `56aabc7` (R-1070):** a guest with NO cloudflared container reads `absent` (Docker 29's lower-case
|
||||
„no such object" used to fall through to `unknown`), and `absent` passes when the reading from before the step also
|
||||
listed no cloudflared container (Docker answered) — a box with no tunnel set up. Same 5-minute wait.
|
||||
- **Separate approved sets:** host and guest releases are separate (`os-host-…`, `os-guest-…`), each by the same rule
|
||||
(24 h, 1 night of THAT layer, every ring-0 box). Ring 1 receives `host_release` beside `release`.
|
||||
- **Reboot needed:** reported when PID 1 or `lxc-start` maps a replaced file, with the date of the first scanned
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
Mon Oct 19 18:45:25 UTC 2026
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.331.0
|
||||
filebrowser gtstef/filebrowser:1.5.6-stable Up 16 hours (healthy)
|
||||
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 18 hours (healthy)
|
||||
paperless-postgres postgres:18-alpine Up 18 hours (healthy)
|
||||
paperless-redis redis:7-alpine Up 18 hours (healthy)
|
||||
immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up 18 hours (healthy)
|
||||
immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up 18 hours (healthy)
|
||||
immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up 18 hours (healthy)
|
||||
immich-redis redis:7-alpine Up 18 hours (healthy)
|
||||
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.331.0 Up 22 hours (healthy)
|
||||
traefik traefik:v3.7.13 Up 3 days
|
||||
/opt/docker/stacks/rallly
|
||||
docker-compose.yml
|
||||
hold-logs
|
||||
Filesystem Size Used Avail Use% Mounted on
|
||||
/dev/loop1 69G 11G 54G 17% /var/lib/felhom
|
||||
total 20
|
||||
drwx------ 2 root root 4096 Sep 30 08:40 .
|
||||
drwxr-xr-x 4 root root 4096 Oct 19 18:46 ..
|
||||
-rw-r--r-- 1 root root 91 Sep 30 08:40 known_hosts
|
||||
-rw------- 1 root root 64 Sep 30 08:40 repo_password
|
||||
-rw------- 1 root root 411 Sep 30 08:40 ssh_key
|
||||
83723cedb2aa77ace51345e585937720fde114d1f0b24da21eb37fba4bd39374 settings.json
|
||||
-rw-r--r-- 1 root root 27003 Oct 19 03:14 /root/r893-settings.json.before
|
||||
@@ -0,0 +1,13 @@
|
||||
INSERT 0 1
|
||||
=== A-written 18:49:07 UTC
|
||||
rallly lukevella/rallly:4.15.3 Up About a minute (healthy)
|
||||
rallly-postgres postgres:18-alpine Up About a minute (healthy)
|
||||
markers (phase=rows): A-before-snapshot=1
|
||||
r893_after table exists: 0
|
||||
r893_after rows: ERROR: relation "r893_after" does not exist
|
||||
LINE 1: select count(*) from r893_after;
|
||||
^
|
||||
public tables: 34
|
||||
prisma migrations: 153
|
||||
db volume created: 2026-10-19T18:47:54Z
|
||||
definition: image: lukevella/rallly:4.15.3 image: postgres:18-alpine
|
||||
@@ -0,0 +1,13 @@
|
||||
INSERT 0 1
|
||||
CREATE TABLE
|
||||
INSERT 0 1
|
||||
=== B-written 18:51:46 UTC
|
||||
rallly lukevella/rallly:4.15.3 Up 35 seconds (healthy)
|
||||
rallly-postgres postgres:18-alpine Up 41 seconds (healthy)
|
||||
markers (phase=rows): A-before-snapshot=1, B-after-snapshot=1
|
||||
r893_after table exists: 1
|
||||
r893_after rows: 1
|
||||
public tables: 35
|
||||
prisma migrations: 153
|
||||
db volume created: 2026-10-19T18:47:54Z
|
||||
definition: image: lukevella/rallly:4.15.3 image: postgres:18-alpine
|
||||
@@ -0,0 +1,11 @@
|
||||
126
|
||||
34
|
||||
2660:-- PostgreSQL database dump complete
|
||||
2660-SELECT r893_forced_replay_failure(); -- R-893 measurement 2026-10-19: forced error AFTER the volumes are replaced
|
||||
2661:-- PostgreSQL database dump complete
|
||||
2720-SELECT r893_forced_replay_failure(); -- R-893 measurement 2026-10-19: forced error AFTER the volumes are replaced
|
||||
2721:-- PostgreSQL database dump complete
|
||||
total 67608
|
||||
drwxr-xr-x 2 root root 4096 Oct 19 18:54 .
|
||||
drwxr-xr-x 5 root root 4096 Oct 19 18:54 ..
|
||||
-rw-r--r-- 1 root root 69221888 Oct 19 18:54 rallly_rallly_postgres_data.tar
|
||||
@@ -0,0 +1,15 @@
|
||||
CREATE TABLE
|
||||
INSERT 0 1
|
||||
INSERT 0 1
|
||||
=== before-snapshot-2 18:53:21 UTC
|
||||
rallly lukevella/rallly:4.15.3 Up 2 minutes (healthy)
|
||||
rallly-postgres postgres:18-alpine Up 2 minutes (healthy)
|
||||
markers (phase=rows): A-before-snapshot=1, B-after-snapshot=1, C-in-snapshot-2=1
|
||||
r893_after table exists: 1
|
||||
r893_after rows: 1
|
||||
r893_only_old exists: 1
|
||||
r893_new2 exists: 0
|
||||
public tables: 36
|
||||
prisma migrations: 153
|
||||
db volume created: 2026-10-19T18:47:54Z
|
||||
definition: image: lukevella/rallly:4.15.3 image: postgres:18-alpine
|
||||
@@ -0,0 +1,16 @@
|
||||
DROP TABLE
|
||||
INSERT 0 1
|
||||
CREATE TABLE
|
||||
INSERT 0 1
|
||||
=== LIVE-before-the-restore 18:54:53 UTC
|
||||
rallly lukevella/rallly:4.15.3 Up 30 seconds (healthy)
|
||||
rallly-postgres postgres:18-alpine Up 35 seconds (healthy)
|
||||
markers (phase=rows): A-before-snapshot=1, B-after-snapshot=1, C-in-snapshot-2=1, D-after-snapshot-2=1
|
||||
r893_after table exists: 1
|
||||
r893_after rows: 1
|
||||
r893_only_old exists: 0
|
||||
r893_new2 exists: 1
|
||||
public tables: 36
|
||||
prisma migrations: 153
|
||||
db volume created: 2026-10-19T18:47:54Z
|
||||
definition: image: lukevella/rallly:4.15.3 image: postgres:18-alpine
|
||||
@@ -0,0 +1,40 @@
|
||||
2026/10/19 18:55:43 dbdump.go:277: [DEBUG] DumpOne: starting dump for container=rallly-postgres, stack=rallly, dbType=postgres, dest=/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql
|
||||
2026/10/19 18:55:43 dbdump.go:314: [DEBUG] DumpOne: pg_dump command: docker exec 8ad4e7b50692 pg_dump -U rallly -d rallly --clean --if-exists --no-owner --no-privileges
|
||||
2026/10/19 18:55:43 [WARN] [backup] ValidateDump: /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql is structurally valid (36 tables) but its accounts table has NO rows — the dump may predate the customer's data
|
||||
2026/10/19 18:55:43 dbdump.go:424: [DEBUG] DumpOne: completed rallly-postgres → pre-restore-20261019T185543Z-rallly-postgres.sql (size=110.0 KB, valid=true, tables=36, duration=325ms)
|
||||
2026/10/19 18:55:43 dbdump.go:429: [INFO] [backup] DB dump: rallly-postgres → pre-restore-20261019T185543Z-rallly-postgres.sql (110.0 KB, 325ms, 36 tables)
|
||||
2026/10/19 18:55:43 offbox_reconstitute.go:262: [INFO] [offbox] rallly: pre-restore safety dump written → pre-restore-20261019T185543Z-rallly-postgres.sql (110.0 KB)
|
||||
2026/10/19 18:55:43 manager.go:1437: [DEBUG] [stacks] StopStack rallly: current state=running deployed=true containers=2
|
||||
2026/10/19 18:55:43 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||
2026/10/19 18:55:43 manager.go:1695: [DEBUG] Running: docker compose down (in /opt/docker/stacks/rallly)
|
||||
2026/10/19 18:55:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 1m0s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:55:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 1m10s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:55:54 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 10.7s)
|
||||
2026/10/19 18:55:54 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly
|
||||
2026/10/19 18:55:55 restore.go:253: [DEBUG] [backup] Volume rallly_rallly_postgres_data restored successfully
|
||||
2026/10/19 18:55:55 restore.go:258: [INFO] [backup] Restored 1 Docker volume(s) for rallly
|
||||
2026/10/19 18:55:55 manager.go:1404: [INFO] [stacks] Starting stack rallly services only: [rallly-postgres]
|
||||
2026/10/19 18:55:55 manager.go:1695: [DEBUG] Running: docker compose up -d rallly-postgres (in /opt/docker/stacks/rallly)
|
||||
2026/10/19 18:55:56 manager.go:1415: [INFO] [stacks] Stack rallly services [rallly-postgres] started (took 0.4s)
|
||||
2026/10/19 18:55:56 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: b55136567813 rallly-postgres rallly postgres:18-alpine
|
||||
2026/10/19 18:55:56 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=b55136567813)
|
||||
2026/10/19 18:55:56 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly
|
||||
2026/10/19 18:55:56 restore_db.go:87: [INFO] [backup] Restore rallly: replaying DB dump into rallly-postgres (postgres)
|
||||
2026/10/19 18:55:56 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/19 18:55:57 dbdump.go:814: [ERROR] [backup] postgres import into rallly-postgres FAILED: exit status 3 — engine output follows:
|
||||
2026/10/19 18:55:57 offbox_reconstitute.go:1062: [ERROR] [offbox] rallly: database replay failed, rolling back to the pre-restore state: importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3
|
||||
2026/10/19 18:55:57 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: b55136567813 rallly-postgres rallly postgres:18-alpine
|
||||
2026/10/19 18:55:57 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=b55136567813)
|
||||
2026/10/19 18:55:57 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly
|
||||
2026/10/19 18:55:57 offbox_reconstitute.go:603: [DEBUG] [offbox] rallly: rallly-postgres was re-created during the restore (8ad4e7b50692 → b55136567813) — rolling back into the live container
|
||||
2026/10/19 18:55:57 offbox_reconstitute.go:606: [INFO] [offbox] rallly: rolling back to the pre-restore state from pre-restore-20261019T185543Z-rallly-postgres.sql
|
||||
2026/10/19 18:55:57 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/19 18:55:58 dbdump.go:820: [INFO] [backup] Imported DB dump pre-restore-20261019T185543Z-rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/19 18:55:58 offbox_reconstitute.go:611: [INFO] [offbox] rallly: rollback complete — 1 database(s) returned to the pre-restore state
|
||||
2026/10/19 18:55:58 settings.go:1951: [WARN] [settings] restore hold SET for rallly — the app stays stopped until it is cleared
|
||||
2026/10/19 18:55:58 manager.go:1437: [DEBUG] [stacks] StopStack rallly: current state=starting deployed=true containers=1
|
||||
2026/10/19 18:55:58 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||
2026/10/19 18:55:58 manager.go:1695: [DEBUG] Running: docker compose down (in /opt/docker/stacks/rallly)
|
||||
2026/10/19 18:55:59 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 0.9s)
|
||||
2026/10/19 18:55:59 offbox_reconstitute.go:531: [ERROR] [offbox] rallly: the restore stopped half-way after files/volumes/version had moved — HOLDING the app stopped for support (R-893); error was: importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3
|
||||
2026/10/19 18:55:59 offbox_handlers.go:547: [ERROR] [web] off-box reconstitute rallly (async): a(z) rallly adatbázisának visszaállítása sikertelen. Az adatbázis a visszaállítás előtti állapotban van, de az alkalmazás fájljai és többi adata már a mentésből származnak, ezért nem illenek össze. Az alkalmazást biztonsági okból LEÁLLÍTVA hagytuk. Ehhez segítség kell: vedd fel velünk a kapcsolatot
|
||||
@@ -0,0 +1,14 @@
|
||||
=== after the failed replay 18:56:42 UTC
|
||||
(no rallly container listed = compose down; app held)
|
||||
definition: image: lukevella/rallly:4.15.3 image: postgres:18-alpine
|
||||
Unable to find image 'alpine:3' locally
|
||||
3: Pulling from library/alpine
|
||||
Digest: sha256:294b683cb724975bec92580e1e685676bd4b50bda910ddb8c51d4cabeaec77e6
|
||||
Status: Downloaded newer image for alpine:3
|
||||
markers (phase=rows): A-before-snapshot=1, B-after-snapshot=1, C-in-snapshot-2=1, D-after-snapshot-2=1
|
||||
r893_after exists: 1 rows: 1
|
||||
r893_only_old exists: 1 rows: 1
|
||||
r893_new2 exists: 1 rows: 1
|
||||
public tables: 37
|
||||
prisma migrations: 153
|
||||
read copy removed
|
||||
@@ -0,0 +1,4 @@
|
||||
GET stack: 200 {'status': None, 'state': 'stopped', 'held': None, 'hold': None, 'restore_hold': None, 'deployed': True}
|
||||
POST start: 409 {"ok": false, "data": {"reason": "held"}, "error": "a(z) rallly visszaállítása 2026-10-19 18:55-kor félúton megszakadt: az adatai részben a mentésből származnak, részben nem. Az alkalmazás biztonsági okból leállítva marad. Ehhez segítség kell: vedd fel velünk a kapcsolatot"}
|
||||
restore-status: 200 {"ok": true, "data": {"running": false, "op": "offbox-reconstitute", "stack": "rallly", "started_at": "2026-10-19T18:55:42.483513793Z", "last": {"op": "offbox-reconstitute", "stack": "rallly", "ok": false, "message": "A teljes visszaállítás sikertelen: a(z) rallly adatbázisának visszaállítása sikertelen. Az adatbázis a visszaállítás előtti állapotban van, de az alkalmazás fájljai és többi adata már a mentésből származnak, ezért nem illenek össze. Az alkalmazást biztonsági okból LEÁLLÍTVA hagytuk. Ehhez segítség kell: vedd fel velünk a kapcsolatot", "finished_at": "2026-10-19T18:55:59.246941172
|
||||
settings restore hold: {'stack': 'rallly', 'at': '2026-10-19T18:55:58Z', 'replay_error': 'importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3', 'reason': 'restore_mixed'}
|
||||
@@ -0,0 +1,176 @@
|
||||
2026/10/19 18:55:42 auth.go:160: [DEBUG] [web] login attempt: visitor 192.168.0.180 (peer 172.18.0.3:41876, X-Forwarded-For "192.168.0.180", CF-Connecting-IP "")
|
||||
2026/10/19 18:55:42 auth.go:192: [DEBUG] [web] login successful from 192.168.0.180, creating session
|
||||
2026/10/19 18:55:42 session_store.go:140: [DEBUG] [web] sessions: saved 142 session(s) to dashboard-sessions.json
|
||||
2026/10/19 18:55:42 auth.go:316: [DEBUG] [web] session created, expires=2026-10-26T18:55:42Z, active_sessions=146
|
||||
2026/10/19 18:55:42 auth.go:212: [INFO] [web] Login from 192.168.0.180
|
||||
2026/10/19 18:55:42 auth.go:144: [DEBUG] [web] auth: valid session for GET /
|
||||
2026/10/19 18:55:42 server.go:637: [DEBUG] [web] ServeHTTP: GET / from 172.18.0.3:41876
|
||||
2026/10/19 18:55:42 auth.go:144: [DEBUG] [web] auth: valid session for GET /launcher
|
||||
2026/10/19 18:55:42 server.go:637: [DEBUG] [web] ServeHTTP: GET /launcher from 172.18.0.3:41876
|
||||
2026/10/19 18:55:42 auth.go:144: [DEBUG] [web] auth: valid session for POST /backup/offbox/reconstitute
|
||||
2026/10/19 18:55:42 server.go:637: [DEBUG] [web] ServeHTTP: POST /backup/offbox/reconstitute from 172.18.0.3:41876
|
||||
2026/10/19 18:55:43 [WARN] [backup] ValidateDump: /mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql is structurally valid (36 tables) but its accounts table has NO rows — the dump may predate the customer's data
|
||||
2026/10/19 18:55:43 offbox_reconstitute.go:1206: [WARN] [offbox] rallly: the snapshot dump rallly-postgres.sql has no account rows — it may predate the customer's data
|
||||
2026/10/19 18:55:43 restore_dbcreds.go:114: [INFO] [backup] Restore rallly: database password(s) [DB_PASSWORD] come from the backup, [] stay as installed (older than the data) — values not logged (R-1106)
|
||||
2026/10/19 18:55:43 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers
|
||||
2026/10/19 18:55:43 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: 5e59e7bdadc1 rallly rallly lukevella/rallly:4.15.3
|
||||
8ad4e7b50692 rallly-postgres rallly postgres:18-alpine
|
||||
197253b4df3a paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15
|
||||
f7b4c0e21fd3 paperless-redis paperless-ngx redis:7-alpine
|
||||
8c1fda4f57c7 paperless-postgres paperless-ngx postgres:18-alpine
|
||||
011c9b739e1a immich-server immich ghcr.io/immich-app/immich-server:v3.2.4
|
||||
224ee4af256c immich-machine-learning immich ghcr.io/immich-app/immich-machine-learning:v3.2.4
|
||||
c030b51054cb...
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container rallly (image=lukevella/rallly:4.15.3, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=8ad4e7b50692)
|
||||
2026/10/19 18:55:43 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=8c1fda4f57c7)
|
||||
2026/10/19 18:55:43 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
|
||||
2026/10/19 18:55:43 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-server (image=ghcr.io/immich-app/immich-server:v3.2.4, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-machine-learning (image=ghcr.io/immich-app/immich-machine-learning:v3.2.4, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: immich-postgres (id=c030b51054cb)
|
||||
2026/10/19 18:55:43 dbdump.go:198: [DEBUG] DiscoverDatabases: immich-postgres → stack=immich, dbUser=immich, dbName=immich
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container r893-sftp (image=atmoz/sftp:alpine, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.6-stable, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.335.0, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.13, not a database)
|
||||
2026/10/19 18:55:43 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 10 non-DB container(s)
|
||||
2026/10/19 18:55:43 dbdump.go:208: [INFO] [backup] Discovered 3 databases
|
||||
2026/10/19 18:55:43 dbdump.go:277: [DEBUG] DumpOne: starting dump for container=rallly-postgres, stack=rallly, dbType=postgres, dest=/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql
|
||||
2026/10/19 18:55:43 dbdump.go:314: [DEBUG] DumpOne: pg_dump command: docker exec 8ad4e7b50692 pg_dump -U rallly -d rallly --clean --if-exists --no-owner --no-privileges
|
||||
2026/10/19 18:55:43 [WARN] [backup] ValidateDump: /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql is structurally valid (36 tables) but its accounts table has NO rows — the dump may predate the customer's data
|
||||
2026/10/19 18:55:43 dbdump.go:424: [DEBUG] DumpOne: completed rallly-postgres → pre-restore-20261019T185543Z-rallly-postgres.sql (size=110.0 KB, valid=true, tables=36, duration=325ms)
|
||||
2026/10/19 18:55:43 dbdump.go:429: [INFO] [backup] DB dump: rallly-postgres → pre-restore-20261019T185543Z-rallly-postgres.sql (110.0 KB, 325ms, 36 tables)
|
||||
2026/10/19 18:55:43 offbox_reconstitute.go:262: [INFO] [offbox] rallly: pre-restore safety dump written → pre-restore-20261019T185543Z-rallly-postgres.sql (110.0 KB)
|
||||
2026/10/19 18:55:43 manager.go:1437: [DEBUG] [stacks] StopStack rallly: current state=running deployed=true containers=2
|
||||
2026/10/19 18:55:43 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||
2026/10/19 18:55:43 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DB_PASSWORD, DOMAIN, SECRET_PASSWORD, IMPORT_PATH] (10 app + 0 system)
|
||||
2026/10/19 18:55:43 manager.go:1695: [DEBUG] Running: docker compose down (in /opt/docker/stacks/rallly)
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] GetInfo starting (hddPath="/mnt/felhom-drives/scratch_hdd", hasCPUCollector=true)
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readMemInfo: guest cap=25898MB (host total was 30714348KB) → used≈12771MB avail≈13127MB
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readMemInfo: totalKB=30714348 availKB=15567204 → total=25898MB avail=13127MB used=12771MB (49.3%)
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readDiskUsage: path="/" bsize=4096 total=68.4GB used=12.0GB avail=52.8GB (18.5%)
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readDiskUsage: path="/mnt/felhom-drives/scratch_hdd" bsize=4096 total=937.8GB used=175.4GB avail=714.7GB (19.7%)
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readLoadAvg: raw="2.25 2.62 2.11 8/1611 4393" → 1m=2.25 5m=2.62 15m=2.11
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readThermalZones: /sys — found 1 zones
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] readTemperature: found via hwmon at /sys — 57.6°C (hwmon2)
|
||||
2026/10/19 18:55:44 info.go:13: [DEBUG] [system] GetInfo done in 59ms — mem=12771MB/25898MB (49.3%), rootDisk=12.0GB/68.4GB (18.5%), load=2.25/2.62/2.11, temp=57.6°C (hwmon2), cpu=13.9%
|
||||
2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job deadapp-check: execution starting
|
||||
2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job ring-spill: execution starting
|
||||
2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
|
||||
2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
|
||||
2026/10/19 18:55:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping immich — last check 1m40s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:55:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m0s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:55:44 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 1m0s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:55:44 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (3 skipped not due, 0 skipped no container)
|
||||
2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job after-unclean-boot: execution starting
|
||||
2026/10/19 18:55:44 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
|
||||
2026/10/19 18:55:44 manager.go:838: [INFO] [stacks] Status refresh: 11 containers across 70 stacks
|
||||
2026/10/19 18:55:54 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
|
||||
2026/10/19 18:55:54 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
|
||||
2026/10/19 18:55:54 manager.go:838: [INFO] [stacks] Status refresh: 10 containers across 70 stacks
|
||||
2026/10/19 18:55:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping immich — last check 1m50s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:55:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m10s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:55:54 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping rallly — last check 1m10s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:55:54 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (3 skipped not due, 0 skipped no container)
|
||||
2026/10/19 18:55:54 manager.go:1717: [DEBUG] Command completed: docker compose down (took 10.7s)
|
||||
2026/10/19 18:55:54 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 10.7s)
|
||||
2026/10/19 18:55:54 manager.go:838: [INFO] [stacks] Status refresh: 9 containers across 70 stacks
|
||||
2026/10/19 18:55:54 restore.go:215: [INFO] [backup] Restoring Docker volume rallly_rallly_postgres_data for rallly
|
||||
2026/10/19 18:55:55 restore.go:253: [DEBUG] [backup] Volume rallly_rallly_postgres_data restored successfully
|
||||
2026/10/19 18:55:55 restore.go:258: [INFO] [backup] Restored 1 Docker volume(s) for rallly
|
||||
2026/10/19 18:55:55 manager.go:1404: [INFO] [stacks] Starting stack rallly services only: [rallly-postgres]
|
||||
2026/10/19 18:55:55 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, DOMAIN, SECRET_PASSWORD, SUBDOMAIN, DB_PASSWORD, IMPORT_PATH] (10 app + 0 system)
|
||||
2026/10/19 18:55:55 manager.go:1695: [DEBUG] Running: docker compose up -d rallly-postgres (in /opt/docker/stacks/rallly)
|
||||
2026/10/19 18:55:56 manager.go:1717: [DEBUG] Command completed: docker compose up -d rallly-postgres (took 0.4s)
|
||||
2026/10/19 18:55:56 manager.go:1415: [INFO] [stacks] Stack rallly services [rallly-postgres] started (took 0.4s)
|
||||
2026/10/19 18:55:56 manager.go:838: [INFO] [stacks] Status refresh: 10 containers across 70 stacks
|
||||
2026/10/19 18:55:56 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers
|
||||
2026/10/19 18:55:56 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: b55136567813 rallly-postgres rallly postgres:18-alpine
|
||||
197253b4df3a paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15
|
||||
f7b4c0e21fd3 paperless-redis paperless-ngx redis:7-alpine
|
||||
8c1fda4f57c7 paperless-postgres paperless-ngx postgres:18-alpine
|
||||
011c9b739e1a immich-server immich ghcr.io/immich-app/immich-server:v3.2.4
|
||||
224ee4af256c immich-machine-learning immich ghcr.io/immich-app/immich-machine-learning:v3.2.4
|
||||
c030b51054cb immich-postgres immich ghcr.io/immich-app/postgres...
|
||||
2026/10/19 18:55:56 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=b55136567813)
|
||||
2026/10/19 18:55:56 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=8c1fda4f57c7)
|
||||
2026/10/19 18:55:56 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
|
||||
2026/10/19 18:55:56 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-server (image=ghcr.io/immich-app/immich-server:v3.2.4, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-machine-learning (image=ghcr.io/immich-app/immich-machine-learning:v3.2.4, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: immich-postgres (id=c030b51054cb)
|
||||
2026/10/19 18:55:56 dbdump.go:198: [DEBUG] DiscoverDatabases: immich-postgres → stack=immich, dbUser=immich, dbName=immich
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container r893-sftp (image=atmoz/sftp:alpine, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.6-stable, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.335.0, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.13, not a database)
|
||||
2026/10/19 18:55:56 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 9 non-DB container(s)
|
||||
2026/10/19 18:55:56 dbdump.go:208: [INFO] [backup] Discovered 3 databases
|
||||
2026/10/19 18:55:56 restore_db.go:87: [INFO] [backup] Restore rallly: replaying DB dump into rallly-postgres (postgres)
|
||||
2026/10/19 18:55:56 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly/mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/19 18:55:57 dbdump.go:814: [ERROR] [backup] postgres import into rallly-postgres FAILED: exit status 3 — engine output follows:
|
||||
ERROR: function r893_forced_replay_failure() does not exist
|
||||
LINE 1: SELECT r893_forced_replay_failure();
|
||||
^
|
||||
HINT: No function matches the given name and argument types. You might need to add explicit type casts.
|
||||
2026/10/19 18:55:57 offbox_reconstitute.go:1062: [ERROR] [offbox] rallly: database replay failed, rolling back to the pre-restore state: importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3
|
||||
2026/10/19 18:55:57 dbdump.go:130: [DEBUG] DiscoverDatabases: running docker ps to find database containers
|
||||
2026/10/19 18:55:57 dbdump.go:149: [DEBUG] DiscoverDatabases: docker ps output: b55136567813 rallly-postgres rallly postgres:18-alpine
|
||||
197253b4df3a paperless-webserver paperless-ngx ghcr.io/paperless-ngx/paperless-ngx:2.20.15
|
||||
f7b4c0e21fd3 paperless-redis paperless-ngx redis:7-alpine
|
||||
8c1fda4f57c7 paperless-postgres paperless-ngx postgres:18-alpine
|
||||
011c9b739e1a immich-server immich ghcr.io/immich-app/immich-server:v3.2.4
|
||||
224ee4af256c immich-machine-learning immich ghcr.io/immich-app/immich-machine-learning:v3.2.4
|
||||
c030b51054cb immich-postgres immich ghcr.io/immich-app/postgres...
|
||||
2026/10/19 18:55:57 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: rallly-postgres (id=b55136567813)
|
||||
2026/10/19 18:55:57 dbdump.go:198: [DEBUG] DiscoverDatabases: rallly-postgres → stack=rallly, dbUser=rallly, dbName=rallly
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-webserver (image=ghcr.io/paperless-ngx/paperless-ngx:2.20.15, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container paperless-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: paperless-postgres (id=8c1fda4f57c7)
|
||||
2026/10/19 18:55:57 dbdump.go:915: [DEBUG] DiscoverDatabases: paperless-postgres → stack "paperless-ngx" from the compose project label (the container name would have given "paperless")
|
||||
2026/10/19 18:55:57 dbdump.go:198: [DEBUG] DiscoverDatabases: paperless-postgres → stack=paperless-ngx, dbUser=paperless, dbName=paperless
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-server (image=ghcr.io/immich-app/immich-server:v3.2.4, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-machine-learning (image=ghcr.io/immich-app/immich-machine-learning:v3.2.4, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:178: [DEBUG] DiscoverDatabases: found postgres container: immich-postgres (id=c030b51054cb)
|
||||
2026/10/19 18:55:57 dbdump.go:198: [DEBUG] DiscoverDatabases: immich-postgres → stack=immich, dbUser=immich, dbName=immich
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container immich-redis (image=redis:7-alpine, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container r893-sftp (image=atmoz/sftp:alpine, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container filebrowser (image=gtstef/filebrowser:1.5.6-stable, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container felhom-controller (image=gitea.dooplex.hu/admin/felhom-controller:0.335.0, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:171: [DEBUG] DiscoverDatabases: skipping container traefik (image=traefik:v3.7.13, not a database)
|
||||
2026/10/19 18:55:57 dbdump.go:205: [DEBUG] DiscoverDatabases: found 3 database(s), skipped 9 non-DB container(s)
|
||||
2026/10/19 18:55:57 dbdump.go:208: [INFO] [backup] Discovered 3 databases
|
||||
2026/10/19 18:55:57 offbox_reconstitute.go:603: [DEBUG] [offbox] rallly: rallly-postgres was re-created during the restore (8ad4e7b50692 → b55136567813) — rolling back into the live container
|
||||
2026/10/19 18:55:57 offbox_reconstitute.go:606: [INFO] [offbox] rallly: rolling back to the pre-restore state from pre-restore-20261019T185543Z-rallly-postgres.sql
|
||||
2026/10/19 18:55:57 dbdump.go:800: [DEBUG] [backup] ImportDump: importing /mnt/sys_drive/felhom-data/backups/primary/rallly/db-dumps/pre-restore-20261019T185543Z-rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/19 18:55:58 dbdump.go:820: [INFO] [backup] Imported DB dump pre-restore-20261019T185543Z-rallly-postgres.sql into rallly-postgres (postgres)
|
||||
2026/10/19 18:55:58 offbox_reconstitute.go:611: [INFO] [offbox] rallly: rollback complete — 1 database(s) returned to the pre-restore state
|
||||
2026/10/19 18:55:58 settings.go:1951: [WARN] [settings] restore hold SET for rallly — the app stays stopped until it is cleared
|
||||
2026/10/19 18:55:58 settings.go:922: [DEBUG] [settings] saved to /opt/docker/felhom-controller/data/settings.json (27892 bytes)
|
||||
2026/10/19 18:55:58 settings.go:925: [INFO] [settings] Settings saved
|
||||
2026/10/19 18:55:58 manager.go:1437: [DEBUG] [stacks] StopStack rallly: current state=starting deployed=true containers=1
|
||||
2026/10/19 18:55:58 manager.go:1440: [INFO] [stacks] Stopping stack: rallly
|
||||
2026/10/19 18:55:58 manager.go:1686: [DEBUG] Env vars for compose: [PATH, HOSTNAME, FELHOM_BOOTSTRAP_PATH, HOME, DOMAIN, SUBDOMAIN, DB_PASSWORD, DOMAIN, SECRET_PASSWORD, IMPORT_PATH] (10 app + 0 system)
|
||||
2026/10/19 18:55:58 manager.go:1695: [DEBUG] Running: docker compose down (in /opt/docker/stacks/rallly)
|
||||
2026/10/19 18:55:59 manager.go:1717: [DEBUG] Command completed: docker compose down (took 0.9s)
|
||||
2026/10/19 18:55:59 manager.go:1449: [INFO] [stacks] Stack rallly stopped successfully (took 0.9s)
|
||||
2026/10/19 18:55:59 manager.go:838: [INFO] [stacks] Status refresh: 9 containers across 70 stacks
|
||||
2026/10/19 18:55:59 offbox_reconstitute.go:531: [ERROR] [offbox] rallly: the restore stopped half-way after files/volumes/version had moved — HOLDING the app stopped for support (R-893); error was: importing postgres dump for rallly: postgres import into rallly-postgres failed: exit status 3
|
||||
2026/10/19 18:55:59 notifier.go:1373: [WARN] notifier disabled (no hub configured): DROPPED event backup_run_failures (severity error) — further backup_run_failures events are logged at DEBUG only
|
||||
2026/10/19 18:55:59 offbox_handlers.go:547: [ERROR] [web] off-box reconstitute rallly (async): a(z) rallly adatbázisának visszaállítása sikertelen. Az adatbázis a visszaállítás előtti állapotban van, de az alkalmazás fájljai és többi adata már a mentésből származnak, ezért nem illenek össze. Az alkalmazást biztonsági okból LEÁLLÍTVA hagytuk. Ehhez segítség kell: vedd fel velünk a kapcsolatot
|
||||
2026/10/19 18:55:59 scheduler.go:85: [DEBUG] [scheduler] job docker-socket-watch: execution starting
|
||||
2026/10/19 18:56:04 scheduler.go:85: [DEBUG] [scheduler] job health-probes: execution starting
|
||||
2026/10/19 18:56:04 scheduler.go:85: [DEBUG] [scheduler] job status-refresh: execution starting
|
||||
2026/10/19 18:56:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping immich — last check 2m0s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:56:04 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping paperless-ngx — last check 1m20s ago, effective interval 5m0s, healthy=true
|
||||
2026/10/19 18:56:04 healthprobe.go:110: [DEBUG] [stacks] RunHealthProbes: collected 0 targets (2 skipped not due, 0 skipped no container)
|
||||
2026/10/19 18:56:04 manager.go:838: [INFO] [stacks] Status refresh: 9 containers across 70 stacks
|
||||
@@ -0,0 +1,37 @@
|
||||
20:57:56 [X] stop -> 200 {'ok': True, 'message': 'Stack rallly stop completed'}
|
||||
20:58:28 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'rallly', 'volumes_removed': ['rallly_rallly_postgres_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': '
|
||||
20:58:36 [X] after remove: deployed=False leftovers='/opt/docker/stacks/rallly'
|
||||
remove: 200
|
||||
r893-sftp
|
||||
images-removed
|
||||
ls: cannot access '/mnt/felhom-drives/scratch_hdd/backups/offsite-restore/rallly': No such file or directory
|
||||
ls: cannot access '/mnt/sys_drive/felhom-data/backups/primary/rallly': No such file or directory
|
||||
vol-check-done
|
||||
Stopping 'felhom-controller-bootstrap.service', but its triggering units are still active:
|
||||
felhom-controller-bootstrap.path
|
||||
felhom-controller Up 2 seconds (health: starting)
|
||||
83723cedb2aa77ace51345e585937720fde114d1f0b24da21eb37fba4bd39374 /var/lib/docker/volumes/felhom-controller-data/_data/data/settings.json
|
||||
total 20
|
||||
drwx------ 2 root root 4096 Sep 30 08:40 .
|
||||
drwxr-xr-x 4 root root 4096 Oct 19 18:59 ..
|
||||
-rw-r--r-- 1 root root 91 Sep 30 08:40 known_hosts
|
||||
-rw------- 1 root root 64 Sep 30 08:40 repo_password
|
||||
-rw------- 1 root root 411 Sep 30 08:40 ssh_key
|
||||
0.335.0-image-removed
|
||||
0
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.331.0 Up 50 seconds (healthy)
|
||||
3bc7298260da6566d4c09245124acd92f20f62bbebe46b0181c11d99a8314f2b /var/lib/docker/volumes/felhom-controller-data/_data/data/settings.json
|
||||
0
|
||||
filebrowser Up 2 minutes (healthy)
|
||||
paperless-webserver Up 7 minutes (healthy)
|
||||
paperless-redis Up 7 minutes (healthy)
|
||||
paperless-postgres Up 7 minutes (healthy)
|
||||
immich-server Up 7 minutes (healthy)
|
||||
immich-machine-learning Up 8 minutes (healthy)
|
||||
immich-postgres Up 8 minutes (healthy)
|
||||
immich-redis Up 8 minutes (healthy)
|
||||
traefik Up 3 days
|
||||
/var/lib/docker/volumes/felhom-controller-data/_data/data/settings.json {'immich': None, 'paperless-ngx': None, 'privatebin': None}
|
||||
offbox_target set: False
|
||||
restore holds: ['chaoscrash', 'chaosoom', 'chaosoomb', 'gokapi']
|
||||
0
|
||||
@@ -0,0 +1,95 @@
|
||||
# P3 — R-893: a failed off-site replay, measured on scratch guest 9202 (2026-10-19 evening)
|
||||
|
||||
Night brief 2026-10-19, Part 3 — **measure and design only; no restore code changed.** Tier 0 only (demo-hp, scratch
|
||||
guest 9202). Code read live: `felhom-controller/controller/internal/backup/offbox_reconstitute.go` at `15ea002`
|
||||
(controller 0.335.0). Architecture: `07-backup-architecture.md` §6.3 (the R-893 known limit, the failure ladder
|
||||
„replay → rollback → hold"); `09` §3 decision 192 (D8: first hold, then „put back exactly as it was").
|
||||
|
||||
## How it was reproduced (the product path, one forced error)
|
||||
|
||||
- 9202 put on controller **0.335.0** for the run (was 0.331.0; put back after). Hub is OFF on 9202.
|
||||
- A **scratch off-site target**: an SFTP server container (`atmoz/sftp:alpine`, `r893-sftp`, port 2222) on 9202 itself,
|
||||
repo on 9202's own disk (`/var/lib/felhom/r893-sftp`, nvme-scratch — not `pve/data`). Configured through the
|
||||
household's own endpoint `POST /backup/offbox/config` (a throwaway ed25519 key, the server's host key pinned),
|
||||
escrow confirmed with the legacy manual endpoint (9202 has no agent), only `rallly` switched on.
|
||||
- Throwaway database app: **Rallly 4.15.3 + postgres:18-alpine**, the DB in the named volume
|
||||
`rallly_rallly_postgres_data` (deployed through the product, `box_walk.deploy`).
|
||||
- Markers in the real database (positive observables): table `r893_marker` with one row per phase, plus tables that
|
||||
exist only at certain times.
|
||||
- The **forced error**: one line `SELECT r893_forced_replay_failure();` put into the PREPARED scratch copy of the
|
||||
snapshot's dump, just before its completion marker (so the completeness gate still passes). The real
|
||||
`ReconstituteFromOffsite` then replaced the volume, started the DB alone and failed the replay — i.e. the replay
|
||||
failed **after the volume was replaced**, as the brief asked.
|
||||
|
||||
## Timeline (UTC)
|
||||
|
||||
| Time | Act | File |
|
||||
|---|---|---|
|
||||
| 18:45 | baseline: 9202 on 0.331.0, no off-site target, settings + off-box folder saved | `00-baseline.txt` |
|
||||
| 18:48 | Rallly deployed; marker **A** | `01-marker-A.txt` |
|
||||
| 18:51:19 | off-site run → snapshot 1; then marker **B** + table `r893_after` | `02-snapshot-and-marker-B.txt` |
|
||||
| 18:53 | table `r893_only_old` + marker **C**; off-site run → **snapshot 2** (18:54:29, `93dbb37d`) | `04-before-snapshot-2.txt` |
|
||||
| 18:54:53 | after snapshot 2: `r893_only_old` DROPPED (like a table a newer version removes), marker **D**, table `r893_new2` — the live state the household has | `05-live-before-restore.txt` |
|
||||
| 18:54:59 | full restore prepared into scratch (66.1 MB); the failing line injected | `03-forced-failure.txt` |
|
||||
| 18:55:42 | `POST /backup/offbox/reconstitute` (confirm) | `06-reconstitute-log.txt`, `09-controller-log-window.txt` |
|
||||
| 18:55:43 | undo dump of the LIVE db written (`pre-restore-…-rallly-postgres.sql`, 36 tables) | |
|
||||
| 18:55:55 | the snapshot's volume put in place (`Restored 1 Docker volume(s)`) | |
|
||||
| 18:55:57 | replay FAILED (`exit status 3`, single transaction — nothing of it applied) | |
|
||||
| 18:55:58 | rollback: the undo dump loaded into the database **that now sits on snapshot 2's volume** — „rollback complete" | |
|
||||
| 18:55:58–59 | hold SET (`reason: restore_mixed`), app stopped | |
|
||||
| 18:56:42 | the held volume read through a throwaway COPY (the held state untouched) | `07-after-failed-replay.txt` |
|
||||
| 18:57 | household start → **409 held** | `08-app-state-and-start.txt` |
|
||||
| 18:58–19:01 | teardown | `10-teardown.txt` |
|
||||
|
||||
## What the failed replay left
|
||||
|
||||
| | Live before the restore | After the failed replay + rollback |
|
||||
|---|---|---|
|
||||
| marker rows | A, B, C, D | A, B, C, D (the newest rows — the undo dump won) |
|
||||
| `r893_new2` (written after snapshot 2) | present, 1 row | present, 1 row |
|
||||
| `r893_only_old` (in snapshot 2 only, dropped after) | **absent** | **present, 1 row** — left from the snapshot's volume |
|
||||
| public tables | 36 | **37** |
|
||||
| definition on disk | rallly 4.15.3, postgres 18 | rallly 4.15.3, postgres 18 (unchanged — same version; the R-1106 password overlay was empty, so the definition did not move) |
|
||||
| app | running | **stopped, held** (`restore_mixed`); a start answers 409 „held" |
|
||||
| other volumes / files | — | none for Rallly (its only volume is the DB's) |
|
||||
|
||||
So the rollback gives the **newest rows on top of the older volume**: everything the undo dump knows is exact, and
|
||||
whatever only the older volume had stays behind (the dump's `--clean --if-exists` drops only the objects IT names).
|
||||
That is the R-638 / R-893 mix, now measured, not read. The 0.304.0 hold works as built: the app does not start on it.
|
||||
|
||||
**The sentences the household sees:**
|
||||
- the restore page (`err.backup.db_restore_failed_held_mixed`): „Az adatbázis a visszaállítás előtti állapotban van,
|
||||
de az alkalmazás fájljai és többi adata már a mentésből származnak, ezért nem illenek össze." — **measured false on
|
||||
both halves for this app**: the database is NOT as it was (a table from the backup stayed), and Rallly had no files
|
||||
or other data from the backup (only its database volume moved).
|
||||
- the start refusal: „…félúton megszakadt: az adatai részben a mentésből származnak, részben nem. Az alkalmazás
|
||||
biztonsági okból leállítva marad." — true.
|
||||
|
||||
## Not measured tonight (said plainly)
|
||||
|
||||
- The **version-change** branch (the snapshot's older definition written, then the live one written back by
|
||||
`holdAppAfterMixedRestore`). It needs an app installed at an older catalog step and updated before the restore; the
|
||||
drill catalog that could do it is a git repo this helper was told not to commit to. The branch is pinned by
|
||||
`r893_mixed_restore_test.go`; it ends in the same hold.
|
||||
- What support does next: no product action releases a `restore_mixed` hold with the data made right — that is the
|
||||
gap option A closes.
|
||||
|
||||
## Side observations
|
||||
|
||||
- The manual off-site run's pre-push dump leg dumped and restarted **every** app on the box (paperless, immich), not
|
||||
only the one switched on (log: `App-data backup completed: 3 databases … 4 volume dump(s)`). By design (the
|
||||
coherent pair), outside R-893's scope.
|
||||
- 9202's address moved (DHCP: .114 → .121); `box_walk.py`'s 9202 row still names .114 — run with `BASE=`.
|
||||
|
||||
## Teardown (three layers)
|
||||
|
||||
- **Machine (9202):** Rallly removed through the product (`volumes_removed: rallly_rallly_postgres_data`, its backups
|
||||
and the off-site scratch gone); `r893-sftp` container, `atmoz/sftp:alpine` and `alpine:3` images (pulled tonight),
|
||||
the repo folder, the key, all `/root/r893-*` and `/tmp/r893-*` removed; `settings.json` and the `offbox/` folder put
|
||||
back from the copies taken at 18:45 (off-site target unset, no per-app off-site switch, restore holds as before:
|
||||
chaoscrash, chaosoom, chaosoomb, gokapi); controller back on **0.331.0** (healthy, 0 `[ERROR]`), the 0.335.0 image
|
||||
removed from 9202. The template folder `/opt/docker/stacks/rallly` (catalog files + an old `hold-logs`) was there
|
||||
before and stays.
|
||||
- **Host (demo-hp):** only `/tmp/r893-*` copies, removed. Nothing on `pve/data`.
|
||||
- **Hub:** nothing. 9202 has no hub; the hold's events were dropped on the box („notifier disabled (no hub
|
||||
configured)").
|
||||
@@ -0,0 +1,51 @@
|
||||
# R-893 — after a failed off-site replay: put the app back exactly as it was? (design note, 2026-10-19)
|
||||
|
||||
Status: **design only, for the operator.** Measured first (`README.md` in this folder, 9202, controller 0.335.0).
|
||||
Architecture: `07` §6.3 (the failure ladder „replay → rollback → hold"), `09` §3 decision 192 (D8: first hold —
|
||||
built in 0.304.0 — then „put back exactly as it was"). No restore code was changed.
|
||||
|
||||
## What the measurement showed (plain words)
|
||||
|
||||
An off-site restore of one app replaces the app's database volume with the backup's, then loads the backup's
|
||||
database copy. When that load fails, the box puts the household's own database copy (taken a minute before) back
|
||||
**on top of the backup's volume**. Measured on Rallly: all the newest rows came back, but a table that only the backup
|
||||
had **stayed behind** (37 tables instead of 36). The app is then held stopped, and a start is refused — that part
|
||||
works. But nothing in the product can make the data right again; the household waits for support. And the restore
|
||||
page tells them the database „is back as it was before the restore" — measured not true.
|
||||
|
||||
## The question
|
||||
|
||||
**When an off-site restore of one app fails half-way, should the box put the app back exactly as it was and start
|
||||
it, or keep holding it stopped for support?**
|
||||
|
||||
## The options
|
||||
|
||||
| | A — keep a copy of the app's volumes, and put it back | B — keep holding; only make the database part exact |
|
||||
|---|---|---|
|
||||
| What happens | After the undo dump and the stop, the box saves the app's live volumes (the same volume-dump helper the nightly copy uses) and keeps the live definition (already done since 0.304.0). On a failed replay it puts those volumes and the definition back — the state from one minute before — and starts the app. The undo dump stays as a second way back. | On a failed replay the box empties the database volume before it loads the undo dump (a rebuilding load, on the rollback path only). Then the database is exactly as it was. The app stays held while files or other volumes moved; an app whose only change was its database volume can start. |
|
||||
| Disk | One extra copy of the app's volumes for the length of the restore (Rallly: 69 MB, made in about 1 s). A free-space check refuses BEFORE the stop when the copy does not fit — then nothing is touched. | None. |
|
||||
| Code | ~1 session: the save step, the put-back on every failed branch, the fit check, tests; then one 9202 drill (tonight's setup can be reused). | ~½–1 session: the emptying step and its tests; the R-638 option-B loader, but only on this one path. |
|
||||
| Risk | Low: both helpers exist and already run every night. A put-back that fails leaves the hold, as today. | Higher: the volume is emptied before the undo is loaded, so a bad undo dump leaves an empty database (the backup's volume is gone too). Files and other volumes are still the backup's. |
|
||||
| What the household sees | „The restore did not finish. Nothing changed: the app runs as before." No support call. | Usually still „the app is stopped; contact us" — fewer cases than today, but the same support call. |
|
||||
|
||||
## Pick: A
|
||||
|
||||
It is the direction the operator already ruled (D8, second half), it makes the restore page's promise true, and the
|
||||
measured cost is small: the extra disk is one copy of the app's own volumes, for minutes, and a fit check refuses
|
||||
cleanly when there is no room. B removes only one of the mixed parts and adds a new way to lose the database.
|
||||
|
||||
**One small fix either way, now:** the sentence `err.backup.db_restore_failed_held_mixed` says the database is back as
|
||||
it was and that files and other data came from the backup. Measured: neither is reliably true (Rallly had no files or
|
||||
other volumes from the backup, and its database was not as before). It should say only what is certain: the restore
|
||||
stopped half-way, the app is held stopped, and nothing is lost yet.
|
||||
|
||||
**If the operator does nothing:** a failed off-site restore keeps ending in a held app that only support can bring
|
||||
back, and the page keeps telling the household their database is back as it was.
|
||||
|
||||
## First slice and its red test (for A)
|
||||
|
||||
- Red test beside `r893_mixed_restore_test.go`: a failed replay after a volume replay must call the volume put-back
|
||||
with the live copy, write the live definition back, and START the app (no hold); with too little free space for the
|
||||
copy, the restore must refuse with `StopStack` called 0 times.
|
||||
- Then the 9202 drill (this folder's `README.md`, same steps): after the forced failure, the table count and the
|
||||
markers must equal the live state before the restore (36 tables, A–D, no `r893_only_old`), and the app must run.
|
||||
@@ -26,6 +26,14 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-19 (evening/night brief) — no stop for a busy host, the scanner on every push, the ArgoCD prune danger gone (full text of each row: `git show f92fa5a8:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|---|---|---|---|
|
||||
| **R-921** | **When the off-site tier answers BUSY, the household still gets a short stop of every app — with no copy.** (P3) | CLOSED 2026-10-19 — **the measured case is now seen before the stop** (operator answer „add one field", 19:2x): the agent serves its host-wide gate as `host_busy` on `GET /backup/status` (agent `56aabc7`, committed, NOT released — it reaches a box with the next agent release, STATUS's morning question), and controller 0.335.0 (the floor since 18:39 UTC) reads it before any stop: held → no app stopped, each due tier recorded as contention (15-minute skip, one „BLOCKED" alarm after 3 h). An agent without the field keeps the old order (proven live: demo-hp's agent answers no `host_busy`). The race (free at the read, busy at the start) keeps the old short stop, resumed at once. Tests red-proved: agent `TestBackupStatus_R921_HostBusy`; controller `TestR921_NoStopWhileTheHostIsBusy` (without the branch: 3 apps stopped), `…SkipsStillAlarmAfterTheCeiling`, `…OtherAnswersKeepTheOldOrder`, `TestHostBusy_ReadsTheAgentsGate`. A live busy state was not made on 9201 (it needs the new agent there). | `audits/night-2026-10-19/P9/` (release), controller `15ea002`, agent `56aabc7` |
|
||||
| **R-953** | **A full ArgoCD sync of the `felhom` app with prune would DELETE three live Secrets and so break the hub and Umami.** (P3) | CLOSED 2026-10-19 — **not reproducing: ArgoCD no longer tracks the three Secrets.** Read back 18:21 UTC (read only, no sync): app `felhom` Synced/Healthy at `a28f162c`, 25 tracked resources, none of them a Secret, none `requiresPruning`, `Deployment/umami` Synced; the three Secrets carry no labels and no annotations (no tracking id). A sync with prune touches only tracked resources, so the next deploy does nothing with them. No manifest change was possible or needed (a `Prune=false` annotation would live on the untracked live object). How the tracking marks went after 2026-10-12 is not recorded. The build-deploy skill's „sync only the hub" line stays. | `audits/night-2026-10-19/P4/r953-readback.txt` |
|
||||
| **R-986** | **No dependency or secret scanner runs on any push, so a known-vulnerable library or a committed key is found only by a review like this one.** (P3) | CLOSED 2026-10-19 — **the warn-first gate runs on every push in all four repos.** `felhom.eu/scripts/security_scan_gate.py` (shared, never copied) in each runner's FAST set — pre-push hook and CI: govulncheck v1.8.0 (called vulnerabilities only) on each Go module, gitleaks 8.24.3 on the pushed commits with the default rules pinned; hits listed, exit 0. Accepted hits per repo in `.security-accepted` (gitleaks by rule + a hash of the line + file), reason per line. CI fetches the pinned gitleaks with both sha256 checked (seen in CI 2108: 42 hits, all accepted); CI has no Go, so govulncheck is NOT CHECKED there, out loud — the hook runs it. 8 decoys in scratch repos, red-proved. First scan of today's `main`: 0 called vulnerabilities, 0 real secrets, every gitleaks hit with a verdict. The govulncheck pin moved v1.1.4 → v1.8.0 (v1.1.4 cannot read Go 1.27 code). Warn → fail: R-1168. | `audits/night-2026-10-19/P7/triage.md`, `SCANNERS.md` §5 |
|
||||
|
||||
## 2026-10-19 (third day brief) — the passphrase locks once the box uses its own key, every ep0 household gets a prune job, demo-hp's address (full text of each row: `git show 97f27a75:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|
||||
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user