Night 2026-10-19: register 161 -> 161 (closed R-921, R-953, R-986; opened R-1168..R-1170; R-1001/R-1011/R-1010/R-1070/R-1167 READY, R-893 READY with the measured design note -> D20, R-1083 gap); morning list in STATUS; 03/05/07/11 updated; R-893 evidence
gates / gates (push) Successful in 13m5s
gates / gates (push) Successful in 13m5s
This commit is contained in:
@@ -232,7 +232,11 @@ The controller (in its LXC) reaches the agent (on the host) over the local bridg
|
||||
- `POST /rollback` — roll *this* guest back to a named snapshot (post-deploy failure recovery).
|
||||
- `POST /backup` — request a backup-now of *this* guest (enqueued; non-destructive).
|
||||
- `GET /backup/due` — whether a policy-scheduled backup is due for *this* guest, so the controller can quiesce then call `POST /backup` (the app-consistent path, §8).
|
||||
- `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI.
|
||||
- `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI. **`host_busy`
|
||||
(R-921, agent Unreleased `56aabc7`):** `GET /backup/status` also says whether the host-wide heavy-operation gate is
|
||||
held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off), the
|
||||
state in which `POST /backup` answers 409; absent when the gate is not wired. The controller (≥ 0.335.0) reads it
|
||||
before it stops any app. A read, never a reservation.
|
||||
- **Crash-boot fact (R-856, agent v0.149.0):** `GET /host/crash-guard` — the host crash guard's last-boot record
|
||||
(`present`, `last_boot_at`, `last_boot_unclean`, `tripped`) read from `/var/lib/felhom-crash-guard/state.json`; a missing
|
||||
or garbled file answers 200 `present:false`, an older agent 404 — both read as a normal boot. The controller waits
|
||||
@@ -399,7 +403,10 @@ per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a ba
|
||||
> space on …" — so the hub raises `restore_test_failed`; it is never a pass and never dropped.
|
||||
> - **Leftovers on a timer.** A failed scratch teardown and the stale-lock sweep (R-673) run every 10
|
||||
> minutes, not only at agent start; the sweep holds the one-heavy-operation gate; after 3 failed
|
||||
> teardown tries the operator is told.
|
||||
> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent Unreleased
|
||||
> `56aabc7`):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's
|
||||
> snapshot already gone), it retries once with Proxmox's own `force=1` on the same API call (no sudoers line, no
|
||||
> root command); any other error is returned unchanged, and a failed forced delete is never forced again.
|
||||
> - **A thin pool ≥ 90 % requests an immediate report**; the hub judges a thin pool on the worse of data
|
||||
> and metadata, critical at 90 %, one alarm per pool per 6 hours (`08` §6.2).
|
||||
>
|
||||
|
||||
@@ -90,7 +90,11 @@ Evolves the existing staleness checker (60s **cadence**, a **configured** thresh
|
||||
2026-09-17, **45 m** since operator ruling A on R-549; OK under it, down at 2× = 90 m; today: controller-report recency → `node_stale`/`down`/`recovered`):
|
||||
|
||||
- **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's
|
||||
liveness signal; a silent agent = the box is gone (the critical alert).
|
||||
liveness signal; a silent agent = the box is gone (the critical alert). **A hub restart does not re-mail a known
|
||||
silence (R-1001, hub 0.162.0 — built 2026-10-19, not yet deployed):** a host whose newest `host_stale`/`host_down`
|
||||
event is newer than its last report is seeded with that state at start, silently; `restore_test_stale` reads its edge
|
||||
from the DB on the first sweep (a signal for the tier inside the read window and newer than its last proof). A failed
|
||||
lookup lets the alarm through.
|
||||
- **Guest up/down comes from the host report's per-guest status** — authoritative, every poll, faster
|
||||
than waiting for a guest report to go stale.
|
||||
- **Guest-report recency = secondary** app-level signal.
|
||||
@@ -134,7 +138,11 @@ walk is green; the golden is baked weekly and vouched by the operator; a new box
|
||||
itself to the floor (R-1052: not yet proven end to end). Setting the global floor today needs the operator page with the
|
||||
operator's password — there is no scoped route for CC (R-1051). **An agent can be vouched mid-week; the golden stays
|
||||
weekly** (hub v0.158.0): the R-120 golden guard judges only a CHANGED golden selection, so a Day-0 save that keeps the
|
||||
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet.
|
||||
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. **The agent list
|
||||
offers real agent packages only (R-1011, hub 0.162.0, not yet deployed):** the sha lookup asks for exactly the
|
||||
`felhom-agent` file (the golden: `golden.tar.zst`) and never falls back to another file, so a step bundle
|
||||
(`0.156.1-step1`, only the config bundle) is left out of the list, and a vouch of one is refused
|
||||
(`artifact_file_missing`).
|
||||
|
||||
**The controller floor and its agent requirement (hub v0.112.0, R-472).** The managed controller floor
|
||||
rides the report ACK. `store.ResolveManagedFloor` decides per box whether to serve it, from three
|
||||
|
||||
@@ -728,7 +728,13 @@ definition is written or a named volume replaced — a failed placement, volume
|
||||
the live definition back and HOLDS the app stopped (`restore_mixed`, operator-cleared); the household reads that it
|
||||
needs our help; the operator gets a `backup_run_failures` line. The hold is persisted before the stop. Placed files
|
||||
ALONE (no version change, no volume replaced) still restart after a successful rollback (`TestR379_ScenarioA`): that
|
||||
mix, and the undo of everything, is option A — „put back exactly as it was", the next slice on R-893.
|
||||
mix, and the undo of everything, is option A — „put back exactly as it was", the next slice on R-893. **`[FACT]`
|
||||
2026-10-19 (measured on 9202, controller 0.335.0, `audits/night-2026-10-19/P3/`):** a forced replay failure after the
|
||||
volume was replaced left the NEWEST rows (the undo dump) on top of the backup's volume — a table only the backup had
|
||||
stayed (37 tables, the live app had 36); the definition did not move (same version); the app was held and a start
|
||||
answered 409. The restore page's „the database is back as it was" is false for it. Design note with the two options and
|
||||
the pick (A: a copy of the app's live volumes, put back on failure — one copy's disk for the restore's length, a fit
|
||||
check before the stop): `audits/night-2026-10-19/P3/design-R-893.md`; the decision is on `STATUS.md`.
|
||||
|
||||
**[DESIGN] 2026-08-22 — the failure ladder of a database restore: replay → rollback → hold.**
|
||||
Recorded here rather than only in a closed register row, because a decision that survives only inside
|
||||
@@ -826,8 +832,12 @@ parts above, not a measured press).
|
||||
demo-hp: the off-site tier stopped every app for ~1 min and the agent then refused it (BUSY: the night OS step held the
|
||||
agent's host-wide lock). Before stopping anything the controller now asks the agent which tiers have a job in flight
|
||||
(`GET /backup/status`); another tier's job running or snapshotted → no stop, the tier stays due. A refusal after the
|
||||
stop still resumes the apps at once (pinned by test). **Not covered:** the host-wide busy lock (OS step, restore test,
|
||||
fstrim) — the agent serves it on no endpoint, so the measured case needs an agent change too (R-921, LEFT).
|
||||
stop still resumes the apps at once (pinned by test). **The host-wide busy lock (2026-10-19, operator answer „add one
|
||||
field"):** the agent serves it as `host_busy` on `GET /backup/status` (agent Unreleased `56aabc7`); controller 0.335.0
|
||||
(the floor) reads it before any stop — held → no app stopped, each due tier recorded as contention (retry in 15 min,
|
||||
one „BLOCKED" alarm after 3 h), exactly the verdict a 409 gets. An agent without the field keeps the old order (stop,
|
||||
ask, resume at once). The check is a read, not a reservation: a gate taken between the read and the start costs one
|
||||
short stop, resumed at once. The measured case is closed on a box once it runs the next agent release.
|
||||
**Measured 2026-10-05 on demo-hp (9 apps, controller v0.295.0):** „Mentés most" stopped the apps at 09:19:08Z, the
|
||||
local tier ran 09:19:29–09:24:09, the PBS tier was busy (the controller logged a retry in 15 min; no second stop was seen in the next 55 min), the last app was back at 09:24:55Z — the
|
||||
longest stop **5 min 47 s**, for the local tier alone. The button text and its confirm (v0.296.0) give both
|
||||
|
||||
@@ -724,7 +724,11 @@ Evidence: `audits/os-host-lane-2026-10-04/` (parts A–G).
|
||||
skips the host step.
|
||||
- **The host health rule** (`HostHealthVerdict`, pinned in `internal/osupdate`): `felhom-agent`, `pveproxy`,
|
||||
`pvedaemon`, `pvestatd` and `pve-cluster` are `active`; the customer guest runs; the guest health rule (§8.1)
|
||||
passes; the tunnel is `running` — an `unknown` tunnel does not fail it, `not_running` does. Same 5-minute wait.
|
||||
passes; the tunnel is `running` — `not_running` and `unknown` fail it (CORRECTED 2026-10-19: this line said
|
||||
„an `unknown` tunnel does not fail it"; the code always failed it, which is what Tester4 hit, R-1070). **Since agent
|
||||
Unreleased `56aabc7` (R-1070):** a guest with NO cloudflared container reads `absent` (Docker 29's lower-case
|
||||
„no such object" used to fall through to `unknown`), and `absent` passes when the reading from before the step also
|
||||
listed no cloudflared container (Docker answered) — a box with no tunnel set up. Same 5-minute wait.
|
||||
- **Separate approved sets:** host and guest releases are separate (`os-host-…`, `os-guest-…`), each by the same rule
|
||||
(24 h, 1 night of THAT layer, every ring-0 box). Ring 1 receives `host_release` beside `release`.
|
||||
- **Reboot needed:** reported when PID 1 or `lxc-start` maps a replaced file, with the date of the first scanned
|
||||
|
||||
Reference in New Issue
Block a user