Files
felhom.eu/REPORT-night-2026-10-19.md
T

11 KiB

REPORT — evening/night brief 2026-10-19 (one stop less per night, no false tunnel error, R-893 measured, hub and agent fixes ready, a scanner on every push, ep0's first weekly clean-up)

In plain words

  • Morning list: deploy hub 0.162.0? · release agent 0.161.0? · publish installer 1.35.0? · decide D20 (R-893).
  • One app stop less per night (R-921): the controller (0.335.0, the floor) now asks the agent „are you busy?" before it stops the apps. The agent's answer is written but not released, so it starts working with the agent release.
  • No false error for a box without a tunnel (R-1070): the real cause was Docker 29 writing its „not found" message in small letters. Fixed in the agent (not released).
  • A failed off-site restore, measured (R-893): the box puts the newer rows over the backup's older volume; a table only the backup had stays. The app is held stopped. Two options written; pick A (save the app's volumes first).
  • Hub fixes built, not installed: no repeat mail after a restart; the agent list offers only real agent packages. The ArgoCD danger to three hub secrets is already gone (ArgoCD no longer tracks them).
  • Installer message written, not published: a re-run on a live box says „delete the old host first", EN and HU.
  • A scanner on every push, in all four repos (warn-first). First scan: no real secret, no called weakness.
  • The scratch box cannot get its agent key without an agent change (R-1083): the exact gap is written down.
  • ep0's first weekly clean-up: read after 05:30 — see Part 10.

Decisions CC took: none beyond the brief. The operator answered one question (R-921, 19:2x): „add one field".

Part 1 — R-921 (07 §6.4; 03 local API) — DONE (live once the agent ships)

  • No existing agent answer served the host-wide gate (read: /backup/due, /backup/status, the 409 shape). Operator: add a read-only field to the existing route, no new route.
  • Agent 56aabc7: host_busy {busy, what} on GET /backup/status. Controller 0.335.0: precheckSkips → held → no stop, contention for each due tier (15-minute skip, one „BLOCKED" alarm after 3 h); free / older agent / failed read → the old order; the window's own tier in flight → attach as before.
  • The race (free at the read, busy at the start): one short stop, resumed at once — today's behaviour, pinned.
  • Tests, red-proved: agent TestBackupStatus_R921_HostBusy (assignment removed → free gate: host_busy = <nil>); controller TestR921_NoStopWhileTheHostIsBusy (branch removed → 3 apps stopped, the start refused), …SkipsStillAlarmAfterTheCeiling, …OtherAnswersKeepTheOldOrder (4 rows), TestHostBusy_ReadsTheAgentsGate.
  • Live: demo-hp's agent answers no host_busy (read inside 9201) — the old order, as designed. A live busy state was not made (it needs the new agent on the box). R-921 closed per the brief.

Part 2 — R-1070 (11 §8.2) — DONE in the agent (not released)

  • Where: felhom-agent/internal/osupdate/leg.go HostHealthVerdict (the tunnel check) and internal/hub/cloudflared.go ClassifyTunnel. Measured on demo-hp: docker inspect of a missing name prints error: no such object: … (Docker 29); the match wanted No such object → unknown → the host rule failed.
  • Fix: case-insensitive match → new state absent; the host rule passes absent only when the reading from before the step (Docker answered) also had no cloudflared; not_running and unknown still fail. Hub 0.162.0 shows absent as „none set up". 11 §8.2 had said unknown does not fail — corrected (the code always failed it).
  • Tests, red-proved: TestHostHealthVerdict_R1070_NoTunnelSetUp (7 rows), three TestClassifyTunnel rows, TestSystemRows_R1070_AbsentTunnelIsNotAWarning.

Part 3 — R-893: measured and designed, no restore code changed

Helper on 9202 (fences carried in full). Evidence and the note: documentation/audits/night-2026-10-19/P3/.

  • Rallly 4.15.3 + postgres 18; a scratch SFTP off-site on 9202's own disk; one failing SQL line in the prepared copy of the dump, so the replay failed AFTER the volume was replaced.
  • What it left: the newest rows (the undo dump) over the backup's volume; a table only the backup had stayed (37 tables against 36); definition unchanged (same version); app held (restore_mixed), a start → 409.
  • The restore page's sentence is false on both halves for this app → R-1169.
  • Options: A — copy the app's live volumes before the replay, put them back on failure, start the app (one copy's disk for minutes; Rallly 69 MB in ~1 s; a fit check before the stop). B — empty the DB volume before the undo load (no disk; a bad undo copy leaves an empty database; usually still a support call). Pick: A (the operator's D8 direction). Not measured: the version-change branch (needs a drill-catalog commit).
  • Teardown: 9202 back on 0.331.0 as found, every throwaway removed (checked again by CC); host: /tmp only; hub: nothing.

Part 4 — hub 0.162.0, built, NOT deployed (05)

  • R-1001: start-up seeding from the DB for host_stale/host_down; restore_test_stale edge from the DB. Tests red-proved (each restart = a new checker over the same DB).
  • R-1011: exact-file sha (ErrNoFile); step bundles leave the list; a vouch of one → artifact_file_missing. Registry read: 6 step-bundle versions, every release holds felhom-agent. Tests red-proved.
  • R-953: no change needed — read back 18:21 UTC: ArgoCD tracks none of the three Secrets (no tracking mark), so the next deploy does nothing with them. P4/r953-readback.txt.
  • Image sha256:69285e640a9c5e4ce59b1fbb5cd94623fa47a71798e9d86fa0767ee6c69a6ac8 from a clean clone at d92a712e.

Part 5 — agent, committed, NOT released (03)

R-1010 (56aabc7): on „Failed to find logical volume …_vzdump" one retry with the API's force=1 (pvesh usage lists --force; the same token privilege; no sudoers line, no root command); never a second force. Red-proved. Plus R-921's field and R-1070. CHANGELOG „Unreleased". Not exercised live (it needs a half-removed snapshot on pve/data).

Part 6 — installer 1.35.0, written, NOT published

R-1167: die_live_box in EN and HU at the preflight, host-enroll and the controller.yaml fetch. Harness: a static case and two behaviour cases (the real functions against a local fake hub), red-proved on a mutated copy. CI red twice (2106, 2108): the CI runner has no curl; the behaviour cases now SKIP out loud there (5d0dfeac).

Part 7 — the scanner (R-986), warn-first — DONE

  • Shared scripts/security_scan_gate.py in all four runners' fast sets (hook + CI); .security-accepted per repo; 8 decoys in scratch repos, red-proved; CI fetches the pinned gitleaks (both sha256 checked — seen working in CI 2108).
  • govulncheck pin v1.1.4 → v1.8.0 (v1.1.4 cannot read the Go 1.27 code). CI has no Go: govulncheck NOT CHECKED there, out loud; the hook runs it.
  • First scan of today's main (P7/triage.md): govulncheck 0 called in all three modules. gitleaks history: controller 12, agent 15, catalog 1 (all as on 2026-10-12), felhom.eu 50 (+7: the usenet bench 5, a boot journal 1, a test 1). Today's tree: controller 4, felhom.eu 42, agent 15, catalog 0 — every one triaged: 0 real; accepted = throwaway demo/bench app values (the bench LXC 9401 stopped, the apps gone from 9202); false = test constants, public keys, an SSH fingerprint, a tunnel ID, hashes.

Part 8 — R-1083: the exact gap (no agent change)

The agent mints a per-guest token only inside --selftest=provision, for a guest it brings up from a golden in the same call. No product way exists for an existing guest. Row stays OPEN with the gap and two options.

Part 9 — controller 0.335.0, the floor

rc.1 on demo-hp 18:08 UTC (0 errors) → 0.335.0 sha256:262831f9… → per-box floors 18:22 → all three moved by 18:37 → box walk green (0 unhealthy, 0 refusals, catalog in sync; https-only 28 OK, 0 FAIL; „Fájlok ↗" one sign-in on demo-hp and Tester 1) → global floor 0.335.0 / MinAgent 0.158.1 at 18:39:26 UTC, read back; per-box floors cleared. i18n: no page changed, no fixture changed. Evidence P9/.

Part 10 — ep0's first weekly clean-up

PENDING at the time of writing (runs 03:30 UTC).

Versions, commits

  • Controller 0.335.0 (15ea002, REPORT c4b3b0a, scanner a03c25a). Hub 0.162.0 built (5f31da04). Agent 56aabc7, 84ef581 (not released). Installer 1.35.0 (d92a712e, 5d0dfeac, not published). felhom.eu scanner 15f74288, docs f92fa5a8. Catalog 87f5588, 57ddb20.
  • System poster: no listed fact changed tonight — none of the three.
  • Instruction/doc corrections: 11-os-updates.md §8.2 (the unknown tunnel sentence was false); nodes.md and nightly-rotation.md (9202 is DHCP, .121 now).

Rows

Register 161 before, 161 after (before Part 10). Closed 3: R-921, R-953, R-986. Opened 3: R-1168 (warn → fail), R-1169 (the false restore sentence), R-1170 (a vanished tunnel raises no alarm). Re-stated: R-1001, R-1011, R-1010, R-1070, R-1167 (READY, waiting on the release/deploy/publish), R-893 (READY, D20), R-1083 (the gap).

Observations

  1. The restore page's held-mixed sentence is false for the measured app. FILED: R-1169
  2. A tunnel container that disappears on a box with a tunnel configured raises no alarm. FILED: R-1170
  3. The scanner warns only. FILED: R-1168
  4. 11 §8.2 said an unknown tunnel does not fail the host rule; the code always failed it. NOT-A-FINDING: corrected in the session (the size rule); it is R-1070's cause.
  5. govulncheck v1.1.4 (the 2026-10-12 pin) cannot read Go 1.27 code. NOT-A-FINDING: the pin moved to v1.8.0 tonight, recorded in SCANNERS.md §5.
  6. The CI runner has no curl; two new harness cases failed CI 2106 and 2108. NOT-A-FINDING: fixed the same night (5d0dfeac), they skip out loud there.
  7. R-953's tracking marks went after 2026-10-12 with no record of who removed them. NOT-A-FINDING: the danger the row named is gone, nothing to fix.
  8. 9202's DHCP address moved (.114 → .121) and the walk named the old one. NOT-A-FINDING: fixed (box_walk.py reads it; two docs corrected).
  9. The brief said the installer already prints HU and EN; it printed English only. NOT-A-FINDING: the HU lines are new, named in the scripts CHANGELOG.
  10. The agent's backup cap (R-973) can also refuse a POST after the stop. NOT-A-FINDING: it answers 409, so the apps resume at once (the BUSY path), and it fires only after a flood of presses that each stopped the apps anyway.
  11. The manual off-site run dumped and restarted every app, not only the one switched on. NOT-A-FINDING: by design, the coherent app-data pair (07).
  12. On CI's depth-1 clone, gitleaks scans the whole tree (the root commit), ~100 s for felhom.eu. NOT-A-FINDING: a stronger check than the diff, all current hits accepted; recorded in CONTEXT.md.

Teardown

  • Machine: 9202 back as found (0.331.0, no throwaway left — checked); 9201 on the release 0.335.0; probe scripts removed.
  • Host: /tmp copies only, removed. Nothing on pve/data.
  • Hub: per-box floors cleared; nothing else created.
  • ep0: read only.