03 §12a: trust in Proxmox by the root CA copy is built, not released (R-1380); R-1396 filed; red-proofs
This commit is contained in:
1 parent
fe8247665d
commit
d809a1d444
5 files changed
+194
-14
No files matched your search
@@ -956,21 +956,68 @@ argument for §3's root-minimization and a small, auditable agent.
|
||||
the same name**; a node certificate is not made from a CA that ends in less than 14 days. Read from the code, not
|
||||
forced.
|
||||
|
||||
`[DESIGN — proposed, not built]` Two ways. The operator's ruling of 2026-11-11 13:17 (`09` §3 decision 297) names the
|
||||
first; the spike points to the second, so the choice is back with the operator (`STATUS.md` D80).
|
||||
**The ruling (D80, `09` §3 decision 303, operator 2026-11-14):** way B — the agent trusts Proxmox through a copy of the
|
||||
box's own root CA.
|
||||
|
||||
| | A — the agent follows the fingerprint | B — the agent trusts the box's own root CA (**CC's pick**) |
|
||||
`[FACT — built, not released]` (felhom-agent `main` after v0.174.0, the installer on `main` after `installer-v1.36.1`;
|
||||
2026-11-14 night; **no box has run it** — the proof on a box is a forced new certificate on demo-hp, twice.)
|
||||
|
||||
**1. The root step that keeps the copy** — `felhom-os-apply`, host-layer mode `pve-ca-copy`. It copies exactly
|
||||
`/etc/pve/pve-root-ca.pem` to exactly `/etc/felhom-agent/pve-root-ca.pem` (0644 root — a CA certificate is public),
|
||||
only if the source is ONE certificate with `CA:TRUE` that has not ended; written whole or not at all; „same" when
|
||||
nothing changed. The plan names nothing (T1); no source (T2); not one CA certificate (T3). **On a BYO host it runs
|
||||
too:** the bundle's policy keeps the OWNER'S settings out of our hands (the sshd drop-in is appliance-only), and this
|
||||
copy is the agent's own trust file in the agent's own folder — it changes nothing of the owner's.
|
||||
|
||||
**2. The move of a box that exists** — `internal/pvetrust`, at the agent's start, after a config bundle install, on a
|
||||
timer (hourly until moved, daily after), and on every call to Proxmox that fails on trust:
|
||||
|
||||
| Step | What | If it fails |
|
||||
|---|---|---|
|
||||
| How | A root helper reads the node certificate after a mismatch and gives the agent the new fingerprint | The installer puts a copy of `/etc/pve/pve-root-ca.pem` where the agent can read it, sets `ca_file`, leaves the fingerprint empty |
|
||||
| New agent code | yes: the helper, a sudo rule, the re-pin logic, its tests | none for the trust itself (the key exists and is proven) |
|
||||
| Trust | one exact certificate | every certificate this box's own CA signs; the CA's key is root-only on the same box |
|
||||
| Next hand step | none, if the helper is right | none until the copy's own end date (10 years from install); the copy must be refreshed after Proxmox renews the CA |
|
||||
| Boxes that exist | one root step each (the signed config bundle) | the same |
|
||||
| 1 | the root step makes or refreshes the copy | nothing changes; cause `ca_copy_unavailable` |
|
||||
| 2 | **the proof:** one real call to Proxmox with the copy ALONE, while the pinned fingerprint is still the configured trust | nothing is written, the fingerprint stays; cause `ca_copy_does_not_verify` (no cause when Proxmox did not answer at all — asked again within the hour) |
|
||||
| 3 | ONE write of `agent.json`: `proxmox.tls.ca_file` set, `fingerprint` cleared, every other key kept; read back and checked | the old bytes are put back, the fingerprint stays; cause `config_not_written` |
|
||||
| 4 | the RUNNING client takes the new trust (`proxmox.Client.SetTLS`) — no restart | it keeps the old trust until the next start |
|
||||
|
||||
Why B: it is the standard way to trust a private CA, it is proven on a box, and it removes the failure with no new
|
||||
privileged code. What B still needs before it is built: the root step that refreshes the copy (and a test that an
|
||||
old copy raises an alarm that names the cause), and a rule for what the agent does if both keys are set.
|
||||
Whichever way is built, the mismatch must raise an alarm that names the cause.
|
||||
**A box is never left with neither way working:** the fingerprint is cleared only after a call worked with the copy.
|
||||
**Who writes the config:** the agent itself. `agent.json` is the agent's own file (0600) in a root-owned folder, so it
|
||||
is rewritten in place after a recovery copy (`agent.json.pre-pve-trust`) was parked in the agent's state folder — the
|
||||
same way the backup-target move and the PBS seed write it. No root step touches it. Never moved: an operator's own
|
||||
`ca_file`, a box with `insecure_skip_verify`, a config that is not a file.
|
||||
|
||||
**3. The installer** (`resolve_pve_trust`): a new box gets the copy, `ca_file`, and NO fingerprint — after a TLS
|
||||
handshake to `127.0.0.1:8006` verified with the copy alone (no system roots, the IP checked). If the source is not one
|
||||
CA certificate, the copy cannot be written, or the handshake does not verify, the box gets the pinned certificate as
|
||||
before, and the agent moves by itself later.
|
||||
|
||||
**4. The cause is named** (`proxmox.TrustProblem`): `pinned_cert_changed`, `ca_mismatch`, `cert_expired`,
|
||||
`ca_file_unreadable`, `name_mismatch`, and from the move: `ca_copy_unavailable`, `ca_copy_does_not_verify`,
|
||||
`ca_copy_ends_soon` (30 days), `config_not_written`. Each is an ERROR line in the agent's journal and rides the host
|
||||
report as the `proxmox_trust` stanza (`mode`, `problem`, `detail`, `ca_not_after`, `ca_days_left`, `moved_at`).
|
||||
**Two gaps, both in R-1396:** the hub does not read the stanza yet (no operator alarm from it), and a box that is
|
||||
fully blind sends no host report at all, so that case still reaches the operator only as „host stale".
|
||||
|
||||
**5. Both keys set.** The order in `proxmox.TLSConfig.build` is unchanged: skip-verify, then the fingerprint, then
|
||||
`ca_file`, then the system's roots. **A set fingerprint wins over a CA file.** So: the move clears the fingerprint
|
||||
LAST, in the same write that sets `ca_file`; the installer writes a fingerprint only when the copy was not proven; and
|
||||
a config that holds both is a pinned box — the agent treats it as „not moved" and runs the move.
|
||||
|
||||
**What changes for a pinned box when Proxmox makes a new certificate:** the first failed call names
|
||||
`pinned_cert_changed`, runs the order above, and the next call works — the box is blind for one report cycle at most,
|
||||
not until a person repairs it. `runbooks/RUNBOOK-agent-proxmox-pin-repair.md` stays the hand repair for an agent older
|
||||
than this.
|
||||
|
||||
**The root CA's own renewal** (same key, same name, when under two years are left): the daily run refreshes the copy,
|
||||
proves it and reloads it in the running client. The old copy keeps verifying until its own end date, so nothing is
|
||||
urgent. Read from Proxmox's code and proven against a test CA, **not forced on a box**.
|
||||
|
||||
`[NOT PROVEN]` Everything on a box. Named for the attended session: a new box by the installer (CA only); an old box
|
||||
moves at the agent's start; a forced new certificate on a moved box (no failed call) and on a pinned box (one failed
|
||||
cycle, then healed); the wrapper mode on a BYO host; the torn-write case of step 3 (a power cut inside one small
|
||||
write — the parked copy is the hand recovery, as for every other writer of this file).
|
||||
|
||||
`[DESIGN — the other way, not chosen]` A — the agent follows the fingerprint through a root helper after a mismatch:
|
||||
new privileged code, a sudo rule and re-pin logic; trust stays on one exact certificate.
|
||||
|
||||
## 13. Open items / what this unblocks
|
||||
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
== MUTATION: the config is written BEFORE the proof
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestOldBox_MovesAndTheFingerprintIsClearedOnlyAfterTheProof (0.01s)
|
||||
pvetrust_test.go:254: order = [copy probe(fp="",ca=true) while the file says fp="" apply(fp="",ca=true)]
|
||||
--- FAIL: TestProofFails_TheFingerprintStays (0.00s)
|
||||
pvetrust_test.go:292: the config was written although the proof failed
|
||||
== MUTATION: the proof's answer is ignored
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestProofFails_TheFingerprintStays (0.00s)
|
||||
pvetrust_test.go:288: the trust changed although the proof failed
|
||||
== MUTATION: the write keeps the fingerprint
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestOldBox_MovesAndTheFingerprintIsClearedOnlyAfterTheProof (0.00s)
|
||||
pvetrust_test.go:250: the box did not move: &{Mode:fingerprint Problem:config_not_written Detail:the config did not read back as written (<nil>) — the old config was put back CANotAfter:2036-08-17T15:44:42Z CADaysLeft:0x21c7815067
|
||||
--- FAIL: TestPinnedBox_NewCertificate_HealsWithNoRestart (0.01s)
|
||||
pvetrust_test.go:319: the box is still blind after the move: proxmox: GET /version: Get "https://127.0.0.1:41039/api2/json/version": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint ([copy probe(
|
||||
== MUTATION: a failed call on trust starts nothing
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestPinnedBox_NewCertificate_HealsWithNoRestart (0.01s)
|
||||
pvetrust_test.go:319: the box is still blind after the move: proxmox: GET /version: Get "https://127.0.0.1:33489/api2/json/version": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint ([])
|
||||
== MUTATION: a refreshed copy is not reloaded by the running client
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestCARenewedWithTheSameKey_TheCopyIsRefreshedAndReloaded (0.00s)
|
||||
pvetrust_test.go:350: a refreshed copy must be proven and reloaded: [copy probe(fp="",ca=true) while the file says fp=""]
|
||||
== MUTATION: an operator's own CA file is replaced
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestNeverMoved (0.01s)
|
||||
pvetrust_test.go:412: own CA file: moved ([copy probe(fp="",ca=true) while the file says fp="" apply(fp="",ca=true)])
|
||||
== MUTATION: a box with checks switched off is moved
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestNeverMoved (0.01s)
|
||||
pvetrust_test.go:412: insecure: moved ([copy probe(fp="",ca=true) while the file says fp="" apply(fp="",ca=true)])
|
||||
== MUTATION: a config that is not a file is 'moved' in memory
|
||||
*** NOT A RED-PROOF *** (mutant rc=0 built=True real rc=0)
|
||||
== MUTATION: a failed config write still changes the running client
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestConfigCannotBeWritten_TheFingerprintStays (0.01s)
|
||||
pvetrust_test.go:391: the trust changed without a saved config: [copy probe(fp="",ca=true) while the file says fp="1e4a6394a6837f9a3d32139d497210b4403ed3a3d66804b723f5f99df055d20b" apply(fp="",ca=true)]
|
||||
== MUTATION: the throttle removed
|
||||
*** NOT A RED-PROOF *** (mutant rc=1 built=False real rc=0)
|
||||
FAIL gitea.dooplex.hu/admin/felhom-agent/internal/pvetrust [build failed]
|
||||
internal/pvetrust/pvetrust.go:155:2: declared and not used: tooSoon
|
||||
== MUTATION: 'ends soon' never said
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestCopyEndsSoon (0.00s)
|
||||
pvetrust_test.go:438: status = &{Mode:ca Problem: Detail: CANotAfter:2026-11-26T00:00:00Z CADaysLeft:0x2200200ea7b8 MovedAt: CheckedAt:2026-11-14T15:32:23Z}
|
||||
|
||||
11 mutations, 2 not a red-proof
|
||||
|
||||
-- NOTE: the mutation „a config that is not a file is moved in memory” is NOT a red-proof and is not a rule: without that branch the write itself fails (there is no file) and the outcome is the same problem, the same unchanged client. The branch only gives a clearer log line.
|
||||
== MUTATION: the throttle removed (done again: the first try did not compile)
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestOnCallError_Throttled (0.01s)
|
||||
pvetrust_test.go:457: a second failure inside the gap ran again: [copy probe(fp="",ca=true) while the file says fp="8e4954b82a48cda044d0ac88160dbb028ea5ba173a2afbe1d4fb1e067a3307ca"]
|
||||
|
||||
1 mutations, 0 not a red-proof
|
||||
|
||||
-- the wiring (cmd/felhom-agent/main.go)
|
||||
== MUTATION: main.go does not feed a failed collect to the trust manager
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestMainWiresPVETrust (0.01s)
|
||||
pvetrust_wiring_test.go:59: main.go wires "hook" 0 time(s), want at least 1 (the two-call ones: at start and after a bundle install)
|
||||
== MUTATION: main.go does not run the trust order at start
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestMainWiresPVETrust (0.01s)
|
||||
pvetrust_wiring_test.go:59: main.go wires "ensure" 1 time(s), want at least 2 (the two-call ones: at start and after a bundle install)
|
||||
== MUTATION: main.go does not report the trust stanza
|
||||
RED on the mutant, GREEN on the real file
|
||||
--- FAIL: TestMainWiresPVETrust (0.01s)
|
||||
pvetrust_wiring_test.go:59: main.go wires "stanza" 0 time(s), want at least 1 (the two-call ones: at start and after a bundle install)
|
||||
|
||||
3 mutations, 0 not a red-proof
|
||||
@@ -0,0 +1,22 @@
|
||||
== MUTATION: the proof is ignored (the copy is used whatever the handshake says)
|
||||
RED on the mutant: 1 failing line(s)
|
||||
FAIL test_pve_trust_a_copy_that_does_not_verify_falls_back_to_the_pin -- never neither: no proven copy → the served certificate is pinned. Got ca='/tmp/hostinstall-test-1e6m_k4e/etc-felhom-agent/pve-root-ca.pem' fp=''
|
||||
== MUTATION: the CA check removed (a leaf is copied)
|
||||
RED on the mutant: 1 failing line(s)
|
||||
FAIL test_pve_trust_a_source_that_is_not_a_ca_is_never_copied -- leaf.pem: copied although it is not a CA certificate
|
||||
== MUTATION: the fallback pin dropped (neither way)
|
||||
RED on the mutant: 2 failing line(s)
|
||||
FAIL test_pve_trust_a_copy_that_does_not_verify_falls_back_to_the_pin -- never neither: no proven copy → the served certificate is pinned. Got ca='' fp=''
|
||||
FAIL test_pve_trust_a_source_that_is_not_a_ca_is_never_copied -- leaf.pem: the fallback pin is missing: ''
|
||||
== MUTATION: a proven copy still pins the certificate too (the fingerprint would win over the CA)
|
||||
RED on the mutant: 1 failing line(s)
|
||||
FAIL test_pve_trust_a_new_box_gets_the_ca_copy_and_no_fingerprint -- the box's own CA verified the handshake, so: ca_file set, NO fingerprint. Got ca='/tmp/hostinstall-test-z1ib0ej6/etc-felhom-agent/pve-root-ca.pem' fp='99:36:74:0
|
||||
== MUTATION: the dry run writes the copy
|
||||
RED on the mutant: 1 failing line(s)
|
||||
FAIL test_pve_trust_dry_run_writes_nothing -- [DRY-RUN] copy /tmp/hostinstall-test-nee1qiqg/pki/ca.pem -> /tmp/hostinstall-test-nee1qiqg/etc-felhom-agent/pve-root-ca.pem (0644) and prove a TLS handshake to 127.0.0.1:51319 with the
|
||||
== MUTATION: step_agent_config does not call it
|
||||
RED on the mutant: 1 failing line(s)
|
||||
FAIL test_pve_trust_is_wired_into_the_agent_config -- step_agent_config no longer calls resolve_pve_trust
|
||||
|
||||
the real file: 0 failing line(s)
|
||||
6 mutations, 0 not a red-proof
|
||||
@@ -0,0 +1,38 @@
|
||||
== MUTATION: the CA check removed (a leaf is copied)
|
||||
if not re.search(r"\bCA:TRUE\b", out): -> if False:
|
||||
tests: 2 mutant: Ran 2 tests in 0.211s | FAILED (failures=1, errors=1) real: Ran 2 tests in 0.251s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
TypeError: 'NoneType' object is not subscriptable
|
||||
AssertionError: 'written' != 'T3'
|
||||
== MUTATION: the plan may name things (T1 removed)
|
||||
if extra: -> if False:
|
||||
tests: 1 mutant: Ran 1 test in 0.008s | FAILED (errors=1) real: Ran 1 test in 0.002s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
TypeError: 'NoneType' object is not subscriptable
|
||||
== MUTATION: the one-certificate check removed
|
||||
if len(certs) != 1 or b"PRIVATE KEY" in data: -> if False:
|
||||
tests: 1 mutant: Ran 1 test in 0.009s | FAILED (errors=1) real: Ran 1 test in 0.004s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
TypeError: 'NoneType' object is not subscriptable
|
||||
== MUTATION: the end-date check removed
|
||||
if days < 0: -> if False:
|
||||
tests: 1 mutant: Ran 1 test in 0.009s | FAILED (errors=1) real: Ran 1 test in 0.004s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
TypeError: 'NoneType' object is not subscriptable
|
||||
== MUTATION: 'same' removed (the copy is rewritten every time)
|
||||
if have == data: -> if False:
|
||||
tests: 1 mutant: Ran 1 test in 0.009s | FAILED (failures=1) real: Ran 1 test in 0.007s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: Tuples differ: (0, 'written', [('/etc/felhom-agent/pve-root-c[108 chars]20)]) != (0, 'same', [])
|
||||
== MUTATION: the copy goes to another place
|
||||
PVE_CA_COPY = "/etc/felhom-agent/pve-root-ca.pem" -> PVE_CA_COPY = "/etc/felhom-agent/other.pem"
|
||||
tests: 1 mutant: Ran 1 test in 0.008s | FAILED (failures=1) real: Ran 1 test in 0.007s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
AssertionError: Tuples differ: ('written', '/etc/felhom-agent/other.pem', False) != ('written', '/etc/felhom-agent/pve-root-ca.pem', False)
|
||||
== MUTATION: an appliance check added (a BYO host gets no copy)
|
||||
return self.pve_ca_copy() -> self.check_appliance()
|
||||
tests: 1 mutant: Ran 1 test in 0.002s | FAILED (errors=1) real: Ran 1 test in 0.007s | OK
|
||||
RED on the mutant, GREEN on the real file
|
||||
KeyError: 'pve_ca_copy'
|
||||
|
||||
7 mutations, 0 not a red-proof
|
||||
@@ -292,7 +292,7 @@ stopping line that lies.
|
||||
| **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC |
|
||||
| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB21); the newest golden passed its 7-day limit on 2026-11-10 and no waiver is in force, so the gate refuses code pushes to `felhom.eu`. **2026-11-11: golden 0.352.0 baked and published; one fresh box installed from it (`documentation/tests/golden-0.352.0-2026-11-11/README.md`); the gate is green (`python3 scripts/golden_currency_gate.py`, rc 0). The operator vouched it on 2026-11-11 at 14:48:11 CET.** | WATCHING — **2026-11-14: golden 0.352.1 baked (sha256 `4f411cbe…346d`), one fresh install proven on a nested VM on demo-hp, vouched by the operator 10:25:43 UTC (`documentation/tests/golden-0.352.1-2026-11-14/`). The gate judges the bake's age: the next bake is due by 2026-11-21.** Earlier: **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement **2026-11-03: the Sunday 2026-11-01 bake was missed (it was in no brief): golden 0.344.0 is 8 days old, the floor 0.349.0; a CODE push to felhom.eu is refused until a bake and the vouch or a dated waiver. The night's close-out was documents only, which the gate exempts; no waiver written. First on the morning list.** | — | — | CC |
|
||||
| **R-1358** | Box system & updates | P3 | **cloudflared 2026.10.4 misses two dependency fixes (VULN-183456 chi, 2026.10.5; VULN-184686 go-jose, 2026.10.6; the newest is 2026.10.7).** No public text on reach. Not pinned tonight: the tunnel is a box's only door, and 9202 has none to prove it on. | OPEN — filed 2026-11-03 | — | raise `CloudflaredImage` on the next controller release with a tunnel proof on Tester 1 | CC |
|
||||
| **R-1380** | Box system & updates | P3 | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** Found 2026-11-11 (the End-dates list, class T2, `documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB02-3 to DB02-7). MEASURED: the agent trusts one exact Proxmox certificate (a pin), and the certificate on the three test boxes ends about two years after install — the N100's about 2028-07-16, then demo-hp and Tester 1 within three months. Proxmox makes a new certificate by itself about 14 days before the old one ends (so about 2028-07-02 on the N100). No code path gives the agent the new fingerprint, no alarm names the cause, no runbook holds the hand step, and nobody has tried it. Every box installed in the same season fails in the same fortnight. P3 only because the first date is in 2028; it is certain, not a risk. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 297). Not built.** **2026-11-11 (afternoon): tried on demo-hp (`documentation/audits/certpin-spike-2026-11-11/README.md`).** A forced new certificate (`pvecm updatecerts --force`, then a reload of the web service; Proxmox's own renewal does that reload by itself) blinds the agent at once: `TLS pin mismatch`, no reconcile, no host report to the hub, 502 to the controller; guests and apps stay up. One operator mail names a symptom (`agent_channel_unknown`), none names the cause; the hub's Hosts page still read ONLINE eleven minutes after the last report (when it turns offline was not measured). The hand repair works and is written down (`documentation/runbooks/RUNBOOK-agent-proxmox-pin-repair.md`, 2.1 s from restart to a working call). **A CA file in place of the fingerprint works with no agent code:** with `proxmox.tls.ca_file` on a readable copy of the box's root CA and the fingerprint empty, the agent went through a second new certificate with no hand step; a wrong CA file was refused. Two facts for the fix: the agent's user cannot read Proxmox's certificate files, and a set fingerprint wins over the CA file. The design with both ways is `03-host-agent.md` §12a; CC's pick differs from the ruled shape, so it is decision D80. | READY — **2026-11-14 09:55 — RULED (D80, `09` §3 decision 303): the root CA copy.** Built in release 2 (the host release). OPEN — filed 2026-11-11 (end-dates list) | — | the operator rules D80 (CA file, or follow the fingerprint); then the design's open parts in `03-host-agent.md` §12a (the root step that refreshes the copy; an alarm that names the cause), then build, then prove on a scratch host with a forced renewal | CC |
|
||||
| **R-1380** | Box system & updates | P3 | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** Found 2026-11-11 (the End-dates list, class T2, `documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB02-3 to DB02-7). MEASURED: the agent trusts one exact Proxmox certificate (a pin), and the certificate on the three test boxes ends about two years after install — the N100's about 2028-07-16, then demo-hp and Tester 1 within three months. Proxmox makes a new certificate by itself about 14 days before the old one ends (so about 2028-07-02 on the N100). No code path gives the agent the new fingerprint, no alarm names the cause, no runbook holds the hand step, and nobody has tried it. Every box installed in the same season fails in the same fortnight. P3 only because the first date is in 2028; it is certain, not a risk. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 297). Not built.** **2026-11-11 (afternoon): tried on demo-hp (`documentation/audits/certpin-spike-2026-11-11/README.md`).** A forced new certificate (`pvecm updatecerts --force`, then a reload of the web service; Proxmox's own renewal does that reload by itself) blinds the agent at once: `TLS pin mismatch`, no reconcile, no host report to the hub, 502 to the controller; guests and apps stay up. One operator mail names a symptom (`agent_channel_unknown`), none names the cause; the hub's Hosts page still read ONLINE eleven minutes after the last report (when it turns offline was not measured). The hand repair works and is written down (`documentation/runbooks/RUNBOOK-agent-proxmox-pin-repair.md`, 2.1 s from restart to a working call). **A CA file in place of the fingerprint works with no agent code:** with `proxmox.tls.ca_file` on a readable copy of the box's root CA and the fingerprint empty, the agent went through a second new certificate with no hand step; a wrong CA file was refused. Two facts for the fix: the agent's user cannot read Proxmox's certificate files, and a set fingerprint wins over the CA file. The design with both ways is `03-host-agent.md` §12a; CC's pick differs from the ruled shape, so it is decision D80. | READY — **2026-11-14 night: BUILT, not released** (felhom-agent `386a0c4` + `a9ab47e`; the installer on felhom.eu `main`, not tagged). The root step keeps the copy; the agent moves a pinned box in the order copy → proof → config, clears the fingerprint last, and a pinned box whose certificate changed heals at the first failed call; the installer gives a new box the copy and no fingerprint; the cause of a trust failure is named. `03` §12a is `[FACT — built, not released]`. **LEFT: the proof on a box** (demo-hp, a forced new certificate, twice; then the release with both signed jobs and the installer tag). What the hub does not tell yet: R-1396. Earlier: READY — **2026-11-14 09:55 — RULED (D80, `09` §3 decision 303): the root CA copy.** Built in release 2 (the host release). OPEN — filed 2026-11-11 (end-dates list) | — | the operator rules D80 (CA file, or follow the fingerprint); then the design's open parts in `03-host-agent.md` §12a (the root step that refreshes the copy; an alarm that names the cause), then build, then prove on a scratch host with a forced renewal | CC |
|
||||
| **R-1382** | Box system & updates | P3 | **A box that was off past an end date: nobody knows if it comes back by itself (5 answers).** Found 2026-11-11 (the End-dates list, class T4, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; read in source and documents, never tried). After months without power: the web certificate has ended (DB01-6: does Traefik renew at the first start, and what does the household see until then), Debian's package keys may be too old to verify an update (DB16-6), the tunnel client may be below what Cloudflare accepts (DB19-6), a 30-day restore copy is past its date (DB24-6), and the 14-day countdown after a stopped off-site copy ran out while the box was off (DB04-6). Tester 2 has been off since 2026-10-04 and is the natural test. | OPEN — filed 2026-11-11 (end-dates list) | — | When Tester 2 comes back (or with a scratch host and a moved clock on Tier 0, as a task): record what renews by itself and what needs a hand; put the hand steps in a runbook. | CC |
|
||||
| **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 300). Not built.** **2026-11-11 (afternoon): why the N100's clock is wrong is known (R-1301, `documentation/audits/n100-clock-2026-11-11/README.md`).** Numbers from source: a host clock more than 120 s behind refuses every signed job (`felhom-agent/internal/authz/verifier.go:21`, `:153-162`); at about 1.25 s a day the N100 gets there in about 70 days (mid-January 2027, an estimate from one measurement). | OPEN — **2026-11-14: the one box whose clock was wrong (the N100, 32.6 s) is right again (R-1301 closed); this row's check is still not built.** Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC |
|
||||
| **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC |
|
||||
@@ -302,7 +302,7 @@ stopping line that lies.
|
||||
| **R-1395** | Box system & updates | P4 | **A step of the host's clock makes the controller read a new boot: it then recreates every drive-backed app, once.** Found 2026-11-14 on the N100 (`audits/release-2a-2026-11-14/D/13b-n100-container-near-the-step.txt`): the time service came back and stepped the clock by +32.7 s; 9 seconds later the controller logged „[gate] boot 1794112550-…: recreating drive-backed app” for radarr, sonarr and qbittorrent (each down for 1 to 5 s; all healthy after). Cause, read on the box: the controller's boot identity carries `/proc/stat` `btime`, and the kernel computes `btime` from the wall clock, so it moved by the same 32 s (the crash guard's `last_boot_at` reads 04:35:18Z, `btime` now 04:35:50Z). The crash guard was NOT fooled: it reads the kernel's `boot_id`. A step this large needs a clock that was wrong for weeks, so a healthy box does not meet it; a fresh box with a wrong hardware clock can, at its first time sync. | OPEN — filed 2026-11-14 | — | make the controller's boot identity the kernel's `boot_id` (as the crash guard's is), with a test that a moved `btime` is not a new boot | CC |
|
||||
|
||||
|
||||
## Monitoring & notifications — 28 rows (P2 3, P3 17, P4 8)
|
||||
## Monitoring & notifications — 29 rows (P2 3, P3 18, P4 8)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -333,6 +333,7 @@ stopping line that lies.
|
||||
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
|
||||
| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC |
|
||||
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | OPEN — **2026-11-14 (release 2a), two more things the hub does not tell:** (1) a kernel clean-up that removed kernels shows only as „cleaned <time ago>” in the System page's „Kernel step” cell — what was removed, kept and gained is stored and shown nowhere; (2) the security-profile refresh sends nothing to the hub at all, also when it restarted a failed service (it did on the N100). Both for release 3. Earlier: OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
|
||||
| **R-1396** | Monitoring & notifications | P3 | **The agent now names WHY its trust in Proxmox is wrong, and the hub does not tell the operator yet.** Found 2026-11-14 night (release 2b, built). The agent's host report carries a `proxmox_trust` stanza (`mode`, `problem`, `detail`, the CA copy's end date; `03-host-agent.md` §12a). Two gaps: (1) the hub does not read the stanza — a box whose CA copy does not verify, ends within 30 days, or whose move failed is said only in the box's journal; (2) a box that is fully blind (neither the pinned certificate nor the CA copy works) sends NO host report, because the report starts with a call to Proxmox — that case reaches the operator only as „host stale” after 45 minutes, with no cause. With release 2b a pinned box heals itself at the first failed call, so (2) needs both ways to fail at once. | OPEN — filed 2026-11-14 | — | hub: an operator-only event from `proxmox_trust.problem` (raise once per cause, clear when it is empty), and the mode on the System page; agent: send a report without the Proxmox part when the failure is one of trust (a design choice: what the hub does with a report that has no guests) | CC |
|
||||
| **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC |
|
||||
|
||||
## Hub & operator — 18 rows (P2 1, P3 5, P4 12)
|
||||
|
||||
Reference in new issue
Block a user