diff --git a/REPORT-golden-token-clock-certpin-2026-11-11.md b/REPORT-golden-token-clock-certpin-2026-11-11.md new file mode 100644 index 00000000..68a2e96a --- /dev/null +++ b/REPORT-golden-token-clock-certpin-2026-11-11.md @@ -0,0 +1,122 @@ +# Report — the golden, the token fix, the N100's clock, the certificate pin (2026-11-11, afternoon) + +Session start 2026-11-11T12:50:33Z. One STOP (the vouch). No release, no floor change, no hub deploy, no agent, +controller or wrapper code. + +## 1. Claims in the prompt that turned out wrong + +- **"Proxmox has a documented way to force a new certificate" — true, with one more step.** `pvecm updatecerts + --force` writes the new certificate, but the web service serves the old one until `systemctl reload pveproxy`. + Proxmox's own nightly renewal does that reload by itself. +- **"A setting may make the agent follow" — true, with two limits.** The key `proxmox.tls.ca_file` exists and works. + But a set fingerprint wins over it, and the agent's user cannot read Proxmox's CA file; the test used a copy. +- **"The fresh box of Part 1 can show if the clock problem is the product's" — wrong.** The cause needs a kernel that + went back and forth. A fresh box has no such history. It showed only that a new install keeps time. +- **The ruled fix shape of R-1380 ("the agent reads the certificate file on its own host")** cannot work as written: + the agent's user has no right to read that file. It needs a root step. Decision D80 is the result. + +## 2. Five lines + +1. **Golden 0.352.0 is baked, published, tried on one fresh box and vouched by the operator** (14:48:11 CET). The + gate is green (`python3 scripts/golden_currency_gate.py`, rc 0). +2. **No clone holds a login in its address** (16 clones read, 0 with one). 13 clones still use the clear-text file in + the home folder through the global `store` helper. +3. **The N100 keeps no time because the kernel loaded an old cached security profile for the time service.** One + hand step repairs it; it is in `STATUS.md`, not run. +4. **On demo-hp a new Proxmox certificate blinded the agent at once.** Guests and apps stayed up. The hand repair + took 2.1 s from the restart to a working call. +5. **With a CA file in place of the fingerprint the agent went through a second new certificate with no hand step.** + +## 3. Parts + +| Part | State | Where | +|---|---|---| +| 0 — the rulings of 13:17 (D76–D79, all yes) | done | `09-update-architecture.md` §3 decisions 297–302; the four rows left `STATUS.md` | +| 1 — golden: bake, publish, fresh install, vouch | done | `documentation/tests/golden-0.352.0-2026-11-11/README.md` | +| 2 — token fix: the two other clones, the rule line, the skill line | done | §4 below | +| 3 — why the N100 keeps no time | done (diagnosis only) | `documentation/audits/n100-clock-2026-11-11/README.md` | +| 4 — certificate spike on demo-hp | done | `documentation/audits/certpin-spike-2026-11-11/README.md` | + +**Part 1, changed from the brief:** the hub offers only the vouched golden, so the fresh box ran the public installer +with one added line that names golden 0.352.0 and its sha256. All other steps were the product's own. **Not read:** +the agent's pin on the fresh box. CC put no SSH key on the host before the installer closed it. That was CC's mistake; +the lesson is now a line in the build skill. + +## 4. The token fix, and what still depends on the clear-text file + +Fixed in this run: `git/cc-secrets` and the plug-meter clone of `felhom-agent`. With the morning's fix that is three +clones; each uses its own helper (`scripts/git-credential-sec.sh`, now in the repository) and the store entry +`GITEA_GIT_TOKEN`. Measured after: 0 of 16 clones hold a credential in the address. + +Still on the clear-text file (3 lines, mode 600, hosts `gitea.dooplex.hu` and `github.com`): the 13 other clones, +through the global `store` helper. **Proposal, nothing changed:** (1) give each of the 13 clones the same helper; +(2) put the `github.com` login into the store under its own name; (3) the operator then unsets the global helper and +removes the file. Step 3 is the operator's, because it removes a credential. + +No token was made, rotated or deleted. A side effect, filed: `sec put` can no longer push the store by itself +(R-1390); the second command is in `STATUS.md`. + +## 5. Versions, digests, CI + +- Golden 0.352.0, sha256 `d76336054282ec138d8b0c8b9fec63319aa06f192aa421d012890b265b9abced`, 639,712,511 bytes; + round trip from the registry equal. Controller in it: 0.352.0 (MinAgent 0.158.1). +- Vouched manifest: agent 0.173.0 sha256 `752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea` + (unchanged), golden 0.352.0, min agent 0.158.1. +- Baselines at start, all equal to `origin/main`: felhom.eu `978666dde3`, felhom-controller `a9189c9dc4`, + felhom-agent `91a2924ad0`, app-catalog `f4c5c797d9`. Only felhom.eu has a commit from this run. +- CI of this commit: in the chat note, by hash. + +## 6. Register + +**Before 245, after 248, opened 3, closed 0.** Opened: R-1388 (stale security profile after a kernel went back and +forth), R-1389 (installer address 404 on the home network), R-1390 (`sec put` cannot push). Dated lines: R-468, +R-925, R-1301, R-1380, R-1384. One new decision: D80. Capability map: no row; no box proves a new capability. + +## 7. Machines, and how each was left + +| Machine | What changed | Left as | +|---|---|---| +| DooPlex | the bake and its registry copy; the address and helper settings of two clones; the files of this commit | bake guest torn down; nothing else | +| demo-hp host | Part 1: VM 9384 on `nvme-scratch`, an uploaded ISO, a helper script. Part 4: two forced certificates, three reloads of the web service, the agent's `proxmox.tls` block, eight agent restarts | VM, disks, ISO and script removed. Pin = served certificate. Agent active, 77 of 77 capabilities, calls answer 200, hub report at 14:02:53Z. The CA copy and the saved config are removed. **The node certificate is new** (ends 2028-11-09; before: 2028-08-19) | +| The N100 | nothing (reads only) | as found: its time service is still failed | +| Tester 1 | nothing (reads only) | agent active, guest running (read 14:04:07Z) | +| ep0 | nothing by CC. **The hub re-issued Tester4's backup token there during the fresh install** — the product's own act | the operator removes host `Tester4-d35551` at the hub | +| The hub | host `Tester4-d35551` and appliance `3fb5fed3-504c-4c2c-a63f-8108d5c15b6e` (the fresh box) | both wait for the operator | + +**In Part 4 the agent on demo-hp was blind for 8 min 56 s (planned) and stopped for 92 s (not planned).** The second +was CC's mistake: too many restarts in 2 minutes met systemd's start limit. One operator mail went out +(`agent_channel_unknown`, 13:52:05Z), and its recovery event followed. + +## 8. Documents read, by name + +Workspace `CLAUDE.md`, `felhom.eu/CLAUDE.md`, `.claude/rules/live-probes.md`, the `felhom-build-deploy` skill, +`runbooks/publish-train-rules.md`, `RUNBOOK-manual-build.md` §4.2, `runbooks/nightly-rotation.md`, the golden-0.350.0 +README, `cc-secrets/ARCHITECTURE.md`, the morning report §3. Architecture: `01-topology-and-trust.md`, +`03-host-agent.md`, `04-control-plane-authorization.md`, `05-hub-architecture.md`, `09-update-architecture.md`, +`11-os-updates.md`. Lists: Clock CA03, CA04, CA09; End dates DB02. **No architecture document named the agent's +Proxmox pin** (search of 01, 03, 04: 0 lines); it is now `03-host-agent.md` §12a. + +## 9. Edits to instruction files and the poster + +- Workspace `CLAUDE.md`, lines 126–127, and its tracked copy `documentation/runbooks/workspace-CLAUDE.md`. Before: no + line. After: "A repository address never holds a credential (2026-11-11, R-925). A clone that needs a login uses + its own credential helper (`felhom.eu/scripts/git-credential-sec.sh`) and `sec run GITEA_GIT_TOKEN -- git …`. Never + print a git address." Why: the brief's Part 2 asked for it. It adds a limit; it loosens none. +- `skills/felhom-build-deploy/SKILL.md`: two lines added (the felhom-agent clone needs `sec run GITEA_GIT_TOKEN`; put + your SSH key on a proof host before the installer). `check_skills.py` passes. +- System poster: no fact in its facts file changed. Nothing to do. + +## 10. Observations + +- The stale profile after a kernel went back and forth — FILED: R-1388. +- Tester 1's host has the older chrony — FILED: in R-1388. +- The installer address gives 404 on the home network — FILED: R-1389. +- `sec put` cannot push the store — FILED: R-1390. +- No alarm names the certificate cause; the hub reads ONLINE with no new report — FILED: R-1380 (dated line). +- The agent cannot read Proxmox's certificate files; the fingerprint wins over the CA file — FILED: R-1380 and + `03-host-agent.md` §12a. +- The agent unit allows 4 starts in 2 minutes — NOT-A-FINDING: the unit's design; the runbook now says "one restart". +- The pin scripts rewrote `agent.json` in a new layout — NOT-A-FINDING: the content is equal (`evidence/24`). +- The End-dates list still gives demo-hp's old end date (2028-08-19) — NOT-A-FINDING: a dated list; the spike README + holds the new date. +- `felhom-web-design` skill is not installed (a warning of `check_skills.py`) — NOT-A-FINDING: not from this run. diff --git a/STATUS.md b/STATUS.md index cbe339c1..90273885 100644 --- a/STATUS.md +++ b/STATUS.md @@ -2,7 +2,7 @@ **Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).** -**Updated 2026-11-11 (midday).** Two read-only lists are ready: every job on a timer (70) and everything that ends on a date (39); nothing ends in the next 14 days (`documentation/audits/end-dates-list-2026-11-11/README.md`). The golden passed its 7-day limit on 2026-11-10, so code pushes to `felhom.eu` are refused. Four decisions wait (D76–D79). +**Updated 2026-11-11 (afternoon).** Golden 0.352.0 is vouched, so code pushes to `felhom.eu` work again. The N100 keeps no time: one hand step repairs it (below). A new Proxmox certificate blinds the agent; the hand repair is proven on demo-hp and one decision waits (D80). ## What works - **The Plex attack did not reach Felhom's secrets** (R-1241). @@ -32,10 +32,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked | |---|---|---|---|---|---|---| -| D76 | R-1380 | Shall the agent follow Proxmox's new certificate by itself? | **Yes** (one agent change, long before 2028; no box then depends on a calendar) | a hand step in a runbook and a calendar date for each box, for ever | in July 2028 the N100's agent loses guest control and backups, with no alarm that says why; the other boxes follow | — | -| D77 | R-1379 | Shall the product warn you 14 days before an end date (a box's web certificate, a hub token)? | **Yes** (a renewal that fails is silent today) | you keep the dates in a calendar by hand | the list of 2026-11-11 is the only guard | — | -| D78 | R-1383 | Shall a timed job that gives up or is skipped tell you (four cases: the guest-power watchdog, a skipped hub daily check, a lost operator mail, a refused signed job)? | **Yes** (small changes in the agent and the hub) | the logs stay the only trace | you learn of a dead guest from the household | — | -| D79 | R-1384 | Shall the hub check each box's clock and alarm above one minute of error? | **Yes** (a box with a wrong clock can look healthy and cannot take an agent update) | R-1301 (repair the N100's time service) stays the only guard | the N100 stays 32 s behind, and nobody is told when it gets worse | — | +| D80 | R-1380 | How shall the agent trust Proxmox after its certificate changes: a copy of the box's own root CA, or a helper that follows the new fingerprint (your ruling of 11-11 13:17)? | **The root CA copy** (proven on demo-hp: the agent went through a new certificate with no hand step; no new privileged code) | follow the fingerprint: a root helper, a sudo rule and re-pin logic to write and test; trust stays on one exact certificate | nothing is built; the first box goes blind about 2028-07-02 | the fix of R-1380 | | D66 | R-1365 | Should a security-watch source that stays down mail again every 7 days, as an open notice does? Today it mails once, then only the Monday summary lists it. | **Yes** (a real outage then cannot be forgotten; the seven pins that can never be read no longer count, so it stays quiet when all is well) | no: one line in a long Monday mail stays the only reminder | a source that stays down mails once; the Monday list still shows it | — | | D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — | | D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — | @@ -43,7 +40,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te | D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — | ## Waiting on you — do -**2026-11-11.** 1. Bake and vouch a golden, or write a waiver (R-468). If not: no code push to `felhom.eu`. 2. Read the dates of „Only you can read these dates” (`documentation/audits/end-dates-list-2026-11-11/README.md`, R-1381). If not: a domain or a token can end with no warning. 3. Rotate the Gitea token of login `kisfenyo` into the store as `GITEA_GIT_TOKEN` (R-925); it was printed on 11-10. 4. Yes or no: may CC fix the two other clone addresses (`cc-secrets`, the plug-meter clone)? Only `felhom-agent` is fixed. 5. Search Console read (R-1035, now 11-18). +**2026-11-11.** 1. N100, as root, one time: `apparmor_parser -r -T -W /etc/apparmor.d/usr.sbin.chronyd && systemctl restart chrony && chronyc tracking` (R-1301; not tested). If not: about mid-January 2027 it refuses every agent update. 2. Hub Hosts page: delete host `Tester4-d35551`, Discard appliance `3fb5fed3…` (today's test box). If not: a dead entry alarms. 3. Rotate login `kisfenyo`'s Gitea token into the store as `GITEA_GIT_TOKEN` (R-925); the push after `sec put` is in R-1390. 4. Home DNS: remove the `felhom.eu` override (R-1389). 5. Read the dates only you can read (R-1381). 6. Search Console read (R-1035, 11-18). **Still needed? (yes/no)** — answer per line. 1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). (since 10-16) diff --git a/documentation/architecture/03-host-agent.md b/documentation/architecture/03-host-agent.md index 8a193977..5c389437 100644 --- a/documentation/architecture/03-host-agent.md +++ b/documentation/architecture/03-host-agent.md @@ -933,6 +933,45 @@ tunnel token, encrypted storage creds (NFS/CIFS/PBS), and the **live PBS key**. and the secret footprint that left the controller now concentrate here — which is the whole argument for §3's root-minimization and a small, auditable agent. +### 12a. How the agent trusts Proxmox's certificate, and what happens when it changes (R-1380) + +`[FACT — measured on demo-hp 2026-11-11]` (`documentation/audits/certpin-spike-2026-11-11/README.md`) + +- The agent calls Proxmox at `https://127.0.0.1:8006`. It trusts **one exact certificate**: the installer reads the + served certificate and writes its sha256 into `proxmox.tls.fingerprint` (`scripts/felhom-host-install.sh`). The + order in the agent is: skip-verify, then fingerprint, then `proxmox.tls.ca_file`, then the system's roots + (`felhom-agent/internal/proxmox/tls.go`). **A set fingerprint wins over a CA file.** +- Proxmox makes a new node certificate by itself (`pveupdate`, daily timer; `PVE/Cluster/Setup.pm`, pve-cluster + 9.1.9): the certificate lasts at most 2 × 365 days and is renewed when it ends within 14 days. `pveupdate` then + reloads the web service, so the new certificate is served the same night. +- From that moment every agent call to Proxmox fails with `TLS pin mismatch`. Measured: no reconcile, no guest list, + no storage view, no host report to the hub; the controller gets 502 from the local API. Guests and apps stay up. + One operator mail names the symptom (`agent_channel_unknown`), none names the cause. The hand repair is + `runbooks/RUNBOOK-agent-proxmox-pin-repair.md` (2.1 s from restart to a working call). +- The agent's user **cannot read** `/etc/pve/pve-root-ca.pem` or the node certificate (`0640 root:www-data`). +- With the fingerprint empty and `ca_file` set to a readable copy of Proxmox's root CA, the agent connected, and it + went on through a second forced certificate with no restart and no hand step. A CA file that did not sign the + certificate was refused (`x509: certificate signed by unknown authority`). +- Proxmox's root CA lasts 3650 days. Its code renews it when less than two years are left, **with the same key and + the same name**; a node certificate is not made from a CA that ends in less than 14 days. Read from the code, not + forced. + +`[DESIGN — proposed, not built]` Two ways. The operator's ruling of 2026-11-11 13:17 (`09` §3 decision 297) names the +first; the spike points to the second, so the choice is back with the operator (`STATUS.md` D80). + +| | A — the agent follows the fingerprint | B — the agent trusts the box's own root CA (**CC's pick**) | +|---|---|---| +| How | A root helper reads the node certificate after a mismatch and gives the agent the new fingerprint | The installer puts a copy of `/etc/pve/pve-root-ca.pem` where the agent can read it, sets `ca_file`, leaves the fingerprint empty | +| New agent code | yes: the helper, a sudo rule, the re-pin logic, its tests | none for the trust itself (the key exists and is proven) | +| Trust | one exact certificate | every certificate this box's own CA signs; the CA's key is root-only on the same box | +| Next hand step | none, if the helper is right | none until the copy's own end date (10 years from install); the copy must be refreshed after Proxmox renews the CA | +| Boxes that exist | one root step each (the signed config bundle) | the same | + +Why B: it is the standard way to trust a private CA, it is proven on a box, and it removes the failure with no new +privileged code. What B still needs before it is built: the root step that refreshes the copy (and a test that an +old copy raises an alarm that names the cause), and a rule for what the agent does if both keys are set. +Whichever way is built, the mismatch must raise an alarm that names the cause. + ## 13. Open items / what this unblocks Resolved here: tunnel placement (host, agent-managed, own systemd service), the diff --git a/documentation/architecture/09-update-architecture.md b/documentation/architecture/09-update-architecture.md index e0bb2d4f..7ed3d15a 100644 --- a/documentation/architecture/09-update-architecture.md +++ b/documentation/architecture/09-update-architecture.md @@ -1265,6 +1265,28 @@ Recorded before the work (`audits/space-list-2026-11-10/README.md` §5 held the Hetzner and the mail service are read from source and documents only: no hub database read, no account call. *Operator, 2026-11-10 12:22.* +### 2026-11-11 (13:17, operator, planning chat) — four rulings on the End-dates and Clock lists; the clones; the gap review pauses + +Recorded before any build. **None of these is built.** The questions were D76–D79 in `STATUS.md` +(`audits/end-dates-list-2026-11-11/README.md`, `audits/clock-list-2026-11-11/README.md`). + +297. **D76 = yes: the agent follows Proxmox's new certificate by itself** — after the failure is seen on demo-hp + (`audits/certpin-spike-2026-11-11/README.md`). **Cost of the other option:** a hand step in a runbook and a + calendar date for each box, for ever. The two ways to build it are in `03-host-agent.md` (the part on the Proxmox + certificate pin). R-1380. *Operator, 2026-11-11 13:17.* +298. **D77 = yes: the product warns 14 days before an end date** (a box's web certificate, a hub token). **Cost of the + other option:** the dates stay in a calendar kept by hand. R-1379. *Operator, 2026-11-11 13:17.* +299. **D78 = yes: a timed job that gives up or is skipped tells the operator — the four cases:** the guest-power + watchdog gave up; a hub daily check was skipped; an operator mail was lost; a signed job was refused. **Cost of + the other option:** the logs stay the only trace. R-1383. *Operator, 2026-11-11 13:17.* +300. **D79 = yes: the hub checks each box's clock and alarms above one minute of error.** **Cost of the other option:** + the repair of one box's time service (R-1301) stays the only guard. R-1384. *Operator, 2026-11-11 13:17.* +301. **The two other clone addresses are fixed the same way as `felhom-agent`'s** (no login in a repository address; + `audits/no-token-in-address-2026-11-11/`). **The token rotation is the operator's and comes later; CC does not ask + again.** R-925. *Operator, 2026-11-11 13:17.* +302. **The gap review pauses after four lists** (crossings, endings, end dates, clock): the operator chose „build”. + The lists „outside services”, „sizes” and „fleet” are not run. *Operator, 2026-11-11 13:17.* + ### 2026-11-10 (15:19, operator, planning chat) — five rulings on the Endings list and the system disk Recorded before any build. **None of these is built.** The questions were D71–D75 in `STATUS.md` diff --git a/documentation/audits/certpin-spike-2026-11-11/README.md b/documentation/audits/certpin-spike-2026-11-11/README.md new file mode 100644 index 00000000..0ca9b054 --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/README.md @@ -0,0 +1,68 @@ +# The Proxmox certificate changes and the agent's pin does not follow — spike on demo-hp, 2026-11-11 + +Row R-1380. Box: demo-hp only. No agent code, no release. Files: `evidence/` (26 files), `tools/` (the scripts). +Documents read: `architecture/01-topology-and-trust.md`, `03-host-agent.md`, `04-control-plane-authorization.md`; +the End-dates list entry DB02 and its `evidence/box/`. **No architecture document named this pin before today** +(a search of 01, 03 and 04 found no line; now `03-host-agent.md` §12a). + +## What was done, in order (times in UTC) + +| Time | Step | Result | File | +|---|---|---|---| +| 13:49:16 | Before | served certificate = file = agent's pin (`07:B5:72:5D…07:BB`, ends 2028-08-19); agent calls answer 200 | `01`, `05` | +| — | Who else trusts it | port 8006's log: only the agent, from 127.0.0.1 (75,374 lines). No file outside the box holds the fingerprint | `02` | +| 13:49:23 | `pvecm updatecerts --force` | new file (`21:D6:B5:E0…84:12`); **Proxmox still serves the old one, the agent still works** | `06`, `07` | +| 13:50:23 | `systemctl reload pveproxy` | the new certificate is served about 10 s later | `08` | +| 13:50:39 | The agent | first `TLS pin mismatch` line; every call to Proxmox fails from here | `09`, `10` | +| 13:59:25 | Hand repair | new fingerprint into `agent.json`, restart: **a working call 2.1 s later** | `16`, `17` | +| 13:59:39 | CA file in place of the fingerprint | the agent connects (2.2 s) | `18`, `19` | +| 13:59:54 | Second forced certificate (`1F:75:D3:51…0E:74`) + reload | **the agent goes on, no restart, no hand step, 0 TLS lines** | `20` | +| 14:01:02 | Control: a CA file that did not sign it | `x509: certificate signed by unknown authority` (so the CA file is really checked) | `25` | +| 14:02:46 | Back to the standard | pin = served certificate; 77 of 77 capabilities; hub report at 14:02:53 | `22`, `24`, `26` | + +## What broke while the pin was wrong (8 min 56 s, by choice) + +- **The agent's line:** `proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint`, as a + WARN in each loop. It does not say "the certificate changed" and does not say what to do. +- **Failed:** the guest list, the storage view, the guest config, the pool read, the node status. So: reconcile + (`pass failed`), the controller supervisor, the guest-power watchdog, the guest network sweep, the stale-lock + recovery, and the local API's `/storage` and `/disks` (502 to the controller). From the code, not tried: backup, + restore, snapshot, start/stop, resize all use the same client, so they fail too. +- **The hub:** the agent sent **no host report** (`hub: collect failed; skipping this cycle's report`). The Hosts page + still read `ONLINE 1/2` eleven minutes after the last report. Not measured: when it turns to offline. +- **The alarm:** one mail to the operator after 94 s, `agent_channel_unknown (warning) — Controller→agent channel + down: agentapi: GET /storage: HTTP 502`. It names a symptom, not the cause. A recovery event came 39 s after the + repair. +- **Stayed up:** both guests, Tester 1's VM, all 26 containers in the guest, the enrolled drive. + +## Three facts the fix must know + +1. **Proxmox's own renewal reloads the web service by itself** (`pveupdate`, daily timer): node certificate at most + 2 × 365 days, renewed when it ends within 14 days. So on a real box the break comes the same night (`03`). +2. **The agent cannot read Proxmox's certificate files.** It runs as user `felhom-agent`; `/etc/pve/pve-root-ca.pem` + and the node certificate are `0640 root:www-data`. The CA test used a copy in `/etc/felhom-agent/` (`18`). The + ruled fix shape ("the agent reads the certificate file on its own host") needs a root step for the same reason. +3. **The fingerprint wins.** Config key `proxmox.tls.ca_file` exists, but it is used only when + `proxmox.tls.fingerprint` is empty (`felhom-agent/internal/proxmox/tls.go`). + +The root CA lasts 10 years (demo-hp: to 2036-08-17). Proxmox renews it when less than two years are left (about +August 2034) and **keeps its key and its name**, so new node certificates still pass a copy of the old CA file until +that copy's own end date. Read from Proxmox's code (`03`), not forced. + +## The two ways, and the pick → `03-host-agent.md` §12a, decision D80 in `STATUS.md` + +**CA file** (CC's pick): proven here, no agent code. **Follow the fingerprint**: needs new agent code and a root +helper. Both need one root step to reach the boxes that exist. + +## CC's mistakes in this run + +- The first control ran under 1 s (the waiter matched an older call): void. The repeat met systemd's start limit + (4 starts in 2 minutes): **the agent was stopped for 92 s** (14:01:14 to 14:02:46). The failure hook ran and did + nothing ("no pending update"). The proof for the control comes from the 1-second run's own log (`23`, `24`, `25`). +- The scripts rewrote `agent.json` with a new layout. The content is equal to the saved copy (`24`: `rest equal: True`). + +## How demo-hp was left + +Fingerprint pin equal to the served certificate (`1F:75:D3:51…0E:74`, ends 2028-11-09). Agent active, 0 restarts, +calls answer 200. The CA copy and the saved config are removed. **The certificate itself is new**: that is the one +lasting change, and it moves demo-hp's end date from 2028-08-19 to 2028-11-09. diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/01-before.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/01-before.txt new file mode 100644 index 00000000..bf94720c --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/01-before.txt @@ -0,0 +1,187 @@ +== demo-hp 2026-11-11T13:26:29Z pve pve-manager/9.2.26/41a0e2e09b9e6f9d (running kernel: 7.0.14-28-pve) +== served certificate (network read, 127.0.0.1:8006) +subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local +issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA +notBefore=Aug 20 15:44:42 2026 GMT +notAfter=Aug 19 15:44:42 2028 GMT +sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB +X509v3 Subject Alternative Name: + IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1, DNS:localhost, IP Address:192.168.0.104, DNS:demo-hp, DNS:demo-hp.local +== files +-rw------- 1 root www-data 3272 2026-08-21 17:44:42.000000000 +0200 /etc/pve/priv/pve-root-ca.key +-rw-r----- 1 root www-data 2094 2026-08-21 17:44:42.000000000 +0200 /etc/pve/pve-root-ca.pem + +/etc/pve/local/: +total 2 +drwxr-xr-x 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 . +drwxr-xr-x 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 .. +-rw-r----- 1 root www-data 83 2026-11-11 14:26:26.000000000 +0100 lrm_status +drwxr-xr-x 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 lxc +drwxr-xr-x 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 openvz +drwx------ 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 priv +-rw-r----- 1 root www-data 1704 2026-08-21 17:44:42.000000000 +0200 pve-ssl.key +-rw-r----- 1 root www-data 1785 2026-08-21 17:44:42.000000000 +0200 pve-ssl.pem +drwxr-xr-x 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 qemu-server +-rw-r----- 1 root www-data 560 2026-11-10 15:43:03.000000000 +0100 ssh_known_hosts +-- /etc/pve/local/pve-ssl.pem +subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local +issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA +notBefore=Aug 20 15:44:42 2026 GMT +notAfter=Aug 19 15:44:42 2028 GMT +sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB +-- /etc/pve/pve-root-ca.pem +subject=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA +issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA +notBefore=Aug 20 15:44:42 2026 GMT +notAfter=Aug 17 15:44:42 2036 GMT +sha256 Fingerprint=E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7 +ls: cannot access '/etc/pve/local/pveproxy-ssl.pem': No such file or directory +== agent +felhom-agent 0.173.0 +active +Restart=on-failure +NRestarts=0 +ActiveEnterTimestamp=Tue 2026-11-10 15:58:17 CET +proxmox keys: ['endpoint', 'node', 'tls', 'token'] +endpoint : https://127.0.0.1:8006 +node : demo-hp +token : +tls: {'fingerprint': '07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB', 'insecure_skip_verify': False} +total 28 +drwxr-xr-x 2 root root 4096 2026-09-28 16:25:22.412907166 +0200 . +drwxr-xr-x 102 root root 4096 2026-11-08 02:49:07.920613811 +0100 .. +-rw------- 1 felhom-agent felhom-agent 2454 2026-09-28 16:25:22.443523132 +0200 agent.json +-rw------- 1 felhom-agent felhom-agent 2498 2026-09-24 15:33:45.307471226 +0200 agent.json.night-0925-off +-rw------- 1 felhom-agent felhom-agent 2451 2026-09-24 21:56:34.103084862 +0200 agent.json.pre-a4-retention +-rw------- 1 felhom-agent felhom-agent 2451 2026-09-24 21:57:43.930322856 +0200 agent.json.pre-d44 +-rw------- 1 root root 2451 2026-08-21 18:09:15.470472229 +0200 agent.json.pre-r672 +8:# allowlist in /etc/sudoers.d/felhom-agent (configs/felhom-agent.sudoers). The User= here and the +39:User=felhom-agent +41:ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json +== agent log, last proxmox lines +4422 +pam_unix(sudo:session): session closed for user root +felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- cat /etc/network/interfaces +pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +pam_unix(sudo:session): session closed for user root +pam_unix(sudo:session): session closed for user root +felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- pgrep -x dhclient +pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999) +pam_unix(sudo:session): session closed for user root +== who calls port 8006 (pveproxy access log, client addresses, last 2 days) + 75375 ::ffff:127.0.0.1 +-- users seen + 75374 ::ffff:127.0.0.1 felhom-agent@pve!agent + 1 ::ffff:127.0.0.1 - +== listening / other TLS users of this cert +LISTEN 0 4096 127.0.0.1:85 0.0.0.0:* users:(("pvedaemon worke",pid=235161,fd=6),("pvedaemon worke",pid=168847,fd=6),("pvedaemon w +LISTEN 0 4096 *:8006 *:* users:(("pveproxy worker",pid=316496,fd=6),("pveproxy worker",pid=308404,fd=6),("pveproxy wo +LISTEN 0 4096 *:3128 *:* users:(("spiceproxy work",pid=1733482,fd=6),("spiceproxy",pid=1399,fd=6)) +== in flight +running tasks: [] + VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID + 341 night1004-tester1 running 8192 200.00 1950 + 9384 golden1111-box running 10240 64.00 313884 +VMID Status Lock Name +9201 running demo-hp +9202 running demo-hp-scratch +9401 stopped upgrade-harness +== tools' own help +USAGE: pvecm updatecerts [OPTIONS] + + Update node certificates (and generate all needed files/directories). + + --force + Force generation of new SSL certificate. + + --silent + Ignore errors (i.e. when cluster has no quorum). + + --unmerge-known-hosts (default=0) + Unmerge legacy SSH known hosts. + + +USAGE: pvenode cert delete [] + + DELETE custom certificate chain and key. + + (default=0) + Restart pveproxy. + +USAGE: pvenode cert info [FORMAT_OPTIONS] + + Get information about node's certificates. + +USAGE: pvenode cert set [] [OPTIONS] [FORMAT_OPTIONS] + + Upload or update custom certificate chain and key. + + + PEM encoded certificate (chain). + + + PEM encoded private key. + + --force (default=0) + Overwrite existing custom or ACME certificate files. + + --restart (default=0) + Restart pveproxy. + + +== renew rule in code +83:my $pveca_cert_days = 3650; +85:my $pvessl_cert_max_days = 2 * 365; +86:my $pvessl_cert_days_before_ca = 2; +438: '-days', +439: $pveca_cert_days, +478:# certificate chain up to the renewed one, and clients trusting the previous one accept new node +479:# certificates until it expires. Returns the expiry of the previous one if it got renewed. +480:sub renew_pveca_cert { +483: my $renewal_reason = sub { +484: my $min_days = $pvessl_cert_max_days + $pvessl_cert_days_before_ca; +485: if (PVE::Certificate::check_expiry($pveca_cert_fn, time() + $min_days * 24 * 60 * 60)) { +486: return 'expires within two years'; +501: my $reason = $renewal_reason->() or return undef; +503: my $old_expiry = PVE::Cluster::cfs_lock_domain( +507: return undef if !$renewal_reason->(); # another node renewed it in the meantime +519: '-days', $pveca_cert_days, +528: die "subject of renewed certificate differs\n"; +535: die "renewing pve root certificate failed:\n$err"; +543: if ($old_expiry) { +544: my $deadline = localtime($old_expiry); +545: print "Renewed cluster CA certificate, as the previous one $reason. Clients that trust" +546: . " the previous one must import the renewed $pveca_cert_fn before $deadline\n"; +549: return $old_expiry; +579:# and its expiry. A certificate from another issuer was placed there by an admin and must be kept. +685: # get ca expiry +687: my $daysleft = int(($cainfo->{notafter} - time()) / (24 * 60 * 60)); +689: if ($daysleft < 14) { +690: die "CA expires in less than 2 weeks, unable to generate certificate.\n"; +693: $daysleft -= $pvessl_cert_days_before_ca; +695: if ($daysleft >= $pvessl_cert_max_days) { +/usr/bin/pveupdate:86: my $lead_time = PVE::Certificate::get_cert_renew_lead_time($cert); +/usr/bin/pveupdate:88: if (PVE::Certificate::check_expiry($cert, time() + $lead_time)) { +/usr/bin/pveupdate:89: PVE::API2::ACME->renew_certificate({ node => $nodename }); +/usr/bin/pveupdate:93: 'Custom certificate does not expire soon, skipping ACME renewal.', +/usr/bin/pveupdate:99: 'ACME config found for node, but no custom certificate exists. Skipping ACME renewal until initial certificate has been deployed.', +/usr/bin/pveupdate:104:syslog('err', "Renewing ACME certificate failed: $@") if $@; +/usr/bin/pveupdate:106:eval { PVE::Cluster::Setup::renew_pveca_cert() }; +/usr/bin/pveupdate:107:syslog('err', "Renewing cluster CA certificate failed: $@") if $@; +/usr/bin/pveupdate:131: my $renew = sub { +/usr/bin/pveupdate:142: print "Renewed PVE certificate is valid until $expiry\n"; +/usr/bin/pveupdate:144: print "Restarting pveproxy after renewing certificate\n"; +/usr/bin/pveupdate:152: $renew->("was issued by a replaced cluster CA, renewing...", 1); +/usr/bin/pveupdate:153: } elsif (PVE::Certificate::check_expiry($certpath)) { +/usr/bin/pveupdate:155: $renew->("expired, renewing..."); +/usr/bin/pveupdate:156: } elsif (PVE::Certificate::check_expiry($certpath, time() + 14 * 24 * 60 * 60)) { +/usr/bin/pveupdate:158: $renew->("expires soon, renewing..."); +/usr/bin/pveupdate:159: } elsif (!PVE::Certificate::check_expiry($certpath, time() + 2 * 365 * 24 * 60 * 60)) { +/usr/bin/pveupdate:161: $renew->( +/usr/bin/pveupdate:162: "expires in more than 2 years, renewing to reduce certificate life-span for client compatibility..." +/usr/bin/pveupdate:166:syslog('err', "Checking/Renewing SSL certificate failed: $@") if $@; +uid=999(felhom-agent) gid=990(felhom-agent) groups=990(felhom-agent),999(systemd-journal) +agent CANNOT read the root CA file +agent CANNOT read the leaf file +755 root:www-data /etc/pve +640 root:www-data /etc/pve/pve-root-ca.pem diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/02-who-else.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/02-who-else.txt new file mode 100644 index 00000000..313ae911 --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/02-who-else.txt @@ -0,0 +1,20 @@ +== who else holds demo-hp's fingerprint? 2026-11-11T13:26:49Z +-- workspace files (colon form, plain form), names only, audits' evidence left out +(end of list) +-- positive control: the same search with audits in +felhom.eu/documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-hp-host-dates.txt +felhom.eu/documentation/audits/evidence-fixes-first-tester-2026-09-30/partC/agent-0138-live.txt +-- hub code: does the hub store or use a box's Proxmox fingerprint? +felhom.eu/hub/internal/osupdates/service.go:704: if rel, _ := s.Store.LatestOSRelease(LayerPVE); rel != nil && rel.Fingerprint == fp { +(end) +-- agent: where the pin is written +felhom-agent/internal/localapi/cert.go:29:// is absent. It returns the tls.Certificate to serve and the leaf's SHA-256 fingerprint (the +felhom-agent/internal/localapi/cert.go:38:func EnsureLeaf(certPath, keyPath, bridgeHost string) (cert tls.Certificate, fingerprint string, generated bool, err error) { +felhom-agent/internal/config/config.go:916: if v := os.Getenv("FELHOM_AGENT_PROXMOX_TLS_FINGERPRINT"); v != "" { +felhom-agent/internal/config/config.go:917: cfg.Proxmox.TLS.Fingerprint = v +felhom-agent/internal/proxmox/tls.go:55: return fmt.Errorf("proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint") +felhom-agent/cmd/felhom-agent/main.go:340: Fingerprint: cfg.Proxmox.TLS.Fingerprint, +felhom.eu/scripts/felhom-host-install.sh:3182: # TLS pin: the SERVED leaf cert fingerprint (not pvesh node info — may differ) +felhom.eu/scripts/felhom-host-install.sh:3185: [[ -n "$fp" ]] || log_warn " could not compute TLS fingerprint (leaving empty — agent will use system trust)" +felhom.eu/scripts/felhom-host-install.sh:3206: log_dry "write $AGENT_CONFIG (0600): proxmox{endpoint,node=$NODE,token=,tls.fingerprint=$fp} hub{url=$HUB_URL,host_id=$HOST_ID,a +felhom.eu/scripts/felhom-host-install.sh:3335: "tls": {"fingerprint": os.environ['FP'], "insecure_skip_verify": False}, diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/03-pve-renew-code.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/03-pve-renew-code.txt new file mode 100644 index 00000000..5007a725 --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/03-pve-renew-code.txt @@ -0,0 +1,318 @@ +2026-11-11T13:27:23Z +libpve-cluster-perl 9.1.9 +pve-cluster 9.1.9 +== Setup.pm 470-560 + gen_pveca_key(); + + $generate_pveca_cert->($pveca_key_fn, $pveca_cert_fn); + + return 1; +} + +# Keeping the key and the exact encoded subject lets node certificates issued by the previous CA +# certificate chain up to the renewed one, and clients trusting the previous one accept new node +# certificates until it expires. Returns the expiry of the previous one if it got renewed. +sub renew_pveca_cert { + return undef if !-f $pveca_key_fn || !-f $pveca_cert_fn; + + my $renewal_reason = sub { + my $min_days = $pvessl_cert_max_days + $pvessl_cert_days_before_ca; + if (PVE::Certificate::check_expiry($pveca_cert_fn, time() + $min_days * 24 * 60 * 60)) { + return 'expires within two years'; + } + + my $key_usage; + my $cmd = ['openssl', 'x509', '-noout', '-ext', 'keyUsage', '-in', $pveca_cert_fn]; + PVE::Tools::run_command( + $cmd, + outfunc => sub { $key_usage = 1 }, + errfunc => sub { }, + timeout => $pveca_lock_cmd_timeout, + ); + return 'lacks the keyUsage extension that strict TLS clients require' if !$key_usage; + + return undef; + }; + my $reason = $renewal_reason->() or return undef; + + my $old_expiry = PVE::Cluster::cfs_lock_domain( + 'pve-root-ca', + undef, + sub { + return undef if !$renewal_reason->(); # another node renewed it in the meantime + + my $old = PVE::Certificate::get_certificate_info($pveca_cert_fn); + my $tmp_fn = "$pveca_cert_fn.tmp.$$"; + my $cmd = [ + [ + 'openssl', 'x509', '-x509toreq', # keeps the exact encoded subject + '-in', $pveca_cert_fn, '-key', $pveca_key_fn, + ], + [ + 'faketime', 'yesterday', # NOTE: wrap openssl with faketime to prevent bug #904 + 'openssl', 'req', '-x509', '-in', '/dev/stdin', '-key', $pveca_key_fn, + '-days', $pveca_cert_days, + '-addext', 'keyUsage=critical,keyCertSign,cRLSign', + '-out', $tmp_fn, + ], + ]; + eval { + run_silent_cmd($cmd, timeout => $pveca_lock_cmd_timeout); + my $new = PVE::Certificate::get_certificate_info($tmp_fn); + if ($new->{subject} ne $old->{subject}) { + die "subject of renewed certificate differs\n"; + } + rename($tmp_fn, $pveca_cert_fn) + or die "replacing '$pveca_cert_fn' failed - $!\n"; + }; + if (my $err = $@) { + unlink $tmp_fn; + die "renewing pve root certificate failed:\n$err"; + } + + return $old->{notafter}; + }, + ); + die $@ if $@; + + if ($old_expiry) { + my $deadline = localtime($old_expiry); + print "Renewed cluster CA certificate, as the previous one $reason. Clients that trust" + . " the previous one must import the renewed $pveca_cert_fn before $deadline\n"; + } + + return $old_expiry; +} + +sub rotate_pveca { + my $tmp_key_fn = "$pveca_key_fn.tmp.$$"; + my $tmp_cert_fn = "$pveca_cert_fn.tmp.$$"; + eval { + $generate_pveca_key->($tmp_key_fn); + $generate_pveca_cert->($tmp_key_fn, $tmp_cert_fn); + + PVE::Cluster::cfs_lock_domain( + 'pve-root-ca', +== Setup.pm 575-720 + } +} + +# Returns who issued the node certificate, the 'current-ca', a 'replaced-ca' or an 'other' issuer, +# and its expiry. A certificate from another issuer was placed there by an admin and must be kept. +sub pve_ssl_cert_status { + my ($nodename) = @_; + + my $pvessl_cert_fn = "$pmxcfs_base_dir/nodes/$nodename/pve-ssl.pem"; + return undef if !-f $pvessl_cert_fn; + + my $cert = PVE::Certificate::get_certificate_info($pvessl_cert_fn); + my $issuer = $cert->{issuer}; + + my $issued_by = 'other'; + if ($issuer eq PVE::Certificate::get_certificate_info($pveca_cert_fn)->{subject}) { + $issued_by = 'current-ca'; + } elsif ($issuer =~ m!^/CN=Proxmox Virtual Environment/OU=[^/]+/O=PVE Cluster Manager CA$!) { + $issued_by = 'replaced-ca'; + } + + return { 'issued-by' => $issued_by, notafter => $cert->{notafter} }; +} + +sub gen_pve_ssl_key { + my ($nodename) = @_; + + die "no node name specified" if !$nodename; + + my $pvessl_key_fn = "$pmxcfs_base_dir/nodes/$nodename/pve-ssl.key"; + + return if -f $pvessl_key_fn; + + eval { run_silent_cmd(['openssl', 'genrsa', '-out', $pvessl_key_fn, '2048']); }; + + die "unable to generate pve ssl key for node '$nodename':\n$@" if $@; +} + +sub gen_pve_www_key { + + return if -f $pvewww_key_fn; + + eval { run_silent_cmd(['openssl', 'genrsa', '-out', $pvewww_key_fn, '2048']); }; + + die "unable to generate pve www key:\n$@" if $@; +} + +sub update_serial { + my ($serial) = @_; + + PVE::Tools::file_set_contents($pveca_srl_fn, $serial); +} + +sub gen_pve_ssl_cert { + my ($force, $nodename, $ip) = @_; + + die "no node name specified" if !$nodename; + die "no IP specified" if !$ip; + + my $pvessl_cert_fn = "$pmxcfs_base_dir/nodes/$nodename/pve-ssl.pem"; + + return if !$force && -f $pvessl_cert_fn; + + my $names = "IP:127.0.0.1,IP:::1,DNS:localhost,IP:${ip},DNS:${nodename}"; + + my $fqdn = $nodename; + + my $rc = PVE::INotify::read_file('resolvconf'); + if ($rc && $rc->{search}) { + $fqdn .= ".$rc->{search}"; + $names .= ",DNS:$fqdn"; + } + + my $reqfn = "/run/pve-cluster/pvecertreq-$$.tmp"; + unlink $reqfn; + + my $pvessl_key_fn = "$pmxcfs_base_dir/nodes/$nodename/pve-ssl.key"; + eval { + run_silent_cmd([ + 'openssl', + 'req', + '-batch', + '-new', + '-key', + $pvessl_key_fn, + '-out', + $reqfn, + '-subj', + "/OU=PVE Cluster Node/O=Proxmox Virtual Environment/CN=$fqdn", + '-addext', + 'basicConstraints=CA:FALSE', + '-addext', + 'extendedKeyUsage=serverAuth', + '-addext', + "subjectAltName=$names", + ]); + }; + + if (my $err = $@) { + unlink $reqfn; + die "unable to generate pve certificate request:\n$err"; + } + + # the lock also covers the serial file that all nodes share + PVE::Cluster::cfs_lock_domain( + 'pve-root-ca', + undef, + sub { + update_serial("0000000000000000") if !-f $pveca_srl_fn; + + # get ca expiry + my $cainfo = PVE::Certificate::get_certificate_info($pveca_cert_fn); + my $daysleft = int(($cainfo->{notafter} - time()) / (24 * 60 * 60)); + + if ($daysleft < 14) { + die "CA expires in less than 2 weeks, unable to generate certificate.\n"; + } + + $daysleft -= $pvessl_cert_days_before_ca; + + if ($daysleft >= $pvessl_cert_max_days) { + $daysleft = $pvessl_cert_max_days; + } + + my $cmd = [ + 'faketime', 'yesterday', # NOTE: wrap openssl with faketime to prevent bug #904 + 'openssl', 'x509', '-req', '-in', $reqfn, '-days', $daysleft, '-out', + $pvessl_cert_fn, + '-CAkey', $pveca_key_fn, '-CA', $pveca_cert_fn, '-CAserial', $pveca_srl_fn, + '-copy_extensions', 'copyall', + ]; + run_silent_cmd($cmd, timeout => $pveca_lock_cmd_timeout); + }, + ); + my $err = $@; + + unlink $reqfn or $!{ENOENT} or warn "failed to clean up '$reqfn' - $!"; + + die "unable to generate pve ssl certificate:\n$err" if $err; +} + +sub gen_pve_node_files { + my ($nodename, $ip, $opt_force) = @_; + + gen_local_dirs($nodename); + +== pveupdate 100-170 + ); + } + } +}; +syslog('err', "Renewing ACME certificate failed: $@") if $@; + +eval { PVE::Cluster::Setup::renew_pveca_cert() }; +syslog('err', "Renewing cluster CA certificate failed: $@") if $@; + +eval { + my $certpath = PVE::CertHelpers::default_cert_path_prefix($nodename) . ".pem"; + my $capath = "/etc/pve/pve-root-ca.pem"; + + my $check_issued_by_ca = sub { + # get CA info + my $cainfo = PVE::Certificate::get_certificate_info($capath); + + # get cert and check issuer and chain metadata + my $certinfo = PVE::Certificate::get_certificate_info($certpath); + if ($certinfo->{issuer} ne $cainfo->{subject}) { + die "SSL certificate ($certpath) is not issued by root CA ($capath)!\n"; + } + + # check if cert is really signed by the ca + # TODO: replace by low level ssleay interface if version 1.86 is available + my $cmd = [ + '/usr/bin/openssl', 'verify', '-no_check_time', '-CAfile', $capath, '--', $certpath, + ]; + PVE::Tools::run_command($cmd); + }; + + my $renew = sub { + my ($msg, $replaced_ca) = @_; + + # one from a replaced cluster CA fails these checks, but is still ours to replace + $check_issued_by_ca->() if !$replaced_ca; + + print "PVE certificate $msg\n"; + # create new certificate + my $ip = PVE::Cluster::remote_node_ip($nodename); + PVE::Cluster::Setup::gen_pve_ssl_cert(1, $nodename, $ip); + my $expiry = localtime(PVE::Certificate::get_expiration_as_epoch($certpath)); + print "Renewed PVE certificate is valid until $expiry\n"; + + print "Restarting pveproxy after renewing certificate\n"; + PVE::Tools::run_command(['systemctl', 'reload-or-restart', 'pveproxy']); + }; + + my $status = eval { PVE::Cluster::Setup::pve_ssl_cert_status($nodename) }; + syslog('err', "Checking SSL certificate failed: $@") if $@; + + if ($status && $status->{'issued-by'} eq 'replaced-ca') { + $renew->("was issued by a replaced cluster CA, renewing...", 1); + } elsif (PVE::Certificate::check_expiry($certpath)) { + # already expired + $renew->("expired, renewing..."); + } elsif (PVE::Certificate::check_expiry($certpath, time() + 14 * 24 * 60 * 60)) { + # expires in next 2 weeks + $renew->("expires soon, renewing..."); + } elsif (!PVE::Certificate::check_expiry($certpath, time() + 2 * 365 * 24 * 60 * 60)) { + # expires in more than 2 years + $renew->( + "expires in more than 2 years, renewing to reduce certificate life-span for client compatibility..." + ); + } +}; +syslog('err', "Checking/Renewing SSL certificate failed: $@") if $@; + +sub cleanup_tasks { + + my $taskdir = "/var/log/pve/tasks"; +== timer +NEXT LEFT LAST PASSED UNIT ACTIVATES +Thu 2026-11-12 01:46:13 CET 11h Wed 2026-11-11 02:22:58 CET 12h ago pve-daily-update.timer pve-daily-update.service + diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/04-preconditions.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/04-preconditions.txt new file mode 100644 index 00000000..ea0d4492 --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/04-preconditions.txt @@ -0,0 +1,7 @@ +== preconditions 2026-11-11T13:49:13Z +demo-hp running tasks: [] +(end of process list) +config saved on the box (0600, stays on the box) +57f9650d1ca60fcc +tester1 running tasks: [] +(end) diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/05-state-before.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/05-state-before.txt new file mode 100644 index 00000000..0c733288 --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/05-state-before.txt @@ -0,0 +1,20 @@ +== state 2026-11-11T13:49:16Z +subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local +issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA +notBefore=Aug 20 15:44:42 2026 GMT +notAfter=Aug 19 15:44:42 2028 GMT +sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB +agent tls config: {'fingerprint': '07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB', 'insecure_skip_verify': False} +agent: active since Tue 2026-11-10 15:58:17 CET restarts=0 +-- agent calls in pveproxy's access log, last 4 (time, call, answer code) +[11/11/2026:14:49:07 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200 +[11/11/2026:14:49:07 +0100] "GET /api2/json/storage 200 +[11/11/2026:14:49:08 +0100] "GET /api2/json/nodes/demo-hp/storage 200 +[11/11/2026:14:49:08 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200 +-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6) +0 +-- guests +9201 running demo-hp +9202 running demo-hp-scratch +9401 stopped upgrade-harness +341 night1004-tester1 running diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/06-force-1.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/06-force-1.txt new file mode 100644 index 00000000..6ea85e2f --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/06-force-1.txt @@ -0,0 +1,10 @@ +== force 2026-11-11T13:49:23Z +before: sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB +(re)generate node files +generate new node certificate +merge authorized SSH keys +updatecerts rc=0 +file now: sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12 notAfter=Nov 9 13:49:24 2028 GMT +served 5 s later, no restart: sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB +pveproxy: Tue 2026-11-10 15:43:04 CET reloads: +done 2026-11-11T13:49:29Z diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/07-state-after-force-1.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/07-state-after-force-1.txt new file mode 100644 index 00000000..36d431c6 --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/07-state-after-force-1.txt @@ -0,0 +1,20 @@ +== state 2026-11-11T13:50:14Z +subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local +issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA +notBefore=Aug 20 15:44:42 2026 GMT +notAfter=Aug 19 15:44:42 2028 GMT +sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB +agent tls config: {'fingerprint': '07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB', 'insecure_skip_verify': False} +agent: active since Tue 2026-11-10 15:58:17 CET restarts=0 +-- agent calls in pveproxy's access log, last 4 (time, call, answer code) +[11/11/2026:14:50:07 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200 +[11/11/2026:14:50:07 +0100] "GET /api2/json/storage 200 +[11/11/2026:14:50:08 +0100] "GET /api2/json/nodes/demo-hp/storage 200 +[11/11/2026:14:50:08 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200 +-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6) +0 +-- guests +9201 running demo-hp +9202 running demo-hp-scratch +9401 stopped upgrade-harness +341 night1004-tester1 running diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/08-pveproxy-reload-1.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/08-pveproxy-reload-1.txt new file mode 100644 index 00000000..c33d3754 --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/08-pveproxy-reload-1.txt @@ -0,0 +1,24 @@ +== the step Proxmox's own nightly job does after a renewal: systemctl reload-or-restart pveproxy; 2026-11-11T13:50:23Z +reload rc=0 +served: sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB notAfter=Aug 19 15:44:42 2028 GMT +2026-11-11T13:50:28Z +2026-11-11T13:50:34Z +served now: sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12 +2026-11-11T14:50:24+01:00 demo-hp systemd[1]: Reloaded pveproxy.service - PVE API Proxy Server. +2026-11-11T14:50:25+01:00 demo-hp pveproxy[1293]: restarting server +2026-11-11T14:50:25+01:00 demo-hp pveproxy[1293]: starting 3 worker(s) +2026-11-11T14:50:25+01:00 demo-hp pveproxy[1293]: worker 494020 started +2026-11-11T14:50:25+01:00 demo-hp pveproxy[1293]: worker 494021 started +2026-11-11T14:50:25+01:00 demo-hp pveproxy[1293]: worker 494022 started +2026-11-11T14:50:30+01:00 demo-hp pveproxy[316496]: worker exit +2026-11-11T14:50:30+01:00 demo-hp pveproxy[308404]: worker exit +2026-11-11T14:50:30+01:00 demo-hp pveproxy[203540]: worker exit +2026-11-11T14:50:31+01:00 demo-hp pveproxy[1293]: worker 316496 finished +2026-11-11T14:50:31+01:00 demo-hp pveproxy[1293]: worker 203540 finished +2026-11-11T14:50:31+01:00 demo-hp pveproxy[1293]: worker 308404 finished + PID STARTED CMD + 1293 Tue Nov 10 15:43:04 2026 pveproxy + 1293 Tue Nov 10 15:43:04 2026 pveproxy + 494020 Wed Nov 11 14:50:25 2026 pveproxy worker + 494021 Wed Nov 11 14:50:25 2026 pveproxy worker + 494022 Wed Nov 11 14:50:25 2026 pveproxy worker diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/09-state-broken.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/09-state-broken.txt new file mode 100644 index 00000000..730a0a4a --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/09-state-broken.txt @@ -0,0 +1,26 @@ +== state 2026-11-11T13:52:21Z +subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local +issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA +notBefore=Nov 10 13:49:24 2026 GMT +notAfter=Nov 9 13:49:24 2028 GMT +sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12 +agent tls config: {'fingerprint': '07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB', 'insecure_skip_verify': False} +agent: active since Tue 2026-11-10 15:58:17 CET restarts=0 +-- agent calls in pveproxy's access log, last 4 (time, call, answer code) +[11/11/2026:14:50:19 +0100] "GET /api2/json/pools/felhom 200 +[11/11/2026:14:50:19 +0100] "GET /api2/json/nodes/demo-hp/lxc 200 +[11/11/2026:14:50:19 +0100] "GET /api2/json/pools/felhom 200 +[11/11/2026:14:50:19 +0100] "GET /api2/json/nodes/demo-hp/storage 200 +-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6) +22 +time=2026-11-11T14:52:05.144+01:00 level=WARN msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" vmid=9201 target=felhom-pbs err="proxmox: GET /nodes/demo-hp/storage/felhom-pbs/content: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/storage/felhom-pbs/content\": pr +time=2026-11-11T14:52:05.505+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:52:19.211+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:52:19.230+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:52:19.393+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:52:19.394+01:00 level=WARN msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +-- guests +9201 running demo-hp +9202 running demo-hp-scratch +9401 stopped upgrade-harness +341 night1004-tester1 running diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/10-agent-log-broken.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/10-agent-log-broken.txt new file mode 100644 index 00000000..01e7f5b2 --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/10-agent-log-broken.txt @@ -0,0 +1,37 @@ +== all agent log lines since the new certificate was served (no sudo noise), first 25, cut +2026-11-11T14:50:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:50:39.237+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +2026-11-11T14:50:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:50:39.323+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae +2026-11-11T14:50:49+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:50:49.395+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +2026-11-11T14:50:59+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:50:59.230+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +2026-11-11T14:50:59+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:50:59.310+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae +2026-11-11T14:51:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:05.501+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +2026-11-11T14:51:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:19.212+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +2026-11-11T14:51:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:19.231+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +2026-11-11T14:51:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:19.313+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae +2026-11-11T14:51:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:19.395+01:00 level=WARN msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +2026-11-11T14:51:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:19.395+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +2026-11-11T14:51:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:39.233+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +2026-11-11T14:51:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:39.310+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae +2026-11-11T14:51:49+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:49.395+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +2026-11-11T14:51:59+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:59.231+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +2026-11-11T14:51:59+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:59.313+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae +2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.111+01:00 level=WARN msg="local-api: storage view unavailable for the tier presence report" target=local err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.116+01:00 level=WARN msg="local-api: storage view unavailable for the tier presence report" target=felhom-pbs err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.124+01:00 level=WARN msg="local-api: storage view unavailable for the backup-target presence check — assuming present" target=local err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.129+01:00 level=WARN msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" vmid=9201 target=local err="proxmox: GET /nodes/demo-hp/storage/local/content: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/storage/local/content\": proxmox: TLS pin mismatch: server cert sha256 does n +2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.136+01:00 level=WARN msg="local-api: storage view unavailable for the backup-target presence check — assuming present" target=felhom-pbs err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.144+01:00 level=WARN msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" vmid=9201 target=felhom-pbs err="proxmox: GET /nodes/demo-hp/storage/felhom-pbs/content: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/storage/felhom-pbs/content\": proxmox: TLS pin mismatch: server cer +2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.144+01:00 level=INFO msg="local-api: backup storage unreadable — due-check uses the last success saved on disk (R-894)" vmid=9201 target=felhom-pbs saved=2026-11-05T20:50:35Z +2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.505+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +2026-11-11T14:52:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:19.211+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +== count by message + 6 msg="reconcile: storage view unavailable — using mount-table only" + 6 msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" + 4 msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" + 2 msg="local-api: storage view unavailable for the tier presence report" + 2 msg="local-api: storage view unavailable for the backup-target presence check — assuming present" + 2 msg="local-api: /storage guest config" + 2 msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" + 2 msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)" + 2 msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" + 1 msg="local-api: backup storage unreadable — due-check uses the last success saved on disk (R-894)" diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/11-hub-and-apps-broken.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/11-hub-and-apps-broken.txt new file mode 100644 index 00000000..47fc055c --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/11-hub-and-apps-broken.txt @@ -0,0 +1,17 @@ +2026-11-11T13:54:07Z +HUB: .173.0 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 23% 37% running (connected) 37% local demo-hp-bb76ea Demo HP 0.173.0 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 73% 45% running (connected) 59% local-lvm tester-1-d70be4 Tester 1 0.173.0 9.2.2 7.0.14-22-pve ONLINE 1/2 8% 29% 33% running (connected) 33% local Felhom Hub 0.180.0 +-- +== guest 9201 containers: running count / not running +26 +== agent report lines to the hub since 14:50 +time=2026-11-11T14:52:05.111+01:00 level=WARN msg="local-api: storage view unavailable for the tier presence report" target=local err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configure +time=2026-11-11T14:52:05.116+01:00 level=WARN msg="local-api: storage view unavailable for the tier presence report" target=felhom-pbs err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match conf +== agent local API (what the controller asks) last lines +2026/11/11 13:54:05 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping bookstack — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/11/11 13:54:05 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping home-assistant — last check 3m0s ago, effective interval 5m0s, healthy=true +2026/11/11 13:54:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health +2026/11/11 13:54:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting +2026/11/11 13:54:05 client.go:1288: [DEBUG] [agentapi] GET /storage -> 502 (4ms) +2026/11/11 13:54:05 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 5ms) +2026/11/11 13:54:06 client.go:1288: [DEBUG] [agentapi] GET /disks -> 502 (6ms) +2026/11/11 13:54:06 client.go:1288: [DEBUG] [agentapi] GET /disks -> 502 (5ms) diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/12-hub-detail-broken.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/12-hub-detail-broken.txt new file mode 100644 index 00000000..73464ade --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/12-hub-detail-broken.txt @@ -0,0 +1,9 @@ +36 +Forbidden: missing scope hosts:read +pin 0 +TLS 0 +certificate 0 +proxmox: 0 +unavailable 0 +degraded 0 +Storage 0 diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/13-hub-log-broken.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/13-hub-log-broken.txt new file mode 100644 index 00000000..87fc6d4c --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/13-hub-log-broken.txt @@ -0,0 +1,14 @@ +2026-11-11T13:54:34Z +== hub log lines naming demo-hp since 13:45Z (count, then last 15, cut) +9 +2026/11/11 14:43:30 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 22057 bytes) +2026/11/11 14:43:30 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1) +2026/11/11 14:47:07 [INFO] DR-recipe app-half stored for customer demo-hp (v1) +2026/11/11 14:47:07 [INFO] Received report from demo-hp (10182 bytes) +2026/11/11 14:48:15 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 22040 bytes) +2026/11/11 14:48:15 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1) +2026/11/11 14:52:05 [INFO] Event from demo-hp: agent_channel_unknown (warning) — Controller→agent channel down: agentapi: GET /storage: HTTP 502 +2026/11/11 14:52:05 [INFO] Operator email sent for demo-hp/agent_channel_unknown +2026/11/11 14:54:27 [WARN] access token "cc-releases-2" refused: missing scope hosts:read — GET /hosts/demo-hp-bb76ea → 403 +== any hub line with pin/tls/certificate +0 diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/14-hub-after-report.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/14-hub-after-report.txt new file mode 100644 index 00000000..4adf4886 --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/14-hub-after-report.txt @@ -0,0 +1,21 @@ +2026-11-11T13:59:16Z +== hub log lines naming demo-hp, last 12 min +2026/11/11 14:48:15 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 22040 bytes) +2026/11/11 14:48:15 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1) +2026/11/11 14:52:05 [INFO] Event from demo-hp: agent_channel_unknown (warning) — Controller→agent channel down: agentapi: GET /storage: HTTP 502 +2026/11/11 14:52:05 [INFO] Operator email sent for demo-hp/agent_channel_unknown +HUB LINE: demo-hp-bb76ea Demo HP 0.173.0 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 73% 45% running (connected) 59% local-lvm tester-1-d70be4 Tester +== agent lines about the hub report since 14:50 CET + 26 msg="reconcile: storage view unavailable — using mount-table only" + 26 msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" + 17 msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" + 9 msg="local-api: /storage guest config" + 8 msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)" + 8 msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" + 4 msg="local-api: storage view unavailable for the tier presence report" + 4 msg="local-api: storage view unavailable for the backup-target presence check — assuming present" + 4 msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" + 2 msg="local-api: backup storage unreadable — due-check uses the last success saved on disk (R-894)" + 1 msg="stale-lock: guest list unavailable — skipping recovery" + 1 msg="reconcile: pass failed" + 1 msg="hub: collect failed; skipping this cycle's report" diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/15-report-skipped.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/15-report-skipped.txt new file mode 100644 index 00000000..668d1f66 --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/15-report-skipped.txt @@ -0,0 +1,3 @@ +time=2026-11-11T14:58:18.520+01:00 level=ERROR msg="reconcile: pass failed" err="reconcile: actual state: proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:58:19.394+01:00 level=WARN msg="stale-lock: guest list unavailable — skipping recovery" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:58:24.067+01:00 level=WARN msg="hub: collect failed; skipping this cycle's report" err="hub: NodeStatus failed (no useful report): proxmox: GET /nodes/demo-hp/status: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/status\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/16-hand-repair.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/16-hand-repair.txt new file mode 100644 index 00000000..cada8ffb --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/16-hand-repair.txt @@ -0,0 +1,7 @@ +== repair start 2026-11-11T13:59:25Z +== setpin fp start 2026-11-11T13:59:25Z +written tls: {'fingerprint': '21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12', 'insecure_skip_verify': False} +first agent call after restart: [11/11/2026:14:59:27 "GET /api2/json/pools/felhom 200 +seconds from start to a working call: 2.149922736 (loops: 2; 60 = no call seen) +agent: active +== repair end 2026-11-11T13:59:28Z diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/17-state-after-repair.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/17-state-after-repair.txt new file mode 100644 index 00000000..7a931a56 --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/17-state-after-repair.txt @@ -0,0 +1,22 @@ +== state 2026-11-11T13:59:29Z +subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local +issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA +notBefore=Nov 10 13:49:24 2026 GMT +notAfter=Nov 9 13:49:24 2028 GMT +sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12 +agent tls config: {'fingerprint': '21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12', 'insecure_skip_verify': False} +agent: active since Wed 2026-11-11 14:59:25 CET restarts=0 +-- agent calls in pveproxy's access log, last 4 (time, call, answer code) +[11/11/2026:14:59:27 +0100] "GET /api2/json/storage 200 +[11/11/2026:14:59:27 +0100] "GET /api2/json/storage 200 +[11/11/2026:14:59:28 +0100] "GET /api2/json/nodes/demo-hp/lxc 200 +[11/11/2026:14:59:28 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200 +-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6) +33 +time=2026-11-11T14:58:59.229+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:59:05.502+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:59:19.213+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:59:19.232+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:59:19.394+01:00 level=WARN msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:59:19.394+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +-- guests diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/18-ca-file-test.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/18-ca-file-test.txt new file mode 100644 index 00000000..191346e9 --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/18-ca-file-test.txt @@ -0,0 +1,15 @@ +== CA test start 2026-11-11T13:59:39Z +-rw-r----- 1 root felhom-agent 2094 Nov 11 14:59 /etc/felhom-agent/pve-root-ca.pem +-rw-r----- 1 root www-data 2094 Aug 21 17:44 /etc/pve/pve-root-ca.pem +agent user can read the copy +agent user CANNOT read the original +subject=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA +issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA +notBefore=Aug 20 15:44:42 2026 GMT +notAfter=Aug 17 15:44:42 2036 GMT +sha256 Fingerprint=E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7 +== setpin ca start 2026-11-11T13:59:39Z +written tls: {'ca_file': '/etc/felhom-agent/pve-root-ca.pem', 'fingerprint': '', 'insecure_skip_verify': False} +first agent call after restart: [11/11/2026:14:59:41 "GET /api2/json/storage 200 +seconds from start to a working call: 2.160335125 (loops: 2; 60 = no call seen) +agent: active diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/19-state-ca.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/19-state-ca.txt new file mode 100644 index 00000000..a7023680 --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/19-state-ca.txt @@ -0,0 +1,26 @@ +== state 2026-11-11T13:59:43Z +subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local +issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA +notBefore=Nov 10 13:49:24 2026 GMT +notAfter=Nov 9 13:49:24 2028 GMT +sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12 +agent tls config: {'ca_file': '/etc/felhom-agent/pve-root-ca.pem', 'fingerprint': '', 'insecure_skip_verify': False} +agent: active since Wed 2026-11-11 14:59:39 CET restarts=0 +-- agent calls in pveproxy's access log, last 4 (time, call, answer code) +[11/11/2026:14:59:41 +0100] "GET /api2/json/storage 200 +[11/11/2026:14:59:41 +0100] "GET /api2/json/storage 200 +[11/11/2026:14:59:41 +0100] "GET /api2/json/nodes/demo-hp/lxc 200 +[11/11/2026:14:59:41 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200 +-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6) +32 +time=2026-11-11T14:58:59.229+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:59:05.502+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:59:19.213+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:59:19.232+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:59:19.394+01:00 level=WARN msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:59:19.394+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +-- guests +9201 running demo-hp +9202 running demo-hp-scratch +9401 stopped upgrade-harness +341 night1004-tester1 running diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/20-force-2-with-ca.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/20-force-2-with-ca.txt new file mode 100644 index 00000000..13097ed4 --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/20-force-2-with-ca.txt @@ -0,0 +1,43 @@ +== force 2 start 2026-11-11T13:59:54Z +== force 2026-11-11T13:59:54Z +before: sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12 +(re)generate node files +generate new node certificate +merge authorized SSH keys +updatecerts rc=0 +file now: sha256 Fingerprint=1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74 notAfter=Nov 9 13:59:54 2028 GMT +served 5 s later, no restart: sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12 +pveproxy: Tue 2026-11-10 15:43:04 CET reloads: +done 2026-11-11T13:59:59Z +reload rc=0 at 2026-11-11T14:00:01Z +== state 2026-11-11T14:00:46Z +subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local +issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA +notBefore=Nov 10 13:59:54 2026 GMT +notAfter=Nov 9 13:59:54 2028 GMT +sha256 Fingerprint=1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74 +agent tls config: {'ca_file': '/etc/felhom-agent/pve-root-ca.pem', 'fingerprint': '', 'insecure_skip_verify': False} +agent: active since Wed 2026-11-11 14:59:39 CET restarts=0 +-- agent calls in pveproxy's access log, last 4 (time, call, answer code) +[11/11/2026:15:00:41 +0100] "GET /api2/json/nodes/demo-hp/lxc 200 +[11/11/2026:15:00:41 +0100] "GET /api2/json/pools/felhom 200 +[11/11/2026:15:00:41 +0100] "GET /api2/json/pools/felhom 200 +[11/11/2026:15:00:42 +0100] "GET /api2/json/nodes/demo-hp/storage 200 +-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6) +18 +time=2026-11-11T14:58:59.229+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:59:05.502+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:59:19.213+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:59:19.232+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:59:19.394+01:00 level=WARN msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +time=2026-11-11T14:59:19.394+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint" +-- guests +9201 running demo-hp +9202 running demo-hp-scratch +9401 stopped upgrade-harness +341 night1004-tester1 running +== agent start time and restarts (must be unchanged: 14:59:39) +NRestarts=0 +ActiveEnterTimestamp=Wed 2026-11-11 14:59:39 CET +== agent lines with x509/pin/tls since 14:59:45 +0 diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/21-negative-control-and-standard.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/21-negative-control-and-standard.txt new file mode 100644 index 00000000..ee55203c --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/21-negative-control-and-standard.txt @@ -0,0 +1,13 @@ +== negative control: a CA file that did NOT sign the certificate. Start 2026-11-11T14:01:01Z +== setpin ca start 2026-11-11T14:01:01Z +written tls: {'ca_file': '/etc/ssl/certs/ISRG_Root_X1.pem', 'fingerprint': '', 'insecure_skip_verify': False} +first agent call after restart: [11/11/2026:15:01:02 "GET /api2/json/nodes/demo-hp/storage 500 +seconds from start to a working call: .112020654 (loops: 0; 60 = no call seen) +agent: active +x509 lines: 0 +== back to the standard: fingerprint pin. 2026-11-11T14:01:02Z +== setpin fp start 2026-11-11T14:01:02Z +written tls: {'fingerprint': '1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74', 'insecure_skip_verify': False} +first agent call after restart: [11/11/2026:15:01:02 "GET /api2/json/nodes/demo-hp/storage 500 +seconds from start to a working call: .718520877 (loops: 0; 60 = no call seen) +agent: active diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/22-negative-control-2.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/22-negative-control-2.txt new file mode 100644 index 00000000..793fe12e --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/22-negative-control-2.txt @@ -0,0 +1,36 @@ +== the first control was void: the waiter matched a call of the same second; the agent ran under 1 s on the wrong file. Second try. +== state now 2026-11-11T14:01:14Z +[11/11/2026:15:01:07 "GET /api2/json/nodes/demo-hp/lxc/9201/config 200 +[11/11/2026:15:01:08 "GET /api2/json/nodes/demo-hp/storage 200 +[11/11/2026:15:01:08 "GET /api2/json/nodes/demo-hp/lxc/9201/config 200 +::ffff:127.0.0.1 - felhom-agent@pve!agent [11/11/2026:15:01:02 +0100] "GET /api2/json/nodes/demo-hp/storage HTTP/1.1" 500 - +== wrong CA, 30 s +Job for felhom-agent.service failed because start of the service was attempted too often. +See "systemctl status felhom-agent.service" and "journalctl -xeu felhom-agent.service" for details. +To force a start use "systemctl reset-failed felhom-agent.service" +followed by "systemctl start felhom-agent.service" again. +== setpin ca start 2026-11-11T14:01:14Z +written tls: {'ca_file': '/etc/ssl/certs/ISRG_Root_X1.pem', 'fingerprint': '', 'insecure_skip_verify': False} +seconds from start to a working call: 61.364869823 (loops: 60; 60 = no call seen) +x509 lines in 32 s: 0 +agent calls answered 200 in the last 25 s: 0 +== back to the standard 2026-11-11T14:02:46Z +== setpin fp start 2026-11-11T14:02:46Z +written tls: {'fingerprint': '1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74', 'insecure_skip_verify': False} +first agent call after restart: [11/11/2026:15:02:48 "GET /api2/json/nodes/demo-hp/lxc/9201/config 200 +seconds from start to a working call: 2.148034679 (loops: 2; 60 = no call seen) +agent: active +== state 2026-11-11T14:03:09Z +subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local +issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA +notBefore=Nov 10 13:59:54 2026 GMT +notAfter=Nov 9 13:59:54 2028 GMT +sha256 Fingerprint=1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74 +agent tls config: {'fingerprint': '1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74', 'insecure_skip_verify': False} +agent: active since Wed 2026-11-11 15:02:46 CET restarts=0 +-- agent calls in pveproxy's access log, last 4 (time, call, answer code) +[11/11/2026:15:03:08 +0100] "GET /api2/json/nodes/demo-hp/storage 200 +[11/11/2026:15:03:08 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200 +[11/11/2026:15:03:08 +0100] "GET /api2/json/storage 200 +[11/11/2026:15:03:09 +0100] "GET /api2/json/nodes/demo-hp/storage 200 +-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6) diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/23-start-limit.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/23-start-limit.txt new file mode 100644 index 00000000..474a510a --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/23-start-limit.txt @@ -0,0 +1,50 @@ +2026-11-11T14:03:22Z +Restart=on-failure +RestartUSec=5s +Result=success +NRestarts=0 +ActiveState=active +SubState=running +StartLimitIntervalUSec=2min +StartLimitBurst=4 +2026-11-11T15:01:02+01:00 demo-hp systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)... +2026-11-11T15:01:02+01:00 demo-hp systemd[1]: felhom-agent.service: Deactivated successfully. +2026-11-11T15:01:02+01:00 demo-hp systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-11T15:01:02+01:00 demo-hp systemd[1]: felhom-agent.service: Consumed 28.460s CPU time, 348.1M memory peak. +2026-11-11T15:01:02+01:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-11T15:01:02+01:00 demo-hp systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)... +2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability= +2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability= +2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability= +2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.135+01:00 level=WARN msg="hub: collect failed; skipping this cycle's report" err="hub: NodeStatus failed (no useful rep +2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="reconcile: pass failed" err="reconcile: actual state: proxmox: GET /nodes/demo-hp/lxc: Get \ +2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.136+01:00 level=WARN msg="reconcile: shared parent ensure failed — binds may not propagate live" err="shared-parent: +2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.138+01:00 level=ERROR msg="reconcile: AttachDrive failed" vmid=9201 where=/mnt/hdd_1 err="guest-attach: namespace /mnt/ +2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.139+01:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3 +2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.139+01:00 level=WARN msg="selfheal: node check FAILED" heal=networking detail="networking.service active=false, default +2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.139+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_windo +2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.139+01:00 level=ERROR msg="node self-heal: networking start attempt failed" attempt=1 err="context canceled: " +2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.139+01:00 level=WARN msg="lanresolver: EnsureDnsmasq failed — will retry" err="enable dnsmasq: context canceled" +2026-11-11T15:01:03+01:00 demo-hp systemd[1]: felhom-agent.service: Deactivated successfully. +2026-11-11T15:01:03+01:00 demo-hp systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-11T15:01:03+01:00 demo-hp systemd[1]: felhom-agent.service: Consumed 1.212s CPU time, 23.5M memory peak. +2026-11-11T15:01:03+01:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-11T15:01:05+01:00 demo-hp felhom-agent[527856]: time=2026-11-11T15:01:05.230+01:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3 +2026-11-11T15:01:05+01:00 demo-hp felhom-agent[527856]: time=2026-11-11T15:01:05.230+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_windo +2026-11-11T15:01:14+01:00 demo-hp felhom-agent[527856]: time=2026-11-11T15:01:14.815+01:00 level=WARN msg="pbs: verify task wait failed; reporting current snapshot state" datastore=felhom-offsite err= +2026-11-11T15:01:14+01:00 demo-hp felhom-agent[527856]: time=2026-11-11T15:01:14.815+01:00 level=WARN msg="pbs: snapshot list failed" datastore=felhom-offsite err="pbs: GET /admin/datastore/felhom-off +2026-11-11T15:01:14+01:00 demo-hp systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)... +2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent.service: Deactivated successfully. +2026-11-11T15:01:14+01:00 demo-hp systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent.service: Consumed 15.975s CPU time, 327.3M memory peak. +2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent.service: Start request repeated too quickly. +2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent.service: Failed with result 'start-limit-hit'. +2026-11-11T15:01:14+01:00 demo-hp systemd[1]: Failed to start felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent.service: Triggering OnFailure= dependencies. +2026-11-11T15:02:46+01:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog). +2026-11-11T15:02:49+01:00 demo-hp felhom-agent[535281]: time=2026-11-11T15:02:49.047+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_windo +2026-11-11T15:02:49+01:00 demo-hp felhom-agent[535281]: time=2026-11-11T15:02:49.047+01:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3 +== tests in the agent repo for the CA path +5:func TestNormalizeFingerprint(t *testing.T) { +20:func TestNormalizeFingerprint_Bad(t *testing.T) { +28:func TestTLSConfig_Build(t *testing.T) { diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/24-wrong-ca-run-and-onfailure.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/24-wrong-ca-run-and-onfailure.txt new file mode 100644 index 00000000..ce207886 --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/24-wrong-ca-run-and-onfailure.txt @@ -0,0 +1,26 @@ +2026-11-11T14:03:37Z +== the 1-second run on the WRONG CA file (pid 527553): its proxmox errors +time=2026-11-11T15:01:03.135+01:00 level=WARN msg="hub: collect failed; skipping this cycle's report" err="hub: NodeStatus failed (no useful report): proxmox: GET /nodes/demo-hp/status: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/status\": terminated signal received" +time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="reconcile: pass failed" err="reconcile: actual state: proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": terminated signal received" +== x509 count for that pid: 3 +== OnFailure unit +OnFailure=felhom-agent-rollback.service +Result=success +ExecMainStartTimestamp=Wed 2026-11-11 15:01:14 CET +ActiveState=inactive +2026-11-11T15:01:14+01:00 demo-hp systemd[1]: Starting felhom-agent-rollback.service - Felhom agent self-update auto-rollback (pending-guarded; fires per crash, no-ops without a pending update)... +2026-11-11T15:01:14+01:00 demo-hp felhom-selfupdate-guarded[529488]: felhom-selfupdate-guarded: rollback: no pending update — no-op +2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent-rollback.service: Deactivated successfully. +2026-11-11T15:01:14+01:00 demo-hp systemd[1]: Finished felhom-agent-rollback.service - Felhom agent self-update auto-rollback (pending-guarded; fires per crash, no-ops without a pending update). +== now: capability / attach lines since 15:02:46 +time=2026-11-11T15:02:48.053+01:00 level=INFO msg="capabilities self-check" ok=77 total=77 degraded=0 inactive=0 +time=2026-11-11T15:02:48.851+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae +time=2026-11-11T15:03:09.617+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae +time=2026-11-11T15:03:29.562+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae +(end) +== drive in guest 9201 +/mnt/felhom-drives/hdd_1 /dev/nvme0n1[/felhom-data] ext4 +/mnt/felhom-drives/hdd_1 /dev/nvme0n1[/felhom-data] ext4 +== config: same as the backup but for tls? +rest equal: True | tls keys now: ['fingerprint', 'insecure_skip_verify'] | before: ['fingerprint', 'insecure_skip_verify'] +-rw------- felhom-agent felhom-agent diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/25-control-proof-and-hub.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/25-control-proof-and-hub.txt new file mode 100644 index 00000000..60903ba5 --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/25-control-proof-and-hub.txt @@ -0,0 +1,17 @@ +2026-11-11T14:03:44Z +time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=pve:pool-read feature="stale-lock recovery scoping (pool ownership check)" reason="proxmox: GET /pools/felhom: Get \"https://127.0.0.1:8006/api2/json/pools/felhom\": tls: failed to verify certificate: x509: certificate signed by unknown authority" critical=false +time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=pve:store-grant:local feature="backup tier local readable by the agent (archive listing, restore-test candidacy)" reason="could not read own permissions: proxmox: GET /access/permissions?path=%2Fstorage%2Flocal: Get \"https://127.0.0.1:8006/api2/json/access/permissions?path=%2Fstorage% +time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=pve:store-grant:felhom-pbs feature="backup tier felhom-pbs readable by the agent (archive listing, restore-test candidacy)" reason="could not read own permissions: proxmox: GET /access/permissions?path=%2Fstorage%2Ffelhom-pbs: Get \"https://127.0.0.1:8006/api2/json/access/permissions?p +== hub: demo-hp lines since the repair +2026/11/11 14:59:32 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 21349 bytes) +2026/11/11 14:59:32 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1) +2026/11/11 14:59:47 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 21349 bytes) +2026/11/11 14:59:47 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1) +2026/11/11 15:00:06 [INFO] Event from demo-hp: agent_channel_recovered (info) — Controller→agent channel recovered — local-API reachable again. +2026/11/11 15:01:10 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 21351 bytes) +2026/11/11 15:01:10 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1) +2026/11/11 15:02:07 [INFO] DR-recipe app-half stored for customer demo-hp (v1) +2026/11/11 15:02:07 [INFO] Received report from demo-hp (14341 bytes) +2026/11/11 15:02:07 [INFO] managed floor SERVED for demo-hp: floor 0.352.0, agent requirement "0.158.1" from manifest (golden 0.352.0) +2026/11/11 15:02:53 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 21349 bytes) +2026/11/11 15:02:53 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1) diff --git a/documentation/audits/certpin-spike-2026-11-11/evidence/26-left-as-standard.txt b/documentation/audits/certpin-spike-2026-11-11/evidence/26-left-as-standard.txt new file mode 100644 index 00000000..d28fc0ec --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/evidence/26-left-as-standard.txt @@ -0,0 +1,18 @@ +2026-11-11T14:03:55Z +0 +(0 = both files gone) +active +active +NRestarts=0 +ActiveEnterTimestamp=Wed 2026-11-11 15:02:46 CET +[11/11/2026:15:03:49 "GET /api2/json/pools/felhom 200 +[11/11/2026:15:03:49 "GET /api2/json/nodes/demo-hp/storage 200 +9201 running demo-hp +9202 running demo-hp-scratch +9401 stopped upgrade-harness +341 night1004-tester1 running +26 +Host key verification failed. +active +9201 running +2026-11-11T14:04:07Z diff --git a/documentation/audits/certpin-spike-2026-11-11/tools/before.sh b/documentation/audits/certpin-spike-2026-11-11/tools/before.sh new file mode 100644 index 00000000..def0e02f --- /dev/null +++ b/documentation/audits/certpin-spike-2026-11-11/tools/before.sh @@ -0,0 +1,41 @@ +main() { +echo "== $(hostname) $(date -u +%FT%TZ) pve $(pveversion | head -1)" +echo "== served certificate (network read, 127.0.0.1:8006)" +echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -subject -issuer -dates -fingerprint -sha256 -ext subjectAltName 2>/dev/null +echo "== files" +ls -la --time-style=full-iso /etc/pve/local/ /etc/pve/pve-root-ca.pem /etc/pve/priv/pve-root-ca.key 2>&1 | cut -c1-140 +for f in /etc/pve/local/pve-ssl.pem /etc/pve/pve-root-ca.pem; do echo "-- $f"; openssl x509 -in $f -noout -subject -issuer -dates -fingerprint -sha256; done +ls /etc/pve/local/pveproxy-ssl.pem 2>&1 +echo "== agent" +/usr/local/bin/felhom-agent --version; systemctl is-active felhom-agent; systemctl show felhom-agent -p ActiveEnterTimestamp -p NRestarts -p Restart +python3 - <<'PY' +import json +c=json.load(open('/etc/felhom-agent/agent.json')) +p=c.get('proxmox',{}) +print('proxmox keys:',sorted(p.keys())) +for k,v in p.items(): + if k=='tls': print('tls:',v) + elif 'secret' in k.lower() or 'token' in k.lower(): print(k,': ' if v else ': ') + else: print(k,':',v) +PY +ls -la --time-style=full-iso /etc/felhom-agent/ | cut -c1-120 +systemctl cat felhom-agent | grep -i -n 'Environment\|ExecStart\|User=' +echo "== agent log, last proxmox lines" +journalctl -u felhom-agent --since '-20 min' --no-pager -o cat | grep -i -c . +journalctl -u felhom-agent --since '-20 min' --no-pager -o cat | cut -c1-230 | tail -8 +echo "== who calls port 8006 (pveproxy access log, client addresses, last 2 days)" +for f in /var/log/pveproxy/access.log /var/log/pveproxy/access.log.1; do [ -f $f ] && awk '{print $1}' $f; done | sort | uniq -c | sort -rn | head +echo "-- users seen" +for f in /var/log/pveproxy/access.log /var/log/pveproxy/access.log.1; do [ -f $f ] && awk '{print $1, $3}' $f; done | sort | uniq -c | sort -rn | head +echo "== listening / other TLS users of this cert" +ss -ltnp | grep -E ':8006|:3128|:8007|:85 ' | cut -c1-150 +echo "== in flight" +pvesh get /cluster/tasks --output-format json 2>/dev/null | python3 -c "import json,sys; t=json.load(sys.stdin); r=[x for x in t if not x.get('endtime')]; print('running tasks:',[(x['type'],x.get('id')) for x in r])" +qm list; pct list +echo "== tools' own help" +pvecm help updatecerts 2>&1 | head -20 +pvenode help cert 2>&1 | head -30 +echo "== renew rule in code" +grep -n -i 'expir\|renew\|check_expiry\|2 \* 365\|730\|days' /usr/share/perl5/PVE/Cluster/Setup.pm | head -30 +grep -rn -i 'renew\|check_expiry\|updatecerts' /usr/share/perl5/PVE/Service/pveproxy.pm /usr/bin/pve-daily-update /usr/share/perl5/PVE/CLI/pveupdate.pm /usr/bin/pveupdate 2>/dev/null | head -20 +}; main /dev/null | openssl x509 -noout -fingerprint -sha256)" +pvecm updatecerts --force; echo "updatecerts rc=$?" +echo "file now: $(openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -fingerprint -sha256 -enddate | tr '\n' ' ')" +sleep 5 +echo "served 5 s later, no restart: $(echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256)" +echo "pveproxy: $(systemctl show pveproxy -p ActiveEnterTimestamp --value) reloads: $(journalctl -u pveproxy --since '-1 min' --no-pager -o cat | tail -3 | tr '\n' ';')" +echo "done $(date -u +%FT%TZ)" +}; main /dev/null | openssl x509 -noout -fingerprint -sha256 | cut -d= -f2) +MODE=$mode FP=$FP python3 - <<'PY' +import json,os +f='/etc/felhom-agent/agent.json'; c=json.load(open(f)); st=os.stat(f) +if os.environ['MODE']=='fp': c['proxmox']['tls']={'fingerprint':os.environ['FP'],'insecure_skip_verify':False} +else: c['proxmox']['tls']={'ca_file':'/etc/ssl/certs/ISRG_Root_X1.pem','fingerprint':'','insecure_skip_verify':False} +tmp=f+'.tmp'; fd=os.open(tmp,os.O_WRONLY|os.O_CREAT|os.O_TRUNC,0o600) +with os.fdopen(fd,'w') as o: json.dump(c,o,indent=2); o.write('\n') +os.chown(tmp,st.st_uid,st.st_gid); os.rename(tmp,f); print('written tls:',c['proxmox']['tls']) +PY +systemctl restart felhom-agent +n=0; mark=$(date '+%d/%m/%Y:%H:%M:%S'); until [ $n -ge 60 ]; do l=$(grep 'felhom-agent@pve' /var/log/pveproxy/access.log | tail -1); ts=$(echo "$l" | awk '{print $4}' | tr -d '['); + if [ "$(date -d "$(echo $ts | sed 's#\([0-9]*\)/\([0-9]*\)/\([0-9]*\):#\3-\2-\1 #')" +%s 2>/dev/null || echo 0)" -ge "${t0%.*}" ]; then echo "first agent call after restart: $(echo "$l" | awk '{print $4,$6,$7,$9}')"; break; fi; n=$((n+1)); sleep 1; done +t1=$(date +%s.%N); echo "seconds from start to a working call: $(echo "$t1 - $t0" | bc) (loops: $n; 60 = no call seen)" +echo "agent: $(systemctl is-active felhom-agent)" +}; main "$@" /dev/null | openssl x509 -noout -fingerprint -sha256 | cut -d= -f2) +MODE=$mode FP=$FP python3 - <<'PY' +import json,os +f='/etc/felhom-agent/agent.json'; c=json.load(open(f)); st=os.stat(f) +if os.environ['MODE']=='fp': c['proxmox']['tls']={'fingerprint':os.environ['FP'],'insecure_skip_verify':False} +else: c['proxmox']['tls']={'ca_file':'/etc/felhom-agent/pve-root-ca.pem','fingerprint':'','insecure_skip_verify':False} +tmp=f+'.tmp'; fd=os.open(tmp,os.O_WRONLY|os.O_CREAT|os.O_TRUNC,0o600) +with os.fdopen(fd,'w') as o: json.dump(c,o,indent=2); o.write('\n') +os.chown(tmp,st.st_uid,st.st_gid); os.rename(tmp,f); print('written tls:',c['proxmox']['tls']) +PY +systemctl restart felhom-agent +n=0; mark=$(date '+%d/%m/%Y:%H:%M:%S'); until [ $n -ge 60 ]; do l=$(grep 'felhom-agent@pve' /var/log/pveproxy/access.log | tail -1); ts=$(echo "$l" | awk '{print $4}' | tr -d '['); + if [ "$(date -d "$(echo $ts | sed 's#\([0-9]*\)/\([0-9]*\)/\([0-9]*\):#\3-\2-\1 #')" +%s 2>/dev/null || echo 0)" -ge "${t0%.*}" ]; then echo "first agent call after restart: $(echo "$l" | awk '{print $4,$6,$7,$9}')"; break; fi; n=$((n+1)); sleep 1; done +t1=$(date +%s.%N); echo "seconds from start to a working call: $(echo "$t1 - $t0" | bc) (loops: $n; 60 = no call seen)" +echo "agent: $(systemctl is-active felhom-agent)" +}; main "$@" /dev/null | openssl x509 -noout -subject -issuer -dates -fingerprint -sha256 2>/dev/null +python3 -c " +import json;c=json.load(open('/etc/felhom-agent/agent.json'));print('agent tls config:',c['proxmox']['tls'])" +echo "agent: $(systemctl is-active felhom-agent) since $(systemctl show felhom-agent -p ActiveEnterTimestamp --value) restarts=$(systemctl show felhom-agent -p NRestarts --value)" +echo "-- agent calls in pveproxy's access log, last 4 (time, call, answer code)" +grep 'felhom-agent@pve' /var/log/pveproxy/access.log | tail -4 | awk '{print $4,$5,$6,$7,$9}' +echo "-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6)" +journalctl -u felhom-agent --since '-3 min' --no-pager -o cat | grep -v 'pam_unix\|COMMAND=' | grep -i -c 'pin mismatch\|x509\|tls' +journalctl -u felhom-agent --since '-3 min' --no-pager -o cat | grep -v 'pam_unix\|COMMAND=' | grep -i 'pin mismatch\|x509\|tls' | cut -c1-330 | tail -6 +echo "-- guests" +pct list | awk 'NR>1{print $1,$2,$NF}'; qm list | awk 'NR>1{print $1,$2,$3}' +}; main 4.1.1-pmx1 +2026-08-21 15:41:58 install proxmox-kernel-7.0.2-6-pve-signed:amd64 7.0.2-6 +2026-10-04 09:42:20 install proxmox-kernel-7.0.14-20-pve-signed:amd64 7.0.14-20 +2026-10-10 04:40:04 upgrade chrony:amd64 4.6.1-3+deb13u1 4.8-4~bpo13+2 +2026-10-10 04:41:43 install proxmox-kernel-7.0.14-22-pve-signed:amd64 7.0.14-22 +2026-10-11 04:42:12 install proxmox-kernel-7.0.14-23-pve-signed:amd64 7.0.14-23 +2026-10-16 04:48:58 install proxmox-kernel-7.0.14-24-pve-signed:amd64 7.0.14-24 +2026-10-21 04:45:13 install proxmox-kernel-7.0.14-25-pve-signed:amd64 7.0.14-25 +2026-10-29 05:06:43 install proxmox-kernel-7.0.14-26-pve-signed:amd64 7.0.14-26 +2026-11-10 02:50:13 install proxmox-kernel-7.0.14-28-pve-signed:amd64 7.0.14-28 +## kernels booted +IDX BOOT ID FIRST ENTRY LAST ENTRY + -5 182af91122a941ec97ac7ee247a4d470 Sun 2026-10-04 21:42:08 CEST Sun 2026-10-04 21:43:33 CEST + -4 bb68aee4240244718d8a7f39d999a3a8 Wed 2026-11-04 21:06:10 CET Tue 2026-11-10 02:53:28 CET + -3 b6e0edea2b5647dfa36f5bb3825adc3a Tue 2026-11-10 02:53:57 CET Tue 2026-11-10 13:56:35 CET + -2 d1167d9431bf459abf2a569e8997a569 Tue 2026-11-10 13:57:04 CET Tue 2026-11-10 15:32:31 CET + -1 e2cc9e3e60f14bde8949f01d7ba35ccf Tue 2026-11-10 15:32:59 CET Tue 2026-11-10 15:42:26 CET + 0 bed106f79e804eb1ace314a26da7f049 Tue 2026-11-10 15:42:54 CET Wed 2026-11-11 14:16:04 CET +br-1359a945d146: port 1(vethf5b125d) entered disabled state +Linux version 7.0.14-28-pve (build@proxmox) (gcc (Debian 14. +Linux version 7.0.14-28-pve (build@proxmox) (gcc (Debian 14. +Linux version 7.0.14-26-pve (build@proxmox) (gcc (Debian 14. +Linux version 7.0.14-28-pve (build@proxmox) (gcc (Debian 14. +## chronyd confinement now +/usr/sbin/chronyd (enforce) 780 ? 00:00:00 chronyd +/usr/sbin/chronyd (enforce) 903 ? 00:00:00 chronyd +## what installed chrony 4.8 +Start-Date: 2026-10-10 04:39:59 +Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --only-upgrade --no-install-recommends frr=10.6.1-1+pve3 libradosstriper1=19.2.6-pve4 librgw2=19.2.6-pve4 ceph-common=19.2.6-pve4 librbd1=19.2.6-pve4 libra +Requested-By: felhom-agent (999) +Upgrade: librados2:amd64 (19.2.3-pve4, 19.2.6-pve4), pve-docs:amd64 (9.2.1, 9.2.13), pve-edk2-firmware-ovmf:amd64 (4.2025.05-2, 4.2026.08-1), ceph-fuse:amd64 (19.2.3-pve4, 19.2.6-pve4), proxmox-widget-toolkit:amd64 (5.2.2, 5.2.10), corosync:amd64 (3.1.10-pve2, +Start-Date: 2026-10-10 04:41:43 diff --git a/documentation/audits/n100-clock-2026-11-11/evidence/demo-hp-host-2.txt b/documentation/audits/n100-clock-2026-11-11/evidence/demo-hp-host-2.txt new file mode 100644 index 00000000..71116942 --- /dev/null +++ b/documentation/audits/n100-clock-2026-11-11/evidence/demo-hp-host-2.txt @@ -0,0 +1,232 @@ +#### demo-hp 2026-11-11T13:11:00Z +net.unix.max_dgram_qlen = 512 +kernel.printk_ratelimit = 5 +## cmdline +BOOT_IMAGE=/boot/vmlinuz-7.0.14-28-pve root=/dev/mapper/pve-root ro quiet +## apparmor denied this boot: 4 +Nov 10 15:43:11 demo-hp kernel: audit: type=1400 audit(1794321791.730:129): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=2094 comm="vgs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined" +Nov 10 15:43:11 demo-hp kernel: audit: type=1400 audit(1794321791.782:130): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=2095 comm="lvs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined" +Nov 10 15:43:11 demo-hp kernel: audit: type=1400 audit(1794321791.923:131): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=2111 comm="vgs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined" +Nov 10 15:43:11 demo-hp kernel: audit: type=1400 audit(1794321791.986:132): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=2112 comm="lvs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined" +## audit lines with chrony, all boots +## the second around the chrony start, this boot +start: 2026-11-10T15:42:59+01:00 +[ 8.825037] demo-hp kernel: input: HD-Audio Generic HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:08.1/0000:05:00.1/sound/card0/input9 +[ 8.826027] demo-hp kernel: input: HD-Audio Generic HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:08.1/0000:05:00.1/sound/card0/input10 +[ 8.826114] demo-hp kernel: input: HD-Audio Generic HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:08.1/0000:05:00.1/sound/card0/input11 +[ 8.827023] demo-hp kernel: input: HD-Audio Generic HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:08.1/0000:05:00.1/sound/card0/input12 +[ 8.831933] demo-hp systemd[1]: Reached target sound.target - Sound Card. +[ 8.837028] demo-hp kernel: amdgpu: Virtual CRAT table created for CPU +[ 8.837135] demo-hp kernel: amdgpu: Topology: Add CPU node +[ 8.837161] demo-hp kernel: amdgpu 0000:05:00.0: enabling device (0106 -> 0107) +[ 8.837625] demo-hp kernel: amdgpu 0000:05:00.0: initializing kernel modesetting (RAVEN 0x1002:0x15DD 0x103C:0x8522 0x82). +[ 8.838111] demo-hp kernel: amdgpu 0000:05:00.0: register mmio base: 0xFCB00000 +[ 8.838441] demo-hp kernel: amdgpu 0000:05:00.0: register mmio size: 524288 +[ 8.840015] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 0 (soc15_common) +[ 8.840395] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 1 (gmc_v9_0) +[ 8.840735] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 2 (vega10_ih) +[ 8.841064] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 3 (psp) +[ 8.841387] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 4 (powerplay) +[ 8.841727] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 5 (dm) +[ 8.842073] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 6 (gfx_v9_0) +[ 8.842434] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 7 (sdma_v4_0) +[ 8.842772] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 8 (vcn_v1_0) +[ 8.843155] demo-hp kernel: amdgpu 0000:05:00.0: Fetched VBIOS from VFCT +[ 8.843500] demo-hp kernel: amdgpu 0000:05:00.0: [drm] ATOM BIOS: 113-RAVEN-111 +[ 8.843847] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: ALC256: picked fixup (pin match) +[ 8.844233] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: autoconfig for ALC256: line_outs=1 (0x14/0x0/0x0/0x0/0x0) type:speaker +[ 8.844547] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: speaker_outs=0 (0x0/0x0/0x0/0x0/0x0) +[ 8.844874] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: hp_outs=1 (0x21/0x0/0x0/0x0/0x0) +[ 8.845180] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: mono: mono_out=0x0 +[ 8.845473] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: inputs: +[ 8.845756] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: Mic=0x19 +[ 8.870030] demo-hp kernel: Console: switching to colour dummy device 80x25 +[ 8.871013] demo-hp kernel: amdgpu 0000:05:00.0: vgaarb: deactivate vga console +[ 8.871457] demo-hp kernel: amdgpu 0000:05:00.0: Trusted Memory Zone (TMZ) feature enabled +[ 8.871771] demo-hp kernel: amdgpu 0000:05:00.0: vm size is 262144 GB, 4 levels, block size is 9-bit, fragment size is 9-bit +[ 8.872113] demo-hp kernel: amdgpu 0000:05:00.0: VRAM: 2048M 0x000000F400000000 - 0x000000F47FFFFFFF (2048M used) +[ 8.872452] demo-hp kernel: amdgpu 0000:05:00.0: GART: 1024M 0x0000000000000000 - 0x000000003FFFFFFF +[ 8.872757] demo-hp kernel: amdgpu 0000:05:00.0: [drm] Detected VRAM RAM=2048M, BAR=2048M +[ 8.873106] demo-hp kernel: amdgpu 0000:05:00.0: [drm] RAM width 128bits DDR4 +[ 8.873432] demo-hp kernel: amdgpu 0000:05:00.0: 2048M of VRAM memory ready +[ 8.873746] demo-hp kernel: amdgpu 0000:05:00.0: 14997M of GTT memory ready. +[ 8.874109] demo-hp kernel: amdgpu 0000:05:00.0: [drm] GART: num cpu pages 262144, num gpu pages 262144 +[ 8.874443] demo-hp kernel: amdgpu 0000:05:00.0: [drm] PCIE GART of 1024M enabled. +[ 8.874765] demo-hp kernel: amdgpu 0000:05:00.0: [drm] PTB located at 0x000000F400A00000 +[ 8.875104] demo-hp kernel: amdgpu: hwmgr_sw_init smu backed is smu10_smu +[ 8.875136] demo-hp kernel: amdgpu 0000:05:00.0: [VCN instance 0] Found VCN firmware Version ENC: 1.15 DEC: 3 VEP: 0 Revision: 0 +[ 8.882025] demo-hp kernel: input: HD-Audio Generic Mic as /devices/pci0000:00/0000:00:08.1/0000:05:00.6/sound/card1/input13 +[ 8.882116] demo-hp kernel: input: HD-Audio Generic Front Headphone as /devices/pci0000:00/0000:00:08.1/0000:05:00.6/sound/card1/input14 +[ 8.895018] demo-hp kernel: amdgpu 0000:05:00.0: reserve 0x400000 from 0xf47fc00000 for PSP TMR +[ 8.974285] demo-hp kernel: amdgpu 0000:05:00.0: RAS: optional ras ta ucode is not available +[ 8.982279] demo-hp kernel: amdgpu 0000:05:00.0: RAP: optional rap ta ucode is not available +[ 8.982905] demo-hp kernel: amdgpu 0000:05:00.0: SECUREDISPLAY: optional securedisplay ta ucode is not available +[ 8.990029] demo-hp kernel: [drm] DM_PPLIB: values for F clock +[ 8.990101] demo-hp kernel: [drm] DM_PPLIB: 400000 in kHz, 3599 in mV +[ 8.990150] demo-hp kernel: [drm] DM_PPLIB: 933000 in kHz, 3724 in mV +[ 8.990192] demo-hp kernel: [drm] DM_PPLIB: 1067000 in kHz, 3924 in mV +[ 8.990225] demo-hp kernel: [drm] DM_PPLIB: 1200000 in kHz, 4074 in mV +[ 8.990252] demo-hp kernel: [drm] DM_PPLIB: values for DCF clock +[ 8.990295] demo-hp kernel: [drm] DM_PPLIB: 300000 in kHz, 3599 in mV +[ 8.990330] demo-hp kernel: [drm] DM_PPLIB: 600000 in kHz, 3724 in mV +[ 8.990379] demo-hp kernel: [drm] DM_PPLIB: 626000 in kHz, 3924 in mV +[ 8.990415] demo-hp kernel: [drm] DM_PPLIB: 654000 in kHz, 4074 in mV +[ 8.990720] demo-hp kernel: amdgpu 0000:05:00.0: [drm] Display Core v3.2.369 initialized on DCN 1.0 +[ 8.994046] demo-hp kernel: snd_hda_intel 0000:05:00.1: bound 0000:05:00.0 (ops amdgpu_dm_audio_component_bind_ops [amdgpu]) +[ 9.036409] demo-hp kernel: amdgpu 0000:05:00.0: kiq ring mec 2 pipe 1 q 0 +[ 9.053051] demo-hp kernel: kfd kfd: Allocated 3969056 bytes on gart +[ 9.053414] demo-hp kernel: kfd kfd: Total number of KFD nodes to be created: 1 +[ 9.053588] demo-hp kernel: amdgpu: Virtual CRAT table created for GPU +[ 9.053620] demo-hp kernel: amdgpu: Topology: Add GPU node [0x1002:0x15dd] +[ 9.053643] demo-hp kernel: kfd kfd: added device 1002:15dd +[ 9.053792] demo-hp kernel: amdgpu 0000:05:00.0: SE 1, SH per SE 1, CU per SH 11, active_cu_number 8 +[ 9.054099] demo-hp kernel: amdgpu 0000:05:00.0: ring gfx uses VM inv eng 0 on hub 0 +[ 9.054373] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.0.0 uses VM inv eng 1 on hub 0 +[ 9.054628] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.1.0 uses VM inv eng 4 on hub 0 +[ 9.054881] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.2.0 uses VM inv eng 5 on hub 0 +[ 9.055139] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.3.0 uses VM inv eng 6 on hub 0 +[ 9.055384] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.0.1 uses VM inv eng 7 on hub 0 +[ 9.055641] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.1.1 uses VM inv eng 8 on hub 0 +[ 9.055897] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.2.1 uses VM inv eng 9 on hub 0 +[ 9.056158] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.3.1 uses VM inv eng 10 on hub 0 +[ 9.056432] demo-hp kernel: amdgpu 0000:05:00.0: ring kiq_0.2.1.0 uses VM inv eng 11 on hub 0 +[ 9.056680] demo-hp kernel: amdgpu 0000:05:00.0: ring sdma0 uses VM inv eng 0 on hub 8 +[ 9.056932] demo-hp kernel: amdgpu 0000:05:00.0: ring vcn_dec uses VM inv eng 1 on hub 8 +[ 9.057198] demo-hp kernel: amdgpu 0000:05:00.0: ring vcn_enc0 uses VM inv eng 4 on hub 8 +[ 9.057469] demo-hp kernel: amdgpu 0000:05:00.0: ring vcn_enc1 uses VM inv eng 5 on hub 8 +[ 9.057727] demo-hp kernel: amdgpu 0000:05:00.0: ring jpeg_dec uses VM inv eng 6 on hub 8 +[ 9.060024] demo-hp kernel: amdgpu: pp_dpm_get_sclk_od was not implemented. +[ 9.060097] demo-hp kernel: amdgpu: pp_dpm_get_mclk_od was not implemented. +[ 9.060121] demo-hp kernel: amdgpu 0000:05:00.0: Runtime PM not available +[ 9.061030] demo-hp kernel: amdgpu 0000:05:00.0: [drm] Registered 4 planes with drm panic +[ 9.068049] demo-hp kernel: [drm] Initialized amdgpu 3.64.0 for 0000:05:00.0 on minor 1 +[ 9.081054] demo-hp kernel: amdgpu 0000:05:00.0: [drm] Failed to setup vendor infoframe on connector DP-1: -22 +[ 9.086060] demo-hp kernel: fbcon: amdgpudrmfb (fb0) is primary device +[ 9.131047] demo-hp kernel: Console: switching to colour frame buffer device 240x67 +[ 9.147045] demo-hp kernel: amdgpu 0000:05:00.0: [drm] fb0: amdgpudrmfb frame buffer device +[ 9.170506] demo-hp systemd[1]: Finished ifupdown2-pre.service - Helper to synchronize boot up for ifupdown. +[ 9.171027] demo-hp systemd[1]: Finished systemd-udev-settle.service - Wait for udev To Complete Device Initialization. +[ 9.171890] demo-hp systemd[1]: zfs-import-cache.service - Import ZFS pools by cache file skipped, unmet condition check ConditionFileNotEmpty=/etc/zfs/zpool.cache +[ 9.172055] demo-hp systemd[1]: Reached target zfs-import.target - ZFS pool import target. +[ 9.179591] demo-hp systemd[1]: Starting zfs-mount.service - Mount ZFS filesystems... +[ 9.181286] demo-hp systemd[1]: Starting zfs-volume-wait.service - Wait for ZFS Volume (zvol) links in /dev... +[ 9.207137] demo-hp zvol_wait[623]: No zvols found, nothing to do. +[ 9.207944] demo-hp systemd[1]: Finished zfs-mount.service - Mount ZFS filesystems. +[ 9.208529] demo-hp systemd[1]: Finished zfs-volume-wait.service - Wait for ZFS Volume (zvol) links in /dev. +[ 9.209473] demo-hp systemd[1]: Reached target local-fs.target - Local File Systems. +[ 9.209606] demo-hp systemd[1]: Reached target zfs-volumes.target - ZFS volumes are ready. +[ 9.211049] demo-hp systemd[1]: Listening on systemd-sysext.socket - System Extension Image Management. +[ 9.212846] demo-hp systemd[1]: Starting apparmor.service - Load AppArmor profiles... +[ 9.214592] demo-hp systemd[1]: Starting console-setup.service - Set console font and keymap... +[ 9.216370] demo-hp systemd[1]: Starting felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit)... +[ 9.218403] demo-hp systemd[1]: Starting networking.service - Network initialization... +[ 9.220586] demo-hp systemd[1]: Starting pvebanner.service - Proxmox VE Login Banner... +[ 9.222876] demo-hp systemd[1]: Starting pvefw-logger.service - Proxmox VE firewall logger... +[ 9.225689] demo-hp systemd[1]: Starting pvenetcommit.service - Commit Proxmox VE network changes... +[ 9.228763] demo-hp systemd[1]: Starting systemd-binfmt.service - Set Up Additional Binary Formats... +[ 9.230821] demo-hp systemd[1]: Starting systemd-tmpfiles-setup.service - Create System Files and Directories... +[ 9.233511] demo-hp systemd[1]: Finished console-setup.service - Set console font and keymap. +[ 9.242393] demo-hp apparmor.systemd[630]: Restarting AppArmor +[ 9.242393] demo-hp apparmor.systemd[630]: Reloading AppArmor profiles +[ 9.248120] demo-hp systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 640 (systemd-binfmt) +[ 9.250462] demo-hp systemd-tmpfiles[641]: /etc/tmpfiles.d/felhom-privsep.conf:15: Duplicate line for path "/run/sshd", ignoring. +[ 9.250689] demo-hp systemd-tmpfiles[641]: /usr/lib/tmpfiles.d/legacy.conf:14: Duplicate line for path "/run/lock", ignoring. +[ 9.251172] demo-hp systemd[1]: Mounting proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System... +[ 9.251836] demo-hp systemd[1]: Finished pvenetcommit.service - Commit Proxmox VE network changes. +[ 9.253408] demo-hp networking[633]: networking: Configuring network interfaces +[ 9.272681] demo-hp pvefw-logger[668]: starting pvefw logger +[ 9.274360] demo-hp systemd[1]: Started pvefw-logger.service - Proxmox VE firewall logger. +[ 9.276036] demo-hp kernel: audit: type=1400 audit(1794321779.088:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="Discord" pid=660 comm="apparmor_parser" +[ 9.276172] demo-hp kernel: audit: type=1400 audit(1794321779.088:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name=4D6F6E676F444220436F6D70617373 pid=661 comm="apparmor_pars +[ 9.276224] demo-hp kernel: audit: type=1400 audit(1794321779.088:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="QtWebEngineProcess" pid=662 comm="apparmor_parser" +[ 9.277127] demo-hp kernel: audit: type=1400 audit(1794321779.089:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="1password" pid=659 comm="apparmor_parser" +[ 9.277255] demo-hp kernel: audit: type=1400 audit(1794321779.090:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="balena-etcher" pid=664 comm="apparmor_parser" +[ 9.277292] demo-hp kernel: audit: type=1400 audit(1794321779.090:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="buildah" pid=666 comm="apparmor_parser" +[ 9.277340] demo-hp kernel: audit: type=1400 audit(1794321779.090:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="brave" pid=665 comm="apparmor_parser" +[ 9.279225] demo-hp kernel: audit: type=1400 audit(1794321779.091:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="cam" pid=672 comm="apparmor_parser" +[ 9.279392] demo-hp kernel: audit: type=1400 audit(1794321779.092:12): apparmor="STATUS" operation="profile_load" profile="unconfined" name="ch-checkns" pid=673 comm="apparmor_parser" +[ 9.279919] demo-hp systemd[1]: Finished systemd-tmpfiles-setup.service - Create System Files and Directories. +[ 9.282035] demo-hp kernel: audit: type=1400 audit(1794321779.094:13): apparmor="STATUS" operation="profile_load" profile="unconfined" name="busybox" pid=671 comm="apparmor_parser" +[ 9.284316] demo-hp systemd[1]: Mounting run-rpc_pipefs.mount - RPC Pipe File System... +[ 9.284460] demo-hp systemd[1]: ldconfig.service - Rebuild Dynamic Linker Cache skipped, no trigger condition checks were met. +[ 9.287648] demo-hp systemd[1]: Starting rpcbind.service - RPC bind portmap service... +[ 9.287715] demo-hp systemd[1]: systemd-firstboot.service - First Boot Wizard skipped, unmet condition check ConditionFirstBoot=yes +[ 9.287753] demo-hp systemd[1]: first-boot-complete.target - First Boot Complete skipped, unmet condition check ConditionFirstBoot=yes +[ 9.287841] demo-hp systemd[1]: systemd-journal-catalog-update.service - Rebuild Journal Catalog skipped, unmet condition check ConditionNeedsUpdate=/var +[ 9.287922] demo-hp systemd[1]: systemd-machine-id-commit.service - Save Transient machine-id to Disk skipped, unmet condition check ConditionPathIsMountPoint=/etc/machine-id +[ 9.288596] demo-hp systemd[1]: systemd-update-done.service - Update is Completed skipped, no trigger condition checks were met. +[ 9.290860] demo-hp systemd[1]: Mounted proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System. +[ 9.303595] demo-hp systemd[1]: Finished systemd-binfmt.service - Set Up Additional Binary Formats. +[ 9.317151] demo-hp systemd[1]: Started rpcbind.service - RPC bind portmap service. +[ 9.317289] demo-hp systemd[1]: Reached target rpcbind.target - RPC Port Mapper. +[ 9.364582] demo-hp felhom-crash-guard[632]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10 +[ 9.375481] demo-hp felhom-crash-guard[775]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10 +## chrony lines per boot (all boots) +-- boot -5 +-- boot -4 +Nov 10 02:50:56 demo-hp chronyd[774]: chronyd exiting +Nov 10 02:50:56 demo-hp systemd[1]: Stopping chrony.service - chrony, an NTP client/server... +Nov 10 02:50:56 demo-hp systemd[1]: Stopped chrony.service - chrony, an NTP client/server. +-- boot -3 +Nov 10 02:54:02 demo-hp systemd[1]: Starting chrony.service - chrony, an NTP client/server... +Nov 10 02:54:02 demo-hp systemd[1]: Started chrony.service - chrony, an NTP client/server. +Nov 10 13:54:26 demo-hp chronyd[778]: chronyd exiting +Nov 10 13:54:26 demo-hp systemd[1]: Stopping chrony.service - chrony, an NTP client/server... +Nov 10 13:54:26 demo-hp systemd[1]: Stopped chrony.service - chrony, an NTP client/server. +-- boot -2 +Nov 10 13:57:09 demo-hp systemd[1]: Starting chrony.service - chrony, an NTP client/server... +Nov 10 13:57:09 demo-hp systemd[1]: Started chrony.service - chrony, an NTP client/server. +Nov 10 15:30:24 demo-hp chronyd[778]: chronyd exiting +Nov 10 15:30:24 demo-hp systemd[1]: Stopping chrony.service - chrony, an NTP client/server... +Nov 10 15:30:24 demo-hp systemd[1]: Stopped chrony.service - chrony, an NTP client/server. +-- boot -1 +Nov 10 15:33:04 demo-hp systemd[1]: Starting chrony.service - chrony, an NTP client/server... +Nov 10 15:33:04 demo-hp systemd[1]: Started chrony.service - chrony, an NTP client/server. +Nov 10 15:40:09 demo-hp chronyd[780]: chronyd exiting +Nov 10 15:40:09 demo-hp systemd[1]: Stopping chrony.service - chrony, an NTP client/server... +Nov 10 15:40:09 demo-hp systemd[1]: Stopped chrony.service - chrony, an NTP client/server. +-- boot 0 +Nov 10 15:42:59 demo-hp systemd[1]: Starting chrony.service - chrony, an NTP client/server... +Nov 10 15:42:59 demo-hp systemd[1]: Started chrony.service - chrony, an NTP client/server. +## sandbox env of pid1 +total 0 +drwxr-xr-x 20 root root 540 Nov 11 14:07 . +drwxr-xr-x 36 root root 1520 Nov 11 14:11 .. +drwxr-xr-x 2 root root 40 Nov 10 15:42 ask-password +drwxr-xr-x 6 root root 260 Nov 10 15:58 generator +dr-xr-xr-x 3 root root 160 Nov 10 15:42 inaccessible +drwxr-xr-x 2 root root 40 Nov 10 15:42 incoming +drwxr-xr-x 2 root root 40 Nov 10 15:42 inhibit +srw-rw-rw- 1 root root 0 Nov 10 15:42 io.systemd.Credentials +srw-rw-rw- 1 root root 0 Nov 10 15:42 io.systemd.Hostname +srw-rw-rw- 1 root root 0 Nov 10 15:42 io.systemd.ManagedOOM +srw------- 1 root root 0 Nov 10 15:42 io.systemd.sysext +drwxr-xr-x 3 root root 200 Nov 10 15:42 journal +drwxr-xr-x 2 root root 40 Nov 10 15:42 machines +dr-xr-xr-x 2 root root 40 Nov 10 15:42 mount-rootfs +drwxr-xr-x 4 systemd-network systemd-network 80 Nov 10 15:42 netif +srwxrwxrwx 1 root root 0 Nov 10 15:42 notify +srwx------ 1 root root 0 Nov 10 15:42 private +drwxr-xr-x 7 root root 140 Nov 11 14:08 propagate +drwxr-xr-x 2 root root 60 Nov 10 15:42 seats +drwxr-xr-x 2 root root 120 Nov 11 14:11 sessions +drwxr-xr-x 2 root root 40 Nov 10 15:42 shutdown +drwxr-xr-x 2 root root 40 Nov 10 15:42 system +-r--r--r-- 1 root root 0 Nov 10 15:58 systemd-units-load +drwxr-xr-x 2 root root 100 Nov 11 14:10 transient +drwxr-xr-x 2 root root 1940 Nov 11 14:11 units +drwxr-xr-x 2 root root 60 Nov 10 15:42 userdb +drwxr-xr-x 2 root root 60 Nov 11 14:11 users +## tailscale +inactive +## rtc +ls: cannot access '/dev/pps*': No such file or directory +/dev/ptp0 +/dev/ptp1 +/dev/ptp2 +/dev/ptp3 +/dev/rtc +/dev/rtc0 +## cpu +8 +model name : AMD Ryzen Embedded V1756B with Radeon Vega Gfx diff --git a/documentation/audits/n100-clock-2026-11-11/evidence/demo-hp-host.txt b/documentation/audits/n100-clock-2026-11-11/evidence/demo-hp-host.txt new file mode 100644 index 00000000..190cbf41 --- /dev/null +++ b/documentation/audits/n100-clock-2026-11-11/evidence/demo-hp-host.txt @@ -0,0 +1,273 @@ +#### host demo-hp 2026-11-11T13:07:29Z local 14:07:29+0100 +## versions +pve-manager/9.2.26/41a0e2e09b9e6f9d (running kernel: 7.0.14-28-pve) +ii chrony 4.8-4~bpo13+2 +ii systemd 257.13-1~deb13u1 +un systemd-timesyncd +7.0.14-28-pve +## unit state +Type=notify +PIDFile=/run/chrony/chronyd.pid +NotifyAccess=main +Result=success +NRestarts=0 +ExecMainCode=0 +ExecMainStatus=0 +ActiveState=active +SubState=running +FragmentPath=/usr/lib/systemd/system/chrony.service +DropInPaths= +UnitFileState=enabled +ActiveEnterTimestamp=Tue 2026-11-10 15:42:59 CET +InactiveEnterTimestamp= +## unit cat +# /usr/lib/systemd/system/chrony.service +[Unit] +Description=chrony, an NTP client/server +Documentation=man:chronyd(8) man:chronyc(1) man:chrony.conf(5) +Conflicts=openntpd.service ntp.service ntpsec.service +ConditionVirtualization=|!container +ConditionVirtualization=|wsl +ConditionCapability=CAP_SYS_TIME + +[Service] +Type=notify +PIDFile=/run/chrony/chronyd.pid +EnvironmentFile=-/etc/default/chrony +User=_chrony +# Daemon is started as root, but still sandboxed +ExecStart=!/usr/sbin/chronyd -n $DAEMON_OPTS + +CapabilityBoundingSet=~CAP_AUDIT_CONTROL CAP_AUDIT_READ CAP_AUDIT_WRITE +CapabilityBoundingSet=~CAP_BLOCK_SUSPEND CAP_KILL CAP_LEASE CAP_LINUX_IMMUTABLE +CapabilityBoundingSet=~CAP_MAC_ADMIN CAP_MAC_OVERRIDE CAP_MKNOD CAP_SYS_ADMIN +CapabilityBoundingSet=~CAP_SYS_BOOT CAP_SYS_CHROOT CAP_SYS_MODULE CAP_SYS_PACCT +CapabilityBoundingSet=~CAP_SYS_PTRACE CAP_SYS_RAWIO CAP_SYS_TTY_CONFIG CAP_WAKE_ALARM +DeviceAllow=char-pps rw +DeviceAllow=char-ptp rw +DeviceAllow=char-rtc rw +DevicePolicy=closed +LockPersonality=yes +MemoryDenyWriteExecute=yes +NoNewPrivileges=yes +PrivateTmp=yes +ProcSubset=pid +ProtectControlGroups=yes +ProtectHome=yes +ProtectHostname=yes +ProtectKernelLogs=yes +ProtectKernelModules=yes +ProtectKernelTunables=yes +ProtectProc=invisible +ProtectSystem=strict +# Used for gps refclocks +ReadWritePaths=/run +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX +RestrictNamespaces=yes +RestrictSUIDSGID=yes +SystemCallArchitectures=native +SystemCallFilter=~@cpu-emulation @debug @module @mount @obsolete @raw-io @reboot @swap + +ConfigurationDirectory=chrony +RuntimeDirectory=chrony +RuntimeDirectoryMode=0700 +# See dumpdir in chrony.conf(5) +RuntimeDirectoryPreserve=restart +StateDirectory=chrony +StateDirectoryMode=0750 +LogsDirectory=chrony +LogsDirectoryMode=0750 + +# Adjust restrictions for /usr/sbin/sendmail (mailonchange directive) +NoNewPrivileges=no +ReadWritePaths=-/var/spool +RestrictAddressFamilies=AF_NETLINK + +[Install] +Alias=chronyd.service +WantedBy=multi-user.target +## files +ls: cannot access '/etc/systemd/system/chrony.service.d': No such file or directory +ls: cannot access '/etc/systemd/system/chronyd.service.d': No such file or directory +ls: cannot access '/etc/systemd/system/chrony.service': No such file or directory +lrwxrwxrwx 1 root root 38 May 21 13:32 /etc/systemd/system/chronyd.service -> /usr/lib/systemd/system/chrony.service +-rw-r--r-- 1 root root 1929 Jul 1 22:28 /lib/systemd/system/chrony.service +-rw-r--r-- 1 root root 1929 Jul 1 22:28 /usr/lib/systemd/system/chrony.service +## dpkg verify chrony +rc=0 +## conf +/etc/chrony: +total 24 +drwxr-xr-x 4 root root 4096 May 21 13:32 . +drwxr-xr-x 102 root root 4096 Nov 8 02:49 .. +-rw-r--r-- 1 root root 1450 Feb 16 2026 chrony.conf +-rw-r----- 1 root _chrony 481 Feb 16 2026 chrony.keys +drwxr-xr-x 2 root root 4096 Oct 10 04:40 conf.d +drwxr-xr-x 2 root root 4096 Oct 10 04:40 sources.d + +/etc/chrony/conf.d: +total 12 +drwxr-xr-x 2 root root 4096 Oct 10 04:40 . +drwxr-xr-x 4 root root 4096 May 21 13:32 .. +-rw-r--r-- 1 root root 422 Feb 16 2026 README + +/etc/chrony/sources.d: +total 12 +drwxr-xr-x 2 root root 4096 Oct 10 04:40 . +drwxr-xr-x 4 root root 4096 May 21 13:32 .. +-rw-r--r-- 1 root root 498 Feb 16 2026 README +pool 2.debian.pool.ntp.org iburst +sourcedir /run/chrony-dhcp +sourcedir /etc/chrony/sources.d +keyfile /etc/chrony/chrony.keys +driftfile /var/lib/chrony/chrony.drift +ntsdumpdir /var/lib/chrony +logdir /var/log/chrony +maxupdateskew 100.0 +rtcsync +makestep 1 3 +leapseclist /usr/share/zoneinfo/leap-seconds.list +confdir /etc/chrony/conf.d +-- /etc/chrony/conf.d/README +Files found under the /etc/chrony/conf.d directory with the .conf suffix are +parsed in the lexicographical order of the file names when chronyd starts up. +This enables a fragmented configuration of chronyd. +Although those files can contain any directives listed in chrony.conf(5), +it would be wiser to add NTP sources in the /etc/chrony/sources.d +directory. Please read /etc/chrony/sources.d/README for more information. +-- /etc/chrony/sources.d/README +Only NTP sources can be specified in the /etc/chrony/sources.d directory. +Files in this directory must end with the ".sources" suffix. They can only +contain the "peer", "pool" and "server" directives and must have all +lines terminated by a trailing newline. +There is no need to restart chronyd for these time sources to be usable, +running 'chronyc reload sources' is sufficient. +Example: +## default + +DAEMON_OPTS="-F 1" +## apparmor + /usr/sbin/chronyd + /usr/sbin/chronyd (780) + /usr/sbin/chronyd (903) +-rw-r--r-- 1 root root 0 May 21 13:32 /etc/apparmor.d/local/usr.sbin.chronyd +-rw-r--r-- 1 root root 2837 Jul 1 22:28 /etc/apparmor.d/usr.sbin.chronyd + +/etc/apparmor.d/disable: +total 8 +drwxr-xr-x 2 root root 4096 Jul 21 2025 . +drwxr-xr-x 9 root root 4096 Nov 8 02:48 .. +## apparmor profile notify lines +81: # Allow systemd Type=notify using sd_notify's $NOTIFY_SOCKET +82: @{run}/systemd/notify w, +## kernel audit lines for chronyd (last 20) +## journal chrony this boot +Nov 10 15:42:59 demo-hp systemd[1]: Starting chrony.service - chrony, an NTP client/server... +Nov 10 15:42:59 demo-hp chronyd[780]: chronyd version 4.8 starting (+CMDMON +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +NTS +SECHASH +IPV6 -DEBUG) +Nov 10 15:42:59 demo-hp chronyd[780]: Loaded 0 symmetric keys +Nov 10 15:42:59 demo-hp chronyd[780]: Using leap second list /usr/share/zoneinfo/leap-seconds.list +Nov 10 15:42:59 demo-hp chronyd[780]: Frequency -12.230 +/- 1.327 ppm read from /var/lib/chrony/chrony.drift +Nov 10 15:42:59 demo-hp chronyd[780]: Loaded seccomp filter (level 1) +Nov 10 15:42:59 demo-hp systemd[1]: Started chrony.service - chrony, an NTP client/server. +Nov 10 15:43:10 demo-hp chronyd[780]: Selected source 81.0.124.253 (2.debian.pool.ntp.org) +Nov 10 15:43:10 demo-hp chronyd[780]: System clock TAI offset set to 37 seconds +Nov 10 15:43:11 demo-hp chronyd[780]: Selected source 37.221.209.24 (2.debian.pool.ntp.org) +Nov 10 15:44:16 demo-hp chronyd[780]: Selected source 195.111.92.55 (2.debian.pool.ntp.org) +Nov 10 15:46:26 demo-hp chronyd[780]: Selected source 37.221.209.24 (2.debian.pool.ntp.org) +## first failure ever in the journal +count: 0 +## oldest journal line +Oct 04 21:42:08 demo-hp systemd[779137]: Queued start job for default target def +## last good start in the journal +Nov 10 15:43:11 demo-hp chronyd[780]: Selected source 37.221.209.24 (2.debian.pool.ntp.org) +Nov 10 15:44:16 demo-hp chronyd[780]: Selected source 195.111.92.55 (2.debian.pool.ntp.org) +Nov 10 15:46:26 demo-hp chronyd[780]: Selected source 37.221.209.24 (2.debian.pool.ntp.org) +## boots + -5 182af91122a941ec97ac7ee247a4d470 Sun 2026-10-04 21:42:08 CEST Sun 2026-10-04 21:43:33 CEST + -4 bb68aee4240244718d8a7f39d999a3a8 Wed 2026-11-04 21:06:10 CET Tue 2026-11-10 02:53:28 CET + -3 b6e0edea2b5647dfa36f5bb3825adc3a Tue 2026-11-10 02:53:57 CET Tue 2026-11-10 13:56:35 CET + -2 d1167d9431bf459abf2a569e8997a569 Tue 2026-11-10 13:57:04 CET Tue 2026-11-10 15:32:31 CET + -1 e2cc9e3e60f14bde8949f01d7ba35ccf Tue 2026-11-10 15:32:59 CET Tue 2026-11-10 15:42:26 CET + 0 bed106f79e804eb1ace314a26da7f049 Tue 2026-11-10 15:42:54 CET Wed 2026-11-11 14:07:29 CET +## what starts with it +chrony.service +● ├─-.mount +● ├─system.slice +● ├─systemd-journald.socket +● ├─systemd-remount-fs.service +● ├─systemd-tmpfiles-setup.service +● ├─tmp.mount +● ├─basic.target +● │ ├─-.mount +○ │ ├─systemd-pcrphase-sysinit.service +● │ ├─tmp.mount +● │ ├─paths.target +● │ │ ├─systemd-ask-password-console.path +● │ │ └─systemd-ask-password-wall.path +● │ ├─slices.target +● │ │ ├─-.slice +● │ │ ├─system-xfs_scrub.slice +● │ │ ├─system.slice +● │ │ └─user.slice +● │ ├─sockets.target +● │ │ ├─dbus.socket +● │ │ ├─iscsid.socket +● │ │ ├─netavark-dhcp-proxy.socket +● │ │ ├─rrdcached.socket +● │ │ ├─sshd-unix-local.socket +The time when unit became active or started is printed after the "@" character. +The time the unit took to start is printed after the "+" character. + +chrony.service +174ms +└─basic.target @5.760s + └─sockets.target @5.760s + └─systemd-hostnamed.socket @5.760s + └─sysinit.target @5.757s + └─felhom-crash-guard.service @5.583s +173ms + └─local-fs.target @5.576s + └─etc-pve.mount @7.233s + └─local-fs-pre.target @761ms +## other time services +inactive inactive inactive inactive + Local time: Wed 2026-11-11 14:07:32 CET + Universal time: Wed 2026-11-11 13:07:32 UTC + RTC time: Wed 2026-11-11 13:07:33 + Time zone: Europe/Budapest (CET, +0100) +System clock synchronized: yes + NTP service: active + RTC in local TZ: no +## chronyc +Reference ID : 25DDD118 (37.221.209.24) +Stratum : 3 +Ref time (UTC) : Wed Nov 11 12:51:20 2026 +System time : 0.000228395 seconds fast of NTP time +Last offset : +0.000003178 seconds +RMS offset : 0.000066409 seconds +Frequency : 12.927 ppm slow +Residual freq : +0.000 ppm +MS Name/IP address Stratum Poll Reach LastRx Last sample +=============================================================================== +^- cebece.pomaz.hu 2 10 377 978 -7115us[-7112us] +/- 32ms +^* 37.221.209.24 2 10 377 971 -2744us[-2741us] +/- 15ms +^+ mail.deployis.eu 4 10 377 758 -3122us[-3122us] +/- 24ms +^+ ntp2.niif.hu 2 10 377 377 -4436us[-4436us] +/- 39ms +## dpkg history chrony/apparmor/systemd +2026-08-21 15:41:16 install apparmor:amd64 4.1.1-pmx1 +2026-08-21 15:42:37 configure apparmor:amd64 4.1.1-pmx1 +2026-10-10 04:40:04 upgrade chrony:amd64 4.6.1-3+deb13u1 4.8-4~bpo13+2 +2026-10-10 04:40:23 configure chrony:amd64 4.8-4~bpo13+2 +## felhom files that name chrony +## ps + 780 _chrony /usr/sbin/chronyd -n -F 1 + 903 _chrony /usr/sbin/chronyd -n -F 1 +## run dir +srwxrwxrwx 1 root root 0 Nov 10 15:42 /run/systemd/notify + +/run/chrony: +total 4 +drwx------ 2 _chrony _chrony 80 Nov 11 14:07 . +drwxr-xr-x 36 root root 1520 Nov 11 14:07 .. +-rw-r--r-- 1 root root 4 Nov 10 15:42 chronyd.pid +srwxr-xr-x 1 _chrony _chrony 0 Nov 10 15:42 chronyd.sock +## uptime +2026-11-10 15:42:49 diff --git a/documentation/audits/n100-clock-2026-11-11/evidence/demo-hp-stale-proof.txt b/documentation/audits/n100-clock-2026-11-11/evidence/demo-hp-stale-proof.txt new file mode 100644 index 00000000..659389c3 --- /dev/null +++ b/documentation/audits/n100-clock-2026-11-11/evidence/demo-hp-stale-proof.txt @@ -0,0 +1,26 @@ +#### demo-hp 2026-11-11T13:16:38Z +## fresh compile of the on-disk profile to stdout (no load, cache not read, not written) +83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1 - +## loaded +83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1 + +## cache +dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 /var/cache/apparmor/ae1a4050.0/usr.sbin.chronyd +83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1 /var/cache/apparmor/f6da0254.0/usr.sbin.chronyd +drwx------ 2 root root 4096 2026-08-21 17:44:37.686947147 +0200 /var/cache/apparmor/ae1a4050.0/ +-rw-r--r-- 1 root root 3177 2026-08-21 17:44:37.159696460 +0200 /var/cache/apparmor/ae1a4050.0/.features +drwx------ 2 root root 4096 2026-11-08 02:48:51.017396841 +0100 /var/cache/apparmor/f6da0254.0/ +-rw-r--r-- 1 root root 3219 2026-10-04 09:44:36.115533363 +0200 /var/cache/apparmor/f6da0254.0/.features +## state +active +chrony 4.8-4~bpo13+2 +7.0.14-28-pve +System time : 0.000003214 seconds slow of NTP time +Leap status : Normal +1794402998.136055491 +## chrony fatal lines +0 +## stale check, every cached profile: cache differs from a fresh compile +current dir: /var/cache/apparmor/f6da0254.0/ +STALE: lxc-containers +checked=106 stale=1 diff --git a/documentation/audits/n100-clock-2026-11-11/evidence/n100-apparmor-cache.txt b/documentation/audits/n100-clock-2026-11-11/evidence/n100-apparmor-cache.txt new file mode 100644 index 00000000..9d4737f7 --- /dev/null +++ b/documentation/audits/n100-clock-2026-11-11/evidence/n100-apparmor-cache.txt @@ -0,0 +1,76 @@ +#### demo-felhom 2026-11-11T13:15:31Z +## fresh compile of the on-disk profile, to stdout only (no load, no cache write) +e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 - +apparmor_parser: Too many actions given on the command line. +## loaded raw policy sha256 +dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 + +## cache dirs +total 16 +drwxr-xr-x 4 root root 4096 2026-10-07 12:11:32.040268492 +0200 . +drwxr-xr-x 14 root root 4096 2026-07-19 13:08:02.082082737 +0200 .. +drwx------ 2 root root 4096 2026-10-07 14:16:03.051992948 +0200 ae1a4050.0 +drwx------ 2 root root 4096 2026-11-08 05:32:24.065142783 +0100 f6da0254.0 +/var/cache/apparmor/ae1a4050.0/ features-sha=0e6b37757d22af1e files=106 +-rw-r--r-- 1 root root 3177 2026-07-18 18:17:02.251100923 +0200 /var/cache/apparmor/ae1a4050.0//.features +-rw------- 1 root root 296497 2026-07-18 18:15:53.043202621 +0200 /var/cache/apparmor/ae1a4050.0//usr.sbin.chronyd +/var/cache/apparmor/f6da0254.0/ features-sha=611413ba507a5c19 files=106 +-rw-r--r-- 1 root root 3219 2026-10-07 12:11:32.040268492 +0200 /var/cache/apparmor/f6da0254.0//.features +-rw------- 1 root root 295529 2026-07-18 18:15:53.043202621 +0200 /var/cache/apparmor/f6da0254.0//usr.sbin.chronyd +## kernel features now +yes yes yes yes yes +611413ba507a5c19 +## newest source file the profile depends on +2025-07-21 12:21:49.000000000 /etc/apparmor.d/abi/kernel-5.4-vanilla +2025-07-21 12:21:49.000000000 /etc/apparmor.d/abstractions/base +2025-07-21 12:21:49.000000000 /etc/apparmor.d/abstractions/nameservice +2025-07-21 12:21:49.000000000 /etc/apparmor.d/tunables/global +2026-05-21 13:32:03.000000000 /etc/apparmor.d/local/usr.sbin.chronyd +2026-07-01 22:28:16.000000000 /etc/apparmor.d/usr.sbin.chronyd +## files in /etc/apparmor.d changed 2026-07-18 or 2026-08-21 +/etc/apparmor.d/tunables +/etc/apparmor.d/tunables/home.d +/etc/apparmor.d/tunables/home.d/ubuntu +/etc/apparmor.d/tunables/multiarch.d +/etc/apparmor.d/tunables/xdg-user-dirs.d +/etc/apparmor.d/tunables/xdg-user-dirs.d/site.local +/etc/apparmor.d/lxc +/etc/apparmor.d/local +/etc/apparmor.d/local/lsb_release +/etc/apparmor.d/local/usr.bin.swtpm +## versions +apparmor 4.1.1-pmx1 +chrony 4.8-4~bpo13+2 +AppArmor parser version 4.1.1 +## dpkg log +2026-07-18 16:14:58 install apparmor:amd64 4.1.1-pmx1 +2026-07-18 16:15:24 install proxmox-kernel-7.0.2-6-pve-signed:amd64 7.0.2-6 +2026-10-07 12:09:03 install proxmox-kernel-7.0.14-20-pve-signed:amd64 7.0.14-20 +2026-10-07 14:15:40 upgrade chrony:amd64 4.6.1-3+deb13u1 4.8-4~bpo13+2 +2026-10-09 04:44:23 install proxmox-kernel-7.0.14-22-pve-signed:amd64 7.0.14-22 +2026-10-10 04:44:06 install proxmox-kernel-7.0.14-23-pve-signed:amd64 7.0.14-23 +2026-10-16 05:06:43 install proxmox-kernel-7.0.14-24-pve-signed:amd64 7.0.14-24 +2026-10-21 05:21:10 install proxmox-kernel-7.0.14-25-pve-signed:amd64 7.0.14-25 +2026-10-29 04:45:31 install proxmox-kernel-7.0.14-26-pve-signed:amd64 7.0.14-26 +2026-11-08 05:32:59 install proxmox-kernel-7.0.14-28-pve-signed:amd64 7.0.14-28 +## kernels booted +IDX BOOT ID FIRST ENTRY LAST ENTRY + -5 46081f567f384ab88591d4e604555a48 Sun 2026-10-04 21:43:33 CEST Sun 2026-10-04 21:43:46 CEST + -4 e82176ae7cf644518507468fc609eaa9 Wed 2026-10-14 15:43:07 CEST Fri 2026-10-16 05:09:06 CEST + -3 ddd3d7305e244b2a9eafb5cf229476f1 Fri 2026-10-16 05:09:23 CEST Wed 2026-10-21 05:23:34 CEST + -2 56db71938b304a57bdedf30f89860e1f Wed 2026-10-21 05:23:48 CEST Thu 2026-10-29 04:47:58 CET + -1 4f6d8b388e224b8ca8ed45e06a2f96b9 Thu 2026-10-29 04:48:11 CET Sun 2026-11-08 05:35:13 CET + 0 72899d2625244e069e8e8099775d1e51 Sun 2026-11-08 05:35:21 CET Wed 2026-11-11 14:15:31 CET +docker0: port 1(vethbb70052) entered disabled state +Linux version 7.0.14-24-pve (build@proxmox) (gcc (Debian 14. +Linux version 7.0.14-25-pve (build@proxmox) (gcc (Debian 14. +Linux version 7.0.14-26-pve (build@proxmox) (gcc (Debian 14. +Linux version 7.0.14-28-pve (build@proxmox) (gcc (Debian 14. +## chronyd confinement now +no chronyd process +## what installed chrony 4.8 +Start-Date: 2026-10-07 14:15:37 +Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --only-upgrade --no-install-recommends frr=10.6.1-1+pve3 librgw2=19.2.6-pve4 libradosstriper1=19.2.6-pve4 ceph-common=19.2.6-pve4 librbd1=19.2.6-pve4 libra +Requested-By: felhom-agent (999) +Upgrade: librados2:amd64 (19.2.3-pve4, 19.2.6-pve4), pve-docs:amd64 (9.2.1, 9.2.13), ceph-fuse:amd64 (19.2.3-pve4, 19.2.6-pve4), proxmox-widget-toolkit:amd64 (5.2.2, 5.2.10), corosync:amd64 (3.1.10-pve2, 3.1.10-pve3), frr:amd64 (10.6.1-1+pve2, 10.6.1-1+pve3), +Start-Date: 2026-10-08 04:37:54 diff --git a/documentation/audits/n100-clock-2026-11-11/evidence/n100-host-2.txt b/documentation/audits/n100-clock-2026-11-11/evidence/n100-host-2.txt new file mode 100644 index 00000000..df0fd6be --- /dev/null +++ b/documentation/audits/n100-clock-2026-11-11/evidence/n100-host-2.txt @@ -0,0 +1,236 @@ +#### demo-felhom 2026-11-11T13:08:39Z +net.unix.max_dgram_qlen = 512 +kernel.printk_ratelimit = 5 +## cmdline +BOOT_IMAGE=/boot/vmlinuz-7.0.14-28-pve root=/dev/mapper/pve-root ro quiet softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10 +## apparmor denied this boot: 36 +Nov 08 05:35:31 demo-felhom kernel: audit: type=1400 audit(1794112531.870:128): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=1529 comm="vgs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined" +Nov 08 05:35:31 demo-felhom kernel: audit: type=1400 audit(1794112531.898:129): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=1530 comm="lvs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined" +Nov 08 05:35:31 demo-felhom kernel: audit: type=1400 audit(1794112531.979:130): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=1536 comm="vgs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined" +Nov 08 05:35:32 demo-felhom kernel: audit: type=1400 audit(1794112532.008:131): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=1537 comm="lvs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined" +Nov 08 06:06:23 demo-felhom kernel: audit: type=1400 audit(1794114383.198:133): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=43146 comm="vgs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined" +## audit lines with chrony, all boots +Oct 16 05:09:25 demo-felhom kernel: audit: type=1400 audit(1792120165.400:121): apparmor="DENIED" operation="sendmsg" class="file" profile="/usr/sbin/chronyd" name="/run/systemd/notify" pid=692 comm="chronyd" requested_mask="w" denied_mask="w" fsuid=100 ouid=0 +## the second around the chrony start, this boot +start: 2026-11-08T05:35:23+01:00 +[ 3.980022] demo-felhom kernel: intel_pmc_core INT33A1:00: initialized +[ 3.988439] demo-felhom systemd-modules-load[310]: Inserted module 'zfs' +[ 3.989049] demo-felhom kernel: ZFS: Loaded module v2.4.4-pve1, ZFS pool version 5000, ZFS filesystem version 5 +[ 3.989820] demo-felhom systemd[1]: Finished systemd-modules-load.service - Load Kernel Modules. +[ 3.991237] demo-felhom systemd[1]: Starting systemd-sysctl.service - Apply Kernel Variables... +[ 4.004023] demo-felhom kernel: usbcore: registered new interface driver btusb +[ 4.011027] demo-felhom kernel: Bluetooth: hci0: RTL: examining hci_ver=0b hci_rev=000b lmp_ver=0b lmp_subver=8852 +[ 4.011156] demo-felhom kernel: Bluetooth: hci0: RTL: rom_version status=0 version=1 +[ 4.011353] demo-felhom systemd[1]: Starting systemd-rfkill.service - Load/Save RF Kill Switch Status... +[ 4.013369] demo-felhom kernel: Bluetooth: hci0: RTL: btrtl_initialize: key id 0 +[ 4.013474] demo-felhom kernel: Bluetooth: hci0: RTL: loading rtl_bt/rtl8852bu_fw.bin +[ 4.018168] demo-felhom systemd[1]: Reached target bluetooth.target - Bluetooth Support. +[ 4.058030] demo-felhom kernel: Bluetooth: hci0: RTL: loading rtl_bt/rtl8852bu_config.bin +[ 4.060122] demo-felhom kernel: Bluetooth: hci0: RTL: cfg_sz 6, total sz 62539 +[ 4.069224] demo-felhom systemd[1]: Started systemd-rfkill.service - Load/Save RF Kill Switch Status. +[ 4.072229] demo-felhom systemd[1]: Finished systemd-sysctl.service - Apply Kernel Variables. +[ 4.094018] demo-felhom kernel: usb 1-6: Warning! Unlikely big volume range (=511), cval->res is probably wrong. +[ 4.094444] demo-felhom kernel: usb 1-6: [5] FU [Mic Capture Volume] ch = 1, val = -8192/-1/16 +[ 4.100058] demo-felhom kernel: usb 1-6: Warning! Unlikely big volume range (=896), cval->res is probably wrong. +[ 4.100362] demo-felhom kernel: usb 1-6: [6] FU [PCM Playback Volume] ch = 2, val = -16384/-2048/16 +[ 4.100548] demo-felhom kernel: usbcore: registered new interface driver snd-usb-audio +[ 4.106727] demo-felhom systemd[1]: Reached target sound.target - Sound Card. +[ 4.116015] demo-felhom kernel: rtw89_8852be 0000:02:00.0: loaded firmware rtw89/rtw8852b_fw-1.bin +[ 4.116308] demo-felhom kernel: rtw89_8852be 0000:02:00.0: enabling device (0000 -> 0003) +[ 4.124022] demo-felhom kernel: rtw89_8852be 0000:02:00.0: Firmware version 0.29.29.15 (6fb3ec41), cmd version 0, type 5 +[ 4.124318] demo-felhom kernel: rtw89_8852be 0000:02:00.0: Firmware version 0.29.29.15 (6fb3ec41), cmd version 0, type 3 +[ 4.137907] demo-felhom systemd[1]: Mounting boot-efi.mount - /boot/efi... +[ 4.139530] demo-felhom systemd[1]: mnt-hdd_1.mount: Directory /mnt/hdd_1 to mount over is not empty, mounting anyway. +[ 4.141158] demo-felhom systemd[1]: Mounting mnt-hdd_1.mount - Felhom storage mount 47a3361a-91e0-4831-a69d-27f540ed3f48... +[ 4.142916] demo-felhom systemd[1]: Mounting tmp.mount - Temporary Directory /tmp... +[ 4.267728] demo-felhom systemd[1]: Mounted tmp.mount - Temporary Directory /tmp. +[ 4.268510] demo-felhom systemd[1]: Mounted boot-efi.mount - /boot/efi. +[ 4.421015] demo-felhom kernel: Bluetooth: hci0: RTL: fw version 0x127cfd78 +[ 4.475042] demo-felhom kernel: rtw89_8852be 0000:02:00.0: chip info CID: 0, CV: 1, AID: 0, ACV: 1, RFE: 5 +[ 4.476021] demo-felhom kernel: rtw89_8852be 0000:02:00.0: rfkill hardware state changed to enable +[ 4.494046] demo-felhom kernel: intel_tcc_cooling: TCC Offset locked +[ 4.499095] demo-felhom kernel: rtw89_8852be 0000:02:00.0 wlp2s0: renamed from wlan0 +[ 4.499354] demo-felhom kernel: i915 0000:00:02.0: [drm] Found alderlake_p/alderlake_n (device ID 46d1) integrated display version 13.00 stepping D0 +[ 4.499561] demo-felhom kernel: i915 0000:00:02.0: [drm] VT-d active for gfx access +[ 4.499846] demo-felhom kernel: i915 0000:00:02.0: vgaarb: deactivate vga console +[ 4.500020] demo-felhom kernel: i915 0000:00:02.0: [drm] Using Transparent Hugepages +[ 4.501210] demo-felhom kernel: i915 0000:00:02.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=io+mem +[ 4.505030] demo-felhom kernel: i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adlp_dmc.bin (v2.20) +[ 4.505306] demo-felhom kernel: Bluetooth: hci0: AOSP extensions version v1.00 +[ 4.515016] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: GuC firmware i915/tgl_guc_70.bin version 70.49.4 +[ 4.515287] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: HuC firmware i915/tgl_huc.bin version 7.9.3 +[ 4.517022] demo-felhom kernel: intel_rapl_msr: PL4 support detected. +[ 4.517113] demo-felhom kernel: intel_rapl_common: Found RAPL domain package +[ 4.517134] demo-felhom kernel: intel_rapl_common: Found RAPL domain core +[ 4.517152] demo-felhom kernel: intel_rapl_common: Found RAPL domain uncore +[ 4.520023] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: HuC: authenticated for all workloads +[ 4.520252] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: GUC: submission enabled +[ 4.520417] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: GUC: SLPC enabled +[ 4.520574] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: GUC: RC enabled +[ 4.522017] demo-felhom kernel: i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized +[ 4.522233] demo-felhom kernel: i915 0000:00:02.0: [drm] Registered 3 planes with drm panic +[ 4.530018] demo-felhom kernel: [drm] Initialized i915 1.6.0 for 0000:00:02.0 on minor 0 +[ 4.531012] demo-felhom kernel: ACPI: video: Video Device [GFX0] (multi-head: yes rom: no post: no) +[ 4.531087] demo-felhom kernel: input: Video Bus as /devices/pci0000:00/acpi.video_bus.0/input/input4 +[ 4.536037] demo-felhom kernel: EXT4-fs (sdb): mounted filesystem 47a3361a-91e0-4831-a69d-27f540ed3f48 r/w with ordered data mode. Quota mode: none. +[ 4.537227] demo-felhom systemd[1]: Mounted mnt-hdd_1.mount - Felhom storage mount 47a3361a-91e0-4831-a69d-27f540ed3f48. +[ 4.545015] demo-felhom kernel: i915 0000:00:02.0: [drm] Cannot find any crtc or sizes +[ 4.569019] demo-felhom kernel: i915 0000:00:02.0: [drm] Cannot find any crtc or sizes +[ 4.581021] demo-felhom kernel: i915 0000:00:02.0: [drm] Cannot find any crtc or sizes +[ 4.610025] demo-felhom kernel: snd_hda_intel 0000:00:1f.3: enabling device (0000 -> 0002) +[ 4.610349] demo-felhom kernel: snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops intel_audio_component_bind_ops [i915]) +[ 4.917022] demo-felhom kernel: input: HDA Intel PCH HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1f.3/sound/card1/input5 +[ 4.917152] demo-felhom kernel: input: HDA Intel PCH HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:1f.3/sound/card1/input6 +[ 4.917202] demo-felhom kernel: input: HDA Intel PCH HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:1f.3/sound/card1/input7 +[ 4.917241] demo-felhom kernel: input: HDA Intel PCH HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:1f.3/sound/card1/input8 +[ 4.940127] demo-felhom systemd[1]: Finished ifupdown2-pre.service - Helper to synchronize boot up for ifupdown. +[ 4.943663] demo-felhom systemd[1]: Finished systemd-udev-settle.service - Wait for udev To Complete Device Initialization. +[ 4.943871] demo-felhom systemd[1]: zfs-import-cache.service - Import ZFS pools by cache file skipped, unmet condition check ConditionFileNotEmpty=/etc/zfs/zpool.cache +[ 4.943900] demo-felhom systemd[1]: Reached target zfs-import.target - ZFS pool import target. +[ 4.945492] demo-felhom systemd[1]: Starting zfs-mount.service - Mount ZFS filesystems... +[ 4.946429] demo-felhom systemd[1]: Starting zfs-volume-wait.service - Wait for ZFS Volume (zvol) links in /dev... +[ 4.966316] demo-felhom systemd[1]: Finished zfs-mount.service - Mount ZFS filesystems. +[ 4.967027] demo-felhom systemd[1]: Finished zfs-volume-wait.service - Wait for ZFS Volume (zvol) links in /dev. +[ 4.968549] demo-felhom zvol_wait[531]: No zvols found, nothing to do. +[ 4.967458] demo-felhom systemd[1]: Reached target local-fs.target - Local File Systems. +[ 4.967518] demo-felhom systemd[1]: Reached target zfs-volumes.target - ZFS volumes are ready. +[ 4.969218] demo-felhom systemd[1]: Listening on systemd-sysext.socket - System Extension Image Management. +[ 4.970224] demo-felhom systemd[1]: Starting apparmor.service - Load AppArmor profiles... +[ 4.971086] demo-felhom systemd[1]: Starting console-setup.service - Set console font and keymap... +[ 4.972029] demo-felhom systemd[1]: Starting felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit)... +[ 4.973401] demo-felhom systemd[1]: Starting networking.service - Network initialization... +[ 4.977223] demo-felhom systemd[1]: Starting pvebanner.service - Proxmox VE Login Banner... +[ 4.978444] demo-felhom systemd[1]: Starting pvefw-logger.service - Proxmox VE firewall logger... +[ 4.979795] demo-felhom systemd[1]: Starting pvenetcommit.service - Commit Proxmox VE network changes... +[ 4.983183] demo-felhom systemd[1]: Starting systemd-binfmt.service - Set Up Additional Binary Formats... +[ 4.985507] demo-felhom systemd[1]: Starting systemd-tmpfiles-setup.service - Create System Files and Directories... +[ 4.986954] demo-felhom systemd[1]: Finished console-setup.service - Set console font and keymap. +[ 4.998578] demo-felhom systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 547 (systemd-binfmt) +[ 4.999914] demo-felhom systemd[1]: Mounting proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System... +[ 5.000113] demo-felhom systemd-tmpfiles[550]: /etc/tmpfiles.d/felhom-privsep.conf:15: Duplicate line for path "/run/sshd", ignoring. +[ 5.000258] demo-felhom systemd-tmpfiles[550]: /usr/lib/tmpfiles.d/legacy.conf:14: Duplicate line for path "/run/lock", ignoring. +[ 5.001716] demo-felhom systemd[1]: Finished pvenetcommit.service - Commit Proxmox VE network changes. +[ 5.003282] demo-felhom apparmor.systemd[538]: Restarting AppArmor +[ 5.003282] demo-felhom apparmor.systemd[538]: Reloading AppArmor profiles +[ 5.021348] demo-felhom networking[541]: networking: Configuring network interfaces +[ 5.022383] demo-felhom systemd[1]: Mounted proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System. +[ 5.025891] demo-felhom systemd[1]: Finished systemd-binfmt.service - Set Up Additional Binary Formats. +[ 5.048314] demo-felhom pvefw-logger[573]: starting pvefw logger +[ 5.048548] demo-felhom systemd[1]: Started pvefw-logger.service - Proxmox VE firewall logger. +[ 5.049951] demo-felhom systemd[1]: Finished systemd-tmpfiles-setup.service - Create System Files and Directories. +[ 5.050310] demo-felhom systemd[1]: ldconfig.service - Rebuild Dynamic Linker Cache skipped, no trigger condition checks were met. +[ 5.055441] demo-felhom systemd[1]: Starting rpcbind.service - RPC bind portmap service... +[ 5.055478] demo-felhom systemd[1]: systemd-firstboot.service - First Boot Wizard skipped, unmet condition check ConditionFirstBoot=yes +[ 5.055507] demo-felhom systemd[1]: first-boot-complete.target - First Boot Complete skipped, unmet condition check ConditionFirstBoot=yes +[ 5.055569] demo-felhom systemd[1]: systemd-journal-catalog-update.service - Rebuild Journal Catalog skipped, unmet condition check ConditionNeedsUpdate=/var +[ 5.055622] demo-felhom systemd[1]: systemd-machine-id-commit.service - Save Transient machine-id to Disk skipped, unmet condition check ConditionPathIsMountPoint=/etc/machine-id +[ 5.055680] demo-felhom systemd[1]: systemd-update-done.service - Update is Completed skipped, no trigger condition checks were met. +[ 5.069037] demo-felhom kernel: audit: type=1400 audit(1794112523.118:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name=4D6F6E676F444220436F6D70617373 pid=572 comm="apparmor_ +[ 5.069130] demo-felhom kernel: audit: type=1400 audit(1794112523.118:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="QtWebEngineProcess" pid=574 comm="apparmor_parser" +[ 5.069157] demo-felhom kernel: audit: type=1400 audit(1794112523.118:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="1password" pid=570 comm="apparmor_parser" +[ 5.069183] demo-felhom kernel: audit: type=1400 audit(1794112523.119:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="Discord" pid=571 comm="apparmor_parser" +[ 5.072026] demo-felhom kernel: audit: type=1400 audit(1794112523.123:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="brave" pid=581 comm="apparmor_parser" +[ 5.073026] demo-felhom kernel: audit: type=1400 audit(1794112523.124:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="busybox" pid=583 comm="apparmor_parser" +[ 5.074325] demo-felhom kernel: audit: type=1400 audit(1794112523.125:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="balena-etcher" pid=580 comm="apparmor_parser" +[ 5.074387] demo-felhom kernel: audit: type=1400 audit(1794112523.125:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="buildah" pid=582 comm="apparmor_parser" +[ 5.074411] demo-felhom kernel: audit: type=1400 audit(1794112523.125:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="cam" pid=584 comm="apparmor_parser" +[ 5.075909] demo-felhom systemd[1]: Started rpcbind.service - RPC bind portmap service. +[ 5.076068] demo-felhom systemd[1]: Reached target rpcbind.target - RPC Port Mapper. +[ 5.119035] demo-felhom kernel: kauditd_printk_skb: 40 callbacks suppressed +[ 5.119141] demo-felhom kernel: audit: type=1400 audit(1794112523.170:51): apparmor="STATUS" operation="profile_load" profile="unconfined" name="opera" pid=624 comm="apparmor_parser" +[ 5.120172] demo-felhom kernel: audit: type=1400 audit(1794112523.171:52): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default" pid=610 comm="apparmor_parser +[ 5.120257] demo-felhom kernel: audit: type=1400 audit(1794112523.171:53): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-cgns" pid=610 comm="apparmor_p +[ 5.120282] demo-felhom kernel: audit: type=1400 audit(1794112523.171:54): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-mounting" pid=610 comm="a +[ 5.120308] demo-felhom kernel: audit: type=1400 audit(1794112523.171:55): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-nesting" pid=610 comm="ap +[ 5.120329] demo-felhom kernel: audit: type=1400 audit(1794112523.172:56): apparmor="STATUS" operation="profile_load" profile="unconfined" name="polypane" pid=627 comm="apparmor_parser" +[ 5.121021] demo-felhom kernel: audit: type=1400 audit(1794112523.172:57): apparmor="STATUS" operation="profile_load" profile="unconfined" name="pageedit" pid=625 comm="apparmor_parser" +[ 5.124025] demo-felhom kernel: audit: type=1400 audit(1794112523.175:58): apparmor="STATUS" operation="profile_load" profile="unconfined" name="privacybrowser" pid=628 comm="apparmor_parser" +[ 5.124103] demo-felhom kernel: audit: type=1400 audit(1794112523.176:59): apparmor="STATUS" operation="profile_load" profile="unconfined" name="qcam" pid=630 comm="apparmor_parser" +[ 5.126023] demo-felhom kernel: audit: type=1400 audit(1794112523.177:60): apparmor="STATUS" operation="profile_load" profile="unconfined" name="pve-container-mounthotplug" pid=629 comm="apparmor_p +[ 5.133470] demo-felhom felhom-crash-guard[540]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10 +[ 5.138735] demo-felhom systemd[1]: Mounting run-rpc_pipefs.mount - RPC Pipe File System... +[ 5.153807] demo-felhom felhom-crash-guard[665]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10 +[ 5.162616] demo-felhom systemd[1]: Finished felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit). +[ 5.199068] demo-felhom kernel: RPC: Registered named UNIX socket transport module. +[ 5.199201] demo-felhom kernel: RPC: Registered udp transport module. +[ 5.199231] demo-felhom kernel: RPC: Registered tcp transport module. +[ 5.199252] demo-felhom kernel: RPC: Registered tcp-with-tls transport module. +[ 5.199278] demo-felhom kernel: RPC: Registered tcp NFSv4.1 backchannel transport module. +[ 5.201359] demo-felhom systemd[1]: Mounted run-rpc_pipefs.mount - RPC Pipe File System. +[ 5.201831] demo-felhom systemd[1]: Reached target rpc_pipefs.target. +[ 5.204008] demo-felhom systemd[1]: Starting nfs-blkmap.service - pNFS block layout mapping daemon... +[ 5.204272] demo-felhom systemd[1]: rpc-gssd.service - RPC security service for NFS client and server skipped, unmet condition check ConditionPathExists=/etc/krb5.keytab +[ 5.204309] demo-felhom systemd[1]: Reached target nfs-client.target - NFS client services. +[ 5.207340] demo-felhom systemd[1]: Finished apparmor.service - Load AppArmor profiles. +[ 5.207747] demo-felhom systemd[1]: Reached target sysinit.target - System Initialization. +## chrony lines per boot (all boots) +-- boot -5 +-- boot -4 +-- boot -3 +Oct 16 05:09:25 demo-felhom systemd[1]: Starting chrony.service - chrony, an NTP client/server... +Oct 16 05:09:25 demo-felhom chronyd[692]: Fatal error : Could not send notification to $NOTIFY_SOCKET +Oct 16 05:09:25 demo-felhom chronyd[692]: Fatal error : Could not send notification to $NOTIFY_SOCKET +Oct 16 05:09:25 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE +Oct 16 05:09:25 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'. +Oct 16 05:09:25 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server. +-- boot -2 +Oct 21 05:23:50 demo-felhom systemd[1]: Starting chrony.service - chrony, an NTP client/server... +Oct 21 05:23:50 demo-felhom chronyd[676]: Fatal error : Could not send notification to $NOTIFY_SOCKET +Oct 21 05:23:50 demo-felhom chronyd[676]: Fatal error : Could not send notification to $NOTIFY_SOCKET +Oct 21 05:23:50 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE +Oct 21 05:23:50 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'. +Oct 21 05:23:50 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server. +-- boot -1 +Oct 29 04:48:13 demo-felhom systemd[1]: Starting chrony.service - chrony, an NTP client/server... +Oct 29 04:48:14 demo-felhom chronyd[682]: Fatal error : Could not send notification to $NOTIFY_SOCKET +Oct 29 04:48:14 demo-felhom chronyd[682]: Fatal error : Could not send notification to $NOTIFY_SOCKET +Oct 29 04:48:14 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE +Oct 29 04:48:14 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'. +Oct 29 04:48:14 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server. +-- boot 0 +Nov 08 05:35:23 demo-felhom systemd[1]: Starting chrony.service - chrony, an NTP client/server... +Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET +Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET +Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE +Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'. +Nov 08 05:35:23 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server. +## sandbox env of pid1 +total 0 +drwxr-xr-x 21 root root 560 Nov 11 14:06 . +drwxr-xr-x 36 root root 1520 Nov 11 14:08 .. +drwxr-xr-x 2 root root 40 Nov 8 05:35 ask-password +drwx------ 2 root root 60 Nov 8 21:45 ask-password-block +drwxr-xr-x 6 root root 260 Nov 10 15:50 generator +dr-xr-xr-x 3 root root 160 Nov 8 05:35 inaccessible +drwxr-xr-x 2 root root 40 Nov 8 05:35 incoming +drwxr-xr-x 2 root root 40 Nov 9 05:33 inhibit +srw-rw-rw- 1 root root 0 Nov 8 05:35 io.systemd.Credentials +srw-rw-rw- 1 root root 0 Nov 8 05:35 io.systemd.Hostname +srw-rw-rw- 1 root root 0 Nov 8 05:35 io.systemd.ManagedOOM +srw------- 1 root root 0 Nov 8 05:35 io.systemd.sysext +drwxr-xr-x 3 root root 200 Nov 8 05:35 journal +drwxr-xr-x 2 root root 40 Nov 8 05:35 machines +dr-xr-xr-x 2 root root 40 Nov 8 05:35 mount-rootfs +drwxr-xr-x 4 systemd-network systemd-network 80 Nov 8 05:35 netif +srwxrwxrwx 1 root root 0 Nov 8 05:35 notify +srwx------ 1 root root 0 Nov 8 05:35 private +drwxr-xr-x 6 root root 120 Nov 11 14:07 propagate +drwxr-xr-x 2 root root 60 Nov 8 05:35 seats +drwxr-xr-x 2 root root 120 Nov 11 14:08 sessions +drwxr-xr-x 2 root root 40 Nov 8 05:35 shutdown +drwxr-xr-x 2 root root 40 Nov 8 05:35 system +-r--r--r-- 1 root root 0 Nov 10 15:50 systemd-units-load +drwxr-xr-x 2 root root 60 Nov 11 14:08 transient +drwxr-xr-x 2 root root 1880 Nov 11 14:09 units +drwxr-xr-x 2 root root 60 Nov 8 05:35 userdb +drwxr-xr-x 2 root root 60 Nov 11 14:08 users +## tailscale +active +## rtc +ls: cannot access '/dev/ptp*': No such file or directory +ls: cannot access '/dev/pps*': No such file or directory +/dev/rtc +/dev/rtc0 +## cpu +4 +model name : Intel(R) N100 diff --git a/documentation/audits/n100-clock-2026-11-11/evidence/n100-host.txt b/documentation/audits/n100-clock-2026-11-11/evidence/n100-host.txt new file mode 100644 index 00000000..68c67ae1 --- /dev/null +++ b/documentation/audits/n100-clock-2026-11-11/evidence/n100-host.txt @@ -0,0 +1,250 @@ +#### host demo-felhom 2026-11-11T13:06:53Z local 14:06:53+0100 +## versions +pve-manager/9.2.26/41a0e2e09b9e6f9d (running kernel: 7.0.14-28-pve) +ii chrony 4.8-4~bpo13+2 +ii systemd 257.13-1~deb13u1 +un systemd-timesyncd +7.0.14-28-pve +## unit state +Type=notify +PIDFile=/run/chrony/chronyd.pid +NotifyAccess=main +Result=exit-code +NRestarts=0 +ExecMainCode=1 +ExecMainStatus=1 +ActiveState=failed +SubState=failed +FragmentPath=/usr/lib/systemd/system/chrony.service +DropInPaths= +UnitFileState=enabled +ActiveEnterTimestamp= +InactiveEnterTimestamp=Sun 2026-11-08 05:35:23 CET +## unit cat +# /usr/lib/systemd/system/chrony.service +[Unit] +Description=chrony, an NTP client/server +Documentation=man:chronyd(8) man:chronyc(1) man:chrony.conf(5) +Conflicts=openntpd.service ntp.service ntpsec.service +ConditionVirtualization=|!container +ConditionVirtualization=|wsl +ConditionCapability=CAP_SYS_TIME + +[Service] +Type=notify +PIDFile=/run/chrony/chronyd.pid +EnvironmentFile=-/etc/default/chrony +User=_chrony +# Daemon is started as root, but still sandboxed +ExecStart=!/usr/sbin/chronyd -n $DAEMON_OPTS + +CapabilityBoundingSet=~CAP_AUDIT_CONTROL CAP_AUDIT_READ CAP_AUDIT_WRITE +CapabilityBoundingSet=~CAP_BLOCK_SUSPEND CAP_KILL CAP_LEASE CAP_LINUX_IMMUTABLE +CapabilityBoundingSet=~CAP_MAC_ADMIN CAP_MAC_OVERRIDE CAP_MKNOD CAP_SYS_ADMIN +CapabilityBoundingSet=~CAP_SYS_BOOT CAP_SYS_CHROOT CAP_SYS_MODULE CAP_SYS_PACCT +CapabilityBoundingSet=~CAP_SYS_PTRACE CAP_SYS_RAWIO CAP_SYS_TTY_CONFIG CAP_WAKE_ALARM +DeviceAllow=char-pps rw +DeviceAllow=char-ptp rw +DeviceAllow=char-rtc rw +DevicePolicy=closed +LockPersonality=yes +MemoryDenyWriteExecute=yes +NoNewPrivileges=yes +PrivateTmp=yes +ProcSubset=pid +ProtectControlGroups=yes +ProtectHome=yes +ProtectHostname=yes +ProtectKernelLogs=yes +ProtectKernelModules=yes +ProtectKernelTunables=yes +ProtectProc=invisible +ProtectSystem=strict +# Used for gps refclocks +ReadWritePaths=/run +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX +RestrictNamespaces=yes +RestrictSUIDSGID=yes +SystemCallArchitectures=native +SystemCallFilter=~@cpu-emulation @debug @module @mount @obsolete @raw-io @reboot @swap + +ConfigurationDirectory=chrony +RuntimeDirectory=chrony +RuntimeDirectoryMode=0700 +# See dumpdir in chrony.conf(5) +RuntimeDirectoryPreserve=restart +StateDirectory=chrony +StateDirectoryMode=0750 +LogsDirectory=chrony +LogsDirectoryMode=0750 + +# Adjust restrictions for /usr/sbin/sendmail (mailonchange directive) +NoNewPrivileges=no +ReadWritePaths=-/var/spool +RestrictAddressFamilies=AF_NETLINK + +[Install] +Alias=chronyd.service +WantedBy=multi-user.target +## files +ls: cannot access '/etc/systemd/system/chrony.service.d': No such file or directory +ls: cannot access '/etc/systemd/system/chronyd.service.d': No such file or directory +ls: cannot access '/etc/systemd/system/chrony.service': No such file or directory +lrwxrwxrwx 1 root root 38 May 21 13:32 /etc/systemd/system/chronyd.service -> /usr/lib/systemd/system/chrony.service +-rw-r--r-- 1 root root 1929 Jul 1 22:28 /lib/systemd/system/chrony.service +-rw-r--r-- 1 root root 1929 Jul 1 22:28 /usr/lib/systemd/system/chrony.service +## dpkg verify chrony +rc=0 +## conf +/etc/chrony: +total 24 +drwxr-xr-x 4 root root 4096 May 21 13:32 . +drwxr-xr-x 102 root root 4096 Nov 8 05:32 .. +-rw-r--r-- 1 root root 1450 Feb 16 2026 chrony.conf +-rw-r----- 1 root _chrony 481 Feb 16 2026 chrony.keys +drwxr-xr-x 2 root root 4096 Oct 7 14:15 conf.d +drwxr-xr-x 2 root root 4096 Oct 7 14:15 sources.d + +/etc/chrony/conf.d: +total 12 +drwxr-xr-x 2 root root 4096 Oct 7 14:15 . +drwxr-xr-x 4 root root 4096 May 21 13:32 .. +-rw-r--r-- 1 root root 422 Feb 16 2026 README + +/etc/chrony/sources.d: +total 12 +drwxr-xr-x 2 root root 4096 Oct 7 14:15 . +drwxr-xr-x 4 root root 4096 May 21 13:32 .. +-rw-r--r-- 1 root root 498 Feb 16 2026 README +pool 2.debian.pool.ntp.org iburst +sourcedir /run/chrony-dhcp +sourcedir /etc/chrony/sources.d +keyfile /etc/chrony/chrony.keys +driftfile /var/lib/chrony/chrony.drift +ntsdumpdir /var/lib/chrony +logdir /var/log/chrony +maxupdateskew 100.0 +rtcsync +makestep 1 3 +leapseclist /usr/share/zoneinfo/leap-seconds.list +confdir /etc/chrony/conf.d +-- /etc/chrony/conf.d/README +Files found under the /etc/chrony/conf.d directory with the .conf suffix are +parsed in the lexicographical order of the file names when chronyd starts up. +This enables a fragmented configuration of chronyd. +Although those files can contain any directives listed in chrony.conf(5), +it would be wiser to add NTP sources in the /etc/chrony/sources.d +directory. Please read /etc/chrony/sources.d/README for more information. +-- /etc/chrony/sources.d/README +Only NTP sources can be specified in the /etc/chrony/sources.d directory. +Files in this directory must end with the ".sources" suffix. They can only +contain the "peer", "pool" and "server" directives and must have all +lines terminated by a trailing newline. +There is no need to restart chronyd for these time sources to be usable, +running 'chronyc reload sources' is sufficient. +Example: +## default + +DAEMON_OPTS="-F 1" +## apparmor + /usr/sbin/chronyd +-rw-r--r-- 1 root root 0 May 21 13:32 /etc/apparmor.d/local/usr.sbin.chronyd +-rw-r--r-- 1 root root 2837 Jul 1 22:28 /etc/apparmor.d/usr.sbin.chronyd + +/etc/apparmor.d/disable: +total 8 +drwxr-xr-x 2 root root 4096 Jul 21 2025 . +drwxr-xr-x 9 root root 4096 Nov 8 05:32 .. +## apparmor profile notify lines +81: # Allow systemd Type=notify using sd_notify's $NOTIFY_SOCKET +82: @{run}/systemd/notify w, +## kernel audit lines for chronyd (last 20) +## journal chrony this boot +Nov 08 05:35:23 demo-felhom systemd[1]: Starting chrony.service - chrony, an NTP client/server... +Nov 08 05:35:23 demo-felhom chronyd[690]: chronyd version 4.8 starting (+CMDMON +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +NTS +SECHASH +IPV6 -DEBUG) +Nov 08 05:35:23 demo-felhom chronyd[690]: Loaded 0 symmetric keys +Nov 08 05:35:23 demo-felhom chronyd[690]: Using leap second list /usr/share/zoneinfo/leap-seconds.list +Nov 08 05:35:23 demo-felhom chronyd[690]: Frequency -2.666 +/- 0.016 ppm read from /var/lib/chrony/chrony.drift +Nov 08 05:35:23 demo-felhom chronyd[690]: Loaded seccomp filter (level 1) +Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET +Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET +Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE +Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'. +Nov 08 05:35:23 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server. +## first failure ever in the journal +Oct 16 05:09:25 demo-felhom chronyd[692]: Fatal error : Could not send notification to $NOTIFY_SOCKET +Oct 16 05:09:25 demo-felhom chronyd[692]: Fatal error : Could not send notification to $NOTIFY_SOCKET +Oct 21 05:23:50 demo-felhom chronyd[676]: Fatal error : Could not send notification to $NOTIFY_SOCKET +count: 8 +## oldest journal line +Oct 04 21:43:33 demo-felhom systemd[3282788]: Queued start job for default targe +## last good start in the journal +## boots + -5 46081f567f384ab88591d4e604555a48 Sun 2026-10-04 21:43:33 CEST Sun 2026-10-04 21:43:46 CEST + -4 e82176ae7cf644518507468fc609eaa9 Wed 2026-10-14 15:43:07 CEST Fri 2026-10-16 05:09:06 CEST + -3 ddd3d7305e244b2a9eafb5cf229476f1 Fri 2026-10-16 05:09:23 CEST Wed 2026-10-21 05:23:34 CEST + -2 56db71938b304a57bdedf30f89860e1f Wed 2026-10-21 05:23:48 CEST Thu 2026-10-29 04:47:58 CET + -1 4f6d8b388e224b8ca8ed45e06a2f96b9 Thu 2026-10-29 04:48:11 CET Sun 2026-11-08 05:35:13 CET + 0 72899d2625244e069e8e8099775d1e51 Sun 2026-11-08 05:35:21 CET Wed 2026-11-11 14:06:53 CET +## what starts with it +chrony.service +● ├─-.mount +● ├─system.slice +● ├─systemd-journald.socket +● ├─systemd-remount-fs.service +● ├─systemd-tmpfiles-setup.service +● ├─tmp.mount +● ├─basic.target +● │ ├─-.mount +○ │ ├─systemd-pcrphase-sysinit.service +● │ ├─tmp.mount +● │ ├─paths.target +● │ │ ├─systemd-ask-password-console.path +● │ │ └─systemd-ask-password-wall.path +● │ ├─slices.target +● │ │ ├─-.slice +● │ │ ├─system-xfs_scrub.slice +● │ │ ├─system.slice +● │ │ └─user.slice +● │ ├─sockets.target +● │ │ ├─dbus.socket +● │ │ ├─iscsid.socket +● │ │ ├─netavark-dhcp-proxy.socket +● │ │ ├─rrdcached.socket +● │ │ ├─sshd-unix-local.socket +The time when unit became active or started is printed after the "@" character. +The time the unit took to start is printed after the "+" character. + +chrony.service +256ms +└─basic.target @2.481s + └─sockets.target @2.481s + └─systemd-hostnamed.socket @2.481s + └─sysinit.target @2.478s + └─apparmor.service @2.241s +237ms + └─local-fs.target @2.238s + └─mnt-felhom\x2ddrives-hdd_1.mount @11.537s + └─mnt-felhom\x2ddrives.mount @2.549s +## other time services +inactive inactive inactive inactive + Local time: Wed 2026-11-11 14:06:55 CET + Universal time: Wed 2026-11-11 13:06:55 UTC + RTC time: Wed 2026-11-11 13:06:52 + Time zone: Europe/Budapest (CET, +0100) +System clock synchronized: no + NTP service: inactive + RTC in local TZ: no +## chronyc +506 Cannot talk to daemon +506 Cannot talk to daemon +## dpkg history chrony/apparmor/systemd +2026-07-18 16:14:58 install apparmor:amd64 4.1.1-pmx1 +2026-07-18 16:15:52 configure apparmor:amd64 4.1.1-pmx1 +2026-10-07 14:15:40 upgrade chrony:amd64 4.6.1-3+deb13u1 4.8-4~bpo13+2 +2026-10-07 14:15:46 configure chrony:amd64 4.8-4~bpo13+2 +## felhom files that name chrony +## ps +## run dir +ls: cannot access '/run/chrony': No such file or directory +srwxrwxrwx 1 root root 0 Nov 8 05:35 /run/systemd/notify +## uptime +2026-11-08 05:35:18 diff --git a/documentation/audits/n100-clock-2026-11-11/evidence/n100-stale-proof.txt b/documentation/audits/n100-clock-2026-11-11/evidence/n100-stale-proof.txt new file mode 100644 index 00000000..7b0ff083 --- /dev/null +++ b/documentation/audits/n100-clock-2026-11-11/evidence/n100-stale-proof.txt @@ -0,0 +1,25 @@ +#### demo-felhom 2026-11-11T13:16:03Z +## fresh compile of the on-disk profile to stdout (no load, cache not read, not written) +83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1 - +## loaded +dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 + +## cache +83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1 /var/cache/apparmor/ae1a4050.0/usr.sbin.chronyd +dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 /var/cache/apparmor/f6da0254.0/usr.sbin.chronyd +drwx------ 2 root root 4096 2026-10-07 14:16:03.051992948 +0200 /var/cache/apparmor/ae1a4050.0/ +-rw-r--r-- 1 root root 3177 2026-07-18 18:17:02.251100923 +0200 /var/cache/apparmor/ae1a4050.0/.features +drwx------ 2 root root 4096 2026-11-08 05:32:24.065142783 +0100 /var/cache/apparmor/f6da0254.0/ +-rw-r--r-- 1 root root 3219 2026-10-07 12:11:32.040268492 +0200 /var/cache/apparmor/f6da0254.0/.features +## state +failed +chrony 4.8-4~bpo13+2 +7.0.14-28-pve +1794402963.132843793 +## chrony fatal lines +8 +## stale check, every cached profile: cache differs from a fresh compile +current dir: /var/cache/apparmor/f6da0254.0/ +STALE: lxc-containers +STALE: usr.sbin.chronyd +checked=106 stale=2 diff --git a/documentation/audits/n100-clock-2026-11-11/evidence/tester1-stale-proof.txt b/documentation/audits/n100-clock-2026-11-11/evidence/tester1-stale-proof.txt new file mode 100644 index 00000000..32a36455 --- /dev/null +++ b/documentation/audits/n100-clock-2026-11-11/evidence/tester1-stale-proof.txt @@ -0,0 +1,33 @@ +#### felhom 2026-11-11T13:17:03Z +## fresh compile of the on-disk profile to stdout (no load, cache not read, not written) +dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 - +## loaded +dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 + +## cache +dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 /var/cache/apparmor/ae1a4050.0/usr.sbin.chronyd +dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 /var/cache/apparmor/f6da0254.0/usr.sbin.chronyd +drwx------ 2 root root 4096 2026-10-04 21:37:58.088000000 +0200 /var/cache/apparmor/ae1a4050.0/ +-rw-r--r-- 1 root root 3177 2026-10-04 21:37:57.515000000 +0200 /var/cache/apparmor/ae1a4050.0/.features +drwx------ 2 root root 4096 2026-10-07 10:41:12.158575707 +0200 /var/cache/apparmor/f6da0254.0/ +-rw-r--r-- 1 root root 3219 2026-10-07 10:41:11.594575707 +0200 /var/cache/apparmor/f6da0254.0/.features +## state +active +chrony 4.6.1-3+deb13u1 +7.0.14-22-pve +System time : 0.000177019 seconds slow of NTP time +Leap status : Normal +1794403023.309780841 +## chrony fatal lines +0 +## stale check, every cached profile: cache differs from a fresh compile +current dir: /var/cache/apparmor/f6da0254.0/ +STALE: lxc-containers +checked=106 stale=1 +2026-10-04 19:31:38 install apparmor:amd64 4.1.1-pmx1 +2026-10-04 19:32:11 install proxmox-kernel-7.0.2-6-pve-signed:amd64 7.0.2-6 +2026-10-07 10:02:07 install proxmox-kernel-7.0.14-20-pve-signed:amd64 7.0.14-20 +2026-10-07 17:14:55 install proxmox-kernel-7.0.14-22-pve-signed:amd64 7.0.14-22 +NRestarts=0 +ExecMainStartTimestamp=Tue 2026-11-10 15:43:32 CET +ActiveState=active diff --git a/documentation/audits/n100-clock-2026-11-11/tools/chrony-read.sh b/documentation/audits/n100-clock-2026-11-11/tools/chrony-read.sh new file mode 100644 index 00000000..811dc249 --- /dev/null +++ b/documentation/audits/n100-clock-2026-11-11/tools/chrony-read.sh @@ -0,0 +1,26 @@ +# READ ONLY. Part 3: the time service on a Proxmox host. +echo "#### host $(hostname) $(date -u +%FT%TZ) local $(date +%T%z)" +echo "## versions"; pveversion; dpkg -l chrony systemd systemd-timesyncd 2>/dev/null | grep '^[a-z][a-z] ' | awk '{print $1,$2,$3}' +uname -r +echo "## unit state"; systemctl show chrony -p ActiveState -p SubState -p Result -p Type -p NotifyAccess -p ExecMainStatus -p ExecMainCode -p NRestarts -p ActiveEnterTimestamp -p InactiveEnterTimestamp -p FragmentPath -p DropInPaths -p UnitFileState -p PIDFile +echo "## unit cat"; systemctl cat chrony 2>&1 +echo "## files"; ls -la /etc/systemd/system/chrony.service.d /etc/systemd/system/chronyd.service.d /usr/lib/systemd/system/chrony.service /lib/systemd/system/chrony.service /etc/systemd/system/chrony.service /etc/systemd/system/chronyd.service 2>&1 +echo "## dpkg verify chrony"; dpkg -V chrony 2>&1; echo "rc=$?" +echo "## conf"; ls -la /etc/chrony /etc/chrony/conf.d /etc/chrony/sources.d 2>&1; grep -v '^\s*#' /etc/chrony/chrony.conf 2>&1 | grep -v '^$'; for f in /etc/chrony/conf.d/* /etc/chrony/sources.d/*; do [ -f "$f" ] && { echo "-- $f"; grep -v '^\s*#' "$f" | grep -v '^$'; }; done +echo "## default"; cat /etc/default/chrony 2>&1 | grep -v '^#' +echo "## apparmor"; aa-status 2>/dev/null | grep -i chrony; ls -la /etc/apparmor.d/usr.sbin.chronyd /etc/apparmor.d/local/usr.sbin.chronyd /etc/apparmor.d/disable 2>&1; cat /etc/apparmor.d/local/usr.sbin.chronyd 2>/dev/null | grep -v '^#' +echo "## apparmor profile notify lines"; grep -n 'notify\|/run/systemd' /etc/apparmor.d/usr.sbin.chronyd 2>&1 +echo "## kernel audit lines for chronyd (last 20)"; journalctl -k --no-pager 2>/dev/null | grep -i 'chrony' | tail -20 | cut -c1-330 +echo "## journal chrony this boot"; journalctl -u chrony -b --no-pager 2>&1 | tail -40 | cut -c1-300 +echo "## first failure ever in the journal"; journalctl -u chrony --no-pager 2>/dev/null | grep -m3 'NOTIFY_SOCKET' | cut -c1-200; echo "count: $(journalctl -u chrony --no-pager 2>/dev/null | grep -c 'NOTIFY_SOCKET')" +echo "## oldest journal line"; journalctl --no-pager 2>/dev/null | head -1 | cut -c1-80 +echo "## last good start in the journal"; journalctl -u chrony --no-pager 2>/dev/null | grep 'Started chrony\|Selected source' | tail -3 | cut -c1-200 +echo "## boots"; journalctl --list-boots --no-pager 2>/dev/null | tail -6 +echo "## what starts with it"; systemctl list-dependencies --after chrony 2>/dev/null | head -25; systemd-analyze critical-chain chrony.service 2>&1 | head -12 +echo "## other time services"; systemctl is-active systemd-timesyncd ntp ntpsec openntpd 2>&1 | tr '\n' ' '; echo; timedatectl 2>&1 +echo "## chronyc"; chronyc tracking 2>&1 | head -8; chronyc sources 2>&1 | tail -6 +echo "## dpkg history chrony/apparmor/systemd"; zgrep -h ' chrony\| apparmor:\| systemd:' /var/log/dpkg.log* 2>/dev/null | grep ' upgrade \| install \|configure ' | sort | tail -25 +echo "## felhom files that name chrony"; grep -rl chrony /etc/felhom* /usr/local/sbin /usr/local/bin /etc/systemd/system 2>/dev/null | head; +echo "## ps"; ps -eo pid,user,args | grep '[c]hronyd' +echo "## run dir"; ls -la /run/chrony /run/systemd/notify 2>&1 +echo "## uptime"; uptime -s diff --git a/documentation/audits/n100-clock-2026-11-11/tools/read2.sh b/documentation/audits/n100-clock-2026-11-11/tools/read2.sh new file mode 100644 index 00000000..a419b60e --- /dev/null +++ b/documentation/audits/n100-clock-2026-11-11/tools/read2.sh @@ -0,0 +1,14 @@ +# READ ONLY +echo "#### $(hostname) $(date -u +%FT%TZ)" +sysctl net.unix.max_dgram_qlen kernel.printk_ratelimit 2>&1 +echo "## cmdline"; cat /proc/cmdline +echo "## apparmor denied this boot: $(journalctl -b --no-pager 2>/dev/null | grep -c 'apparmor=\"DENIED\"')"; journalctl -b --no-pager | grep 'apparmor="DENIED"' | head -5 | cut -c1-300 +echo "## audit lines with chrony, all boots"; journalctl --no-pager 2>/dev/null | grep -i 'audit.*chrony\|chrony.*audit' | head -5 | cut -c1-300 +echo "## the second around the chrony start, this boot" +T=$(journalctl -u chrony -b --no-pager -o short-iso | grep -m1 'Starting chrony' | awk '{print $1}'); echo "start: $T" +journalctl -b --no-pager -o short-monotonic --since "$(date -d "$T - 1 sec" '+%F %T')" --until "$(date -d "$T + 2 sec" '+%F %T')" | cut -c1-200 | head -150 +echo "## chrony lines per boot (all boots)"; for b in -5 -4 -3 -2 -1 0; do echo "-- boot $b"; journalctl -u chrony -b $b --no-pager 2>/dev/null | grep -v '^-- ' | grep 'Starting\|Started\|Fatal\|Failed\|Stopp\|exit' | head -8 | cut -c1-160; done +echo "## sandbox env of pid1"; systemctl show -p NotifySocket -p DefaultEnvironment 2>&1 | head; ls -la /run/systemd/ | head -30 +echo "## tailscale"; systemctl is-active tailscaled 2>&1 +echo "## rtc"; ls /dev/rtc* /dev/ptp* /dev/pps* 2>&1 +echo "## cpu"; nproc; grep -m1 'model name' /proc/cpuinfo diff --git a/documentation/audits/n100-clock-2026-11-11/tools/read4b.sh b/documentation/audits/n100-clock-2026-11-11/tools/read4b.sh new file mode 100644 index 00000000..9d379ad4 --- /dev/null +++ b/documentation/audits/n100-clock-2026-11-11/tools/read4b.sh @@ -0,0 +1,18 @@ +main() { +echo "#### $(hostname) $(date -u +%FT%TZ)" +echo "## fresh compile of the on-disk profile, to stdout only (no load, no cache write)" +apparmor_parser -S -K --skip-bad-cache-rebuild -o /dev/stdout /etc/apparmor.d/usr.sbin.chronyd 2>/dev/null | sha256sum +apparmor_parser -S -K -o /dev/stdout /etc/apparmor.d/usr.sbin.chronyd 2>&1 >/dev/null | head -3 +echo "## loaded raw policy sha256"; cat /sys/kernel/security/apparmor/policy/profiles/usr.sbin.chronyd*/raw_sha256; echo +echo "## cache dirs"; ls -la --time-style=full-iso /var/cache/apparmor/ +for d in /var/cache/apparmor/*/; do echo "$d features-sha=$(sha256sum < $d/.features | cut -c1-16) files=$(ls $d | wc -l)"; ls -l --time-style=full-iso $d/usr.sbin.chronyd $d/.features; done +echo "## kernel features now"; cat /sys/kernel/security/apparmor/features/policy/versions/* 2>/dev/null | tr '\n' ' '; echo; aa-features-abi -x 2>/dev/null | sha256sum | cut -c1-16 +echo "## newest source file the profile depends on"; for f in /etc/apparmor.d/usr.sbin.chronyd /etc/apparmor.d/local/usr.sbin.chronyd /etc/apparmor.d/tunables/global /etc/apparmor.d/abstractions/base /etc/apparmor.d/abstractions/nameservice /etc/apparmor.d/abi/*; do ls -l --time-style=full-iso $f 2>/dev/null; done | awk '{print $6,$7,$9}' | sort | tail -6 +echo "## files in /etc/apparmor.d changed 2026-07-18 or 2026-08-21"; find /etc/apparmor.d -newermt 2026-07-18 ! -newermt 2026-07-19 | head; find /etc/apparmor.d -newermt 2026-08-21 ! -newermt 2026-08-22 | head +echo "## versions"; dpkg-query -W apparmor chrony; apparmor_parser --version | head -1 +echo "## dpkg log"; zgrep -h -E ' (upgrade|install) (apparmor|chrony|proxmox-kernel-[0-9.]+-[0-9]+-pve-signed):' /var/log/dpkg.log* | sort | tail -14 +echo "## kernels booted"; journalctl --list-boots --no-pager | tail -8; for b in -4 -3 -2 -1 0; do journalctl -b $b -k --no-pager -o cat 2>/dev/null | head -1 | cut -c1-60; done +echo "## chronyd confinement now"; ps -eZ 2>/dev/null | grep -i chrony || echo "no chronyd process" +echo "## what installed chrony 4.8"; zgrep -h -B3 -A3 'chrony' /var/log/apt/history.log* 2>/dev/null | grep -E 'Start-Date|Commandline|Requested-By|chrony' | cut -c1-260 | tail -12 +} +main /dev/null | sha256sum +echo "## loaded"; cat /sys/kernel/security/apparmor/policy/profiles/usr.sbin.chronyd*/raw_sha256; echo +echo "## cache"; sha256sum /var/cache/apparmor/*/usr.sbin.chronyd; ls -ld --time-style=full-iso /var/cache/apparmor/*/ /var/cache/apparmor/*/.features +echo "## state"; systemctl is-active chrony; dpkg-query -W chrony; uname -r; chronyc -n tracking 2>&1 | grep -E 'System time|Leap' ; date -u +%s.%N +echo "## chrony fatal lines"; journalctl -u chrony --no-pager -o short-iso 2>/dev/null | grep -c 'Could not send notification' +echo "## stale check, every cached profile: cache differs from a fresh compile"; n=0; bad=0; cur=$(ls -d /var/cache/apparmor/*/ | while read d; do [ "$(sha256sum < $d/.features)" = "$(aa-features-abi -x 2>/dev/null | sha256sum)" ] && echo $d; done | head -1); echo "current dir: $cur" +for f in /etc/apparmor.d/*; do [ -f "$f" ] || continue; b=$(basename $f); [ -f "$cur$b" ] || continue; n=$((n+1)); a=$(apparmor_parser -S -K "$f" 2>/dev/null | sha256sum | cut -d' ' -f1); c=$(sha256sum < "$cur$b" | cut -d' ' -f1); [ "$a" = "$c" ] || { bad=$((bad+1)); echo "STALE: $b"; }; done; echo "checked=$n stale=$bad" +} +main P4: comfort feature needing a new unmeasured mechanism; the default password hole is closed.** | — | — | CC | -| **R-925** | Security & access | P1 | **`manifests/felhom.secret.yaml` carries REAL secret values, and the Gitea repo that holds it is readable by anyone on the internet with no login.** FOUND 2026-10-09 by a background security review of the legal-pages commit, which flagged the served `` comments; chasing what those comments pointed at found this instead. **MEASURED, in this order:** (1) `scripts/manifest_bearer_gate.py` itself prints `manifests/felhom.secret.yaml:39 KNOWN-BACKLOG committed secret 65cee3c4…7a86`; (2) the file holds **live-shaped values, not placeholders** — `SECRET_KEY` (69 chars), `SUPERUSER_EMAIL`/`SUPERUSER_PASSWORD` (18), Umami `APP_SECRET` (66) and `POSTGRES_PASSWORD` (34), and a `username`/`password` pair (18); values were never printed, only measured by length; (3) `gitea.dooplex.hu` resolves to **37.191.56.193**, the same public address as the website; (4) an **anonymous** `curl` returns HTTP 200 and 1686 bytes for that exact file, and 200 for `hub/internal/store/store.go` and `manifests/hub.yaml`; (5) **the off-network control**, which is what makes (4) mean anything — fetched from outside the operator's network entirely: a real path returns the file's first line, a nonsense path returns 404, so the 200 is genuine anonymous read from the internet and not a LAN-only ACL. **This contradicts the project's own written model**: `documentation/runbooks/secrets.md:3-4` says *“Secret values are never committed to git. The manifests in `manifests/` carry only placeholders + comments that point here.”* That promise is false today, which is the P1 wording in this register's own scale — see the ranking note below. The same runbook already states the rule that governs the fix: *“The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.”* **Known neighbours, none of which cover this:** R-887 records that an outside crawler walks the public Gitea pages and leaves *“Gitea's exposure to the crawler”* as the operator's undone item; R-580 still owes a Gitea admin token rotation. Neither says a secret file is world-readable. **Why P2 and not P1, stated so the operator can overrule it:** the exposed credentials guard **analytics and an undeployed healthchecks instance, not household data** — measured: `umami-db` is a **ClusterIP** service on 5432 with no external IP, so the Postgres password is not reachable from the internet; the realistic harm is session forgery against the public `stats.felhom.eu` via `APP_SECRET`, and reuse of those passwords anywhere else. No customer box, hub token or escrow key is in this file. **CC did NOT change anything**: making the repo private could break the public day-0 path (the installer is fetched from a public tag, R-110), and rotation plus repository visibility are operator decisions on production infrastructure. **PARTLY FIXED 2026-10-09, and RE-RANKED P2 -> P1 on what it turned out to be.** The exposed value was not an analytics password: it was the **Gitea `admin` account password** (`is_admin: true`; `/api/v1/admin/users` answered 200), published in a repo `gitea.dooplex.hu` serves anonymously to the internet. That is push access to every repo — including the one whose `website/` is git-synced live and whose `scripts/` is published by tag to **every new box installer** (R-110), i.e. a supply-chain path onto customer hardware. My first ranking said P2 because I had only measured the analytics blast radius; the credential test is what corrected it. **DONE (CC, operator-instructed):** (1) `umami-config` `APP_SECRET` + `POSTGRES_PASSWORD` rotated — the password changed **inside Postgres** too (`ALTER USER`), because the env var is only read at first init; verified by a real beacon returning 200 with a bogus-site-id control still returning 400. (2) `healthchecks-config` `SECRET_KEY` + `SUPERUSER_PASSWORD` rotated (nothing consumes them — there is no healthchecks Deployment). (3) `gitea-creds` no longer holds the admin password at all: it holds a **scoped token** (`read:package` + `read:repository`). **Both scopes are load-bearing and the second was found by breaking it** — a package-only token made the hub log `Template fetch: unexpected status 403`, because the template fetcher reads a raw file out of the `felhom-controller` repo. (4) The Gitea **admin password** was changed and `gitea-system/gitea-admin` updated; **verified: new password 200, old published password 401 — the leak is dead.** (5) The file is removed from git and `.gitignore`'s `*secret*` now applies to it; `manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out is gone, red-proofed with a decoy (exit 1 with, 0 without). **AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part:** the `rollout restart` needed to pick up the new umami secret put umami into **CrashLoopBackOff and took `stats.felhom.eu` down (503)**. Cause was not the rotation — at `memory: 512Mi` the pod runs for months but **cannot restart**, because startup (Prisma + Next.js) peaks over the limit and is OOMKilled (exit 137). Raised to 1Gi, **in the manifest, not just live** (the `.claude/rules/manifests.md` rule: a bare `kubectl set` is reverted by the next sync and the fix silently disappears). Service restored and verified. New values are in a 0600 file on DooPlex (`~/rotated-secrets-2026-10-09.txt`), never echoed; the operator moves them to the password manager and deletes it. **WHAT REMAINS, and it is bigger than what was fixed:** that one password is still the admin password in about **12 other namespaces** — `nextcloud`, `paperless`, `bookstack` (a **database root** password), `tandoor`, `calibre`, `adventurelog`, `gokapi`, `qbittorrent` (x2), `servarr`, `homepage` (x2). Rotating Gitea does not touch them: each is its own login and each is still the string that was published. **Also owed:** a Gitea token for user `kisfenyo` sits in plaintext in the `origin` URL of the local `homelab-manifests` clone (`.git/config`); CC printed it to a session transcript while investigating, so it should be rotated regardless — this is R-580's shape (store the remote without credentials). **Not a finding:** `homelab-manifests` itself is private (404 anonymously) and does not contain the password; ArgoCD's repo credential is a separate token, so the rotation did not touch it. **2026-11-11:** CC's `felhom-agent` clone no longer holds a login and a token in its address: git gets the token from the secret store for one command (`scripts/git-credential-sec.sh`, on disk only until a code push passes the golden gate; proof and the before/after measurement in `documentation/audits/no-token-in-address-2026-11-11/`). Two clones still hold one (`cc-secrets`, `clones-plugmeter-2026-10-13/felhom-agent`): the operator's yes did not come in the 30 minutes the brief gave. The token of login `kisfenyo` was printed into a session on 2026-11-10; the operator rotates it once, into the store only (`GITEA_GIT_TOKEN`). | **NARROWED 2026-10-09 — the felhom.eu half is DONE and verified (Gitea admin password rotated, old one now 401; umami + healthchecks rotated; file de-gitted; gate carve-out removed). What remains is the ~12 reused logins elsewhere in the homelab and the repo-visibility decision; owner: operator** **2026-10-10 (release session), two consequences measured:** (a) DooPlex's own Docker login for `gitea.dooplex.hu` (`~/.docker/config.json`) still holds the OLD admin password — the registry token endpoint answers 401 for it and 200 for the current one — so every image push from DooPlex fails `unauthorized` until it is refreshed; the 0.145.0/0.305.0 pushes used a one-off login in a scratch `DOCKER_CONFIG`, shredded after. (b) The `felhom` ArgoCD app reads OutOfSync for `Secret/gitea-creds`, `Secret/healthchecks-config`, `Secret/umami-config` and `Deployment/umami`: a whole-app sync would push git's (de-gitted) view over the rotated live Secrets — the hub deploy synced ONLY `Deployment/hub` (`audits/release-2026-10-10/hub-0.145.0-deploy.txt`). Refreshing the DooPlex login is a DooPlex change (operator's word). **2026-10-22 night (Part 6, read only, `audits/night-2026-10-22/P6-repos-private/01-readers.md`):** homelab-manifests is already private; felhom-controller can go private after one change — CI clones with a read token (check first that the hub's `gitea-creds` token user can read a private `admin` repo); felhom.eu must stay public now (the website git-sync has no credential), felhom-agent too (every install fetches 16 config files from `raw/tag` with no login), and the catalog too (every box clones it with no login, and a token on a customer box is readable by its owner). Making a repository private does NOT hide its images or packages: they follow the user `admin`, which is public — and boxes need that (NOT-A-FINDING). **Count corrected:** the served `` comments are 61 on four pages (adatkezeles 33, feltetelek 15, teszteles 7, en/testing 6), not „32 on /adatkezeles”. The visibility question is decision D29 in STATUS. **2026-10-22 evening — D29 = no (operator):** the controller repository stays public for now; this row's secret rotation is unchanged. | — | **OPERATOR RULED 2026-10-09: (a) he rotates the remaining services himself — CC's scope stopped at the Felhom boundary; (b) the repo STAYS anonymously readable for now**, because making it private breaks the website git-sync and the installer tag fetch, which both clone with no credentials (R-110). The standing consequence of (b): no secret may ever enter this repo again, which `manifest_bearer_gate.py` now enforces with no exemption; and if (b) is reversed, the 61 served `` comments (four pages, 33 on /adatkezeles) must be stripped in the same change. **The exact checklist of the 12 remaining secrets** (namespace / secret / key, re-measured after the rotation, with the two traps that bite — a DB password is not changed by editing the Secret, and a pod that has run for months may not restart) is in `documentation/runbooks/secrets.md`. Operator: **(1)** work that checklist, `bookstack-db/root-password` first because it is a database root password and every one of these hosts answers on the public internet; (`bookstack` first — it is a database root password); **(2)** rotate the `kisfenyo` Gitea token embedded in the local `homelab-manifests` remote URL, and store the remote without credentials; **(3)** decide whether `gitea.dooplex.hu` should answer anonymously at all — remembering the website git-sync clones it with no credentials and the installer is fetched from a public tag, so making it private breaks both unless they get credentials first, and the 61 served `` comments become a map of it; **(4)** move `~/rotated-secrets-2026-10-09.txt` into the password manager and delete it. If nothing is done: the published password keeps opening a dozen services, even though Gitea itself is now safe | operator | +| **R-925** | Security & access | P1 | **`manifests/felhom.secret.yaml` carries REAL secret values, and the Gitea repo that holds it is readable by anyone on the internet with no login.** FOUND 2026-10-09 by a background security review of the legal-pages commit, which flagged the served `` comments; chasing what those comments pointed at found this instead. **MEASURED, in this order:** (1) `scripts/manifest_bearer_gate.py` itself prints `manifests/felhom.secret.yaml:39 KNOWN-BACKLOG committed secret 65cee3c4…7a86`; (2) the file holds **live-shaped values, not placeholders** — `SECRET_KEY` (69 chars), `SUPERUSER_EMAIL`/`SUPERUSER_PASSWORD` (18), Umami `APP_SECRET` (66) and `POSTGRES_PASSWORD` (34), and a `username`/`password` pair (18); values were never printed, only measured by length; (3) `gitea.dooplex.hu` resolves to **37.191.56.193**, the same public address as the website; (4) an **anonymous** `curl` returns HTTP 200 and 1686 bytes for that exact file, and 200 for `hub/internal/store/store.go` and `manifests/hub.yaml`; (5) **the off-network control**, which is what makes (4) mean anything — fetched from outside the operator's network entirely: a real path returns the file's first line, a nonsense path returns 404, so the 200 is genuine anonymous read from the internet and not a LAN-only ACL. **This contradicts the project's own written model**: `documentation/runbooks/secrets.md:3-4` says *“Secret values are never committed to git. The manifests in `manifests/` carry only placeholders + comments that point here.”* That promise is false today, which is the P1 wording in this register's own scale — see the ranking note below. The same runbook already states the rule that governs the fix: *“The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.”* **Known neighbours, none of which cover this:** R-887 records that an outside crawler walks the public Gitea pages and leaves *“Gitea's exposure to the crawler”* as the operator's undone item; R-580 still owes a Gitea admin token rotation. Neither says a secret file is world-readable. **Why P2 and not P1, stated so the operator can overrule it:** the exposed credentials guard **analytics and an undeployed healthchecks instance, not household data** — measured: `umami-db` is a **ClusterIP** service on 5432 with no external IP, so the Postgres password is not reachable from the internet; the realistic harm is session forgery against the public `stats.felhom.eu` via `APP_SECRET`, and reuse of those passwords anywhere else. No customer box, hub token or escrow key is in this file. **CC did NOT change anything**: making the repo private could break the public day-0 path (the installer is fetched from a public tag, R-110), and rotation plus repository visibility are operator decisions on production infrastructure. **PARTLY FIXED 2026-10-09, and RE-RANKED P2 -> P1 on what it turned out to be.** The exposed value was not an analytics password: it was the **Gitea `admin` account password** (`is_admin: true`; `/api/v1/admin/users` answered 200), published in a repo `gitea.dooplex.hu` serves anonymously to the internet. That is push access to every repo — including the one whose `website/` is git-synced live and whose `scripts/` is published by tag to **every new box installer** (R-110), i.e. a supply-chain path onto customer hardware. My first ranking said P2 because I had only measured the analytics blast radius; the credential test is what corrected it. **DONE (CC, operator-instructed):** (1) `umami-config` `APP_SECRET` + `POSTGRES_PASSWORD` rotated — the password changed **inside Postgres** too (`ALTER USER`), because the env var is only read at first init; verified by a real beacon returning 200 with a bogus-site-id control still returning 400. (2) `healthchecks-config` `SECRET_KEY` + `SUPERUSER_PASSWORD` rotated (nothing consumes them — there is no healthchecks Deployment). (3) `gitea-creds` no longer holds the admin password at all: it holds a **scoped token** (`read:package` + `read:repository`). **Both scopes are load-bearing and the second was found by breaking it** — a package-only token made the hub log `Template fetch: unexpected status 403`, because the template fetcher reads a raw file out of the `felhom-controller` repo. (4) The Gitea **admin password** was changed and `gitea-system/gitea-admin` updated; **verified: new password 200, old published password 401 — the leak is dead.** (5) The file is removed from git and `.gitignore`'s `*secret*` now applies to it; `manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out is gone, red-proofed with a decoy (exit 1 with, 0 without). **AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part:** the `rollout restart` needed to pick up the new umami secret put umami into **CrashLoopBackOff and took `stats.felhom.eu` down (503)**. Cause was not the rotation — at `memory: 512Mi` the pod runs for months but **cannot restart**, because startup (Prisma + Next.js) peaks over the limit and is OOMKilled (exit 137). Raised to 1Gi, **in the manifest, not just live** (the `.claude/rules/manifests.md` rule: a bare `kubectl set` is reverted by the next sync and the fix silently disappears). Service restored and verified. New values are in a 0600 file on DooPlex (`~/rotated-secrets-2026-10-09.txt`), never echoed; the operator moves them to the password manager and deletes it. **WHAT REMAINS, and it is bigger than what was fixed:** that one password is still the admin password in about **12 other namespaces** — `nextcloud`, `paperless`, `bookstack` (a **database root** password), `tandoor`, `calibre`, `adventurelog`, `gokapi`, `qbittorrent` (x2), `servarr`, `homepage` (x2). Rotating Gitea does not touch them: each is its own login and each is still the string that was published. **Also owed:** a Gitea token for user `kisfenyo` sits in plaintext in the `origin` URL of the local `homelab-manifests` clone (`.git/config`); CC printed it to a session transcript while investigating, so it should be rotated regardless — this is R-580's shape (store the remote without credentials). **Not a finding:** `homelab-manifests` itself is private (404 anonymously) and does not contain the password; ArgoCD's repo credential is a separate token, so the rotation did not touch it. **2026-11-11:** CC's `felhom-agent` clone no longer holds a login and a token in its address: git gets the token from the secret store for one command (`scripts/git-credential-sec.sh`, on disk only until a code push passes the golden gate; proof and the before/after measurement in `documentation/audits/no-token-in-address-2026-11-11/`). Two clones still hold one (`cc-secrets`, `clones-plugmeter-2026-10-13/felhom-agent`): the operator's yes did not come in the 30 minutes the brief gave. The token of login `kisfenyo` was printed into a session on 2026-11-10; the operator rotates it once, into the store only (`GITEA_GIT_TOKEN`). **2026-11-11 (afternoon): no clone holds a credential in its address any more** (16 clones read; three use the clone's own helper `scripts/git-credential-sec.sh` and the store entry `GITEA_GIT_TOKEN`). The other 13 clones still use the global `store` helper and the clear-text file in the home folder (3 lines, two hosts); a proposal to retire it is in `REPORT-golden-token-clock-certpin-2026-11-11.md`, nothing was changed. No token was rotated, made or deleted; the rotation of login `kisfenyo`'s token stays the operator's „do”, and its push step is in R-1390. | **NARROWED 2026-10-09 — the felhom.eu half is DONE and verified (Gitea admin password rotated, old one now 401; umami + healthchecks rotated; file de-gitted; gate carve-out removed). What remains is the ~12 reused logins elsewhere in the homelab and the repo-visibility decision; owner: operator** **2026-10-10 (release session), two consequences measured:** (a) DooPlex's own Docker login for `gitea.dooplex.hu` (`~/.docker/config.json`) still holds the OLD admin password — the registry token endpoint answers 401 for it and 200 for the current one — so every image push from DooPlex fails `unauthorized` until it is refreshed; the 0.145.0/0.305.0 pushes used a one-off login in a scratch `DOCKER_CONFIG`, shredded after. (b) The `felhom` ArgoCD app reads OutOfSync for `Secret/gitea-creds`, `Secret/healthchecks-config`, `Secret/umami-config` and `Deployment/umami`: a whole-app sync would push git's (de-gitted) view over the rotated live Secrets — the hub deploy synced ONLY `Deployment/hub` (`audits/release-2026-10-10/hub-0.145.0-deploy.txt`). Refreshing the DooPlex login is a DooPlex change (operator's word). **2026-10-22 night (Part 6, read only, `audits/night-2026-10-22/P6-repos-private/01-readers.md`):** homelab-manifests is already private; felhom-controller can go private after one change — CI clones with a read token (check first that the hub's `gitea-creds` token user can read a private `admin` repo); felhom.eu must stay public now (the website git-sync has no credential), felhom-agent too (every install fetches 16 config files from `raw/tag` with no login), and the catalog too (every box clones it with no login, and a token on a customer box is readable by its owner). Making a repository private does NOT hide its images or packages: they follow the user `admin`, which is public — and boxes need that (NOT-A-FINDING). **Count corrected:** the served `` comments are 61 on four pages (adatkezeles 33, feltetelek 15, teszteles 7, en/testing 6), not „32 on /adatkezeles”. The visibility question is decision D29 in STATUS. **2026-10-22 evening — D29 = no (operator):** the controller repository stays public for now; this row's secret rotation is unchanged. | — | **OPERATOR RULED 2026-10-09: (a) he rotates the remaining services himself — CC's scope stopped at the Felhom boundary; (b) the repo STAYS anonymously readable for now**, because making it private breaks the website git-sync and the installer tag fetch, which both clone with no credentials (R-110). The standing consequence of (b): no secret may ever enter this repo again, which `manifest_bearer_gate.py` now enforces with no exemption; and if (b) is reversed, the 61 served `` comments (four pages, 33 on /adatkezeles) must be stripped in the same change. **The exact checklist of the 12 remaining secrets** (namespace / secret / key, re-measured after the rotation, with the two traps that bite — a DB password is not changed by editing the Secret, and a pod that has run for months may not restart) is in `documentation/runbooks/secrets.md`. Operator: **(1)** work that checklist, `bookstack-db/root-password` first because it is a database root password and every one of these hosts answers on the public internet; (`bookstack` first — it is a database root password); **(2)** rotate the `kisfenyo` Gitea token embedded in the local `homelab-manifests` remote URL, and store the remote without credentials; **(3)** decide whether `gitea.dooplex.hu` should answer anonymously at all — remembering the website git-sync clones it with no credentials and the installer is fetched from a public tag, so making it private breaks both unless they get credentials first, and the 61 served `` comments become a map of it; **(4)** move `~/rotated-secrets-2026-10-09.txt` into the password manager and delete it. If nothing is done: the published password keeps opening a dozen services, even though Gitea itself is now safe | operator | | **R-1243** | Security & access | P2 | **The hub's Longhorn backups sit unencrypted in the NFS folder any DooPlex pod can read (R-1241) — the two there now (2026-10-18, 2026-10-23) hold every household's Cloudflare and tunnel tokens in plaintext, because they predate hub 0.168.0.** Measured 2026-10-23 night (`audits/hub-token-seal-2026-10-23/` §3): the hub volume is `encrypted=false` and in Longhorn's `default` backup group (daily 04:00 + weekly Sunday 05:00, retain 1 each); the hub's own copy to ep0 is encrypted at rest (16/16 snapshots `crypt-mode: encrypt`, key on DooPlex's root disk and on paper) and in transit (SSH tunnel + pinned PBS TLS), every 6 h, restore-tested weekly. DECIDE: (a) an encrypted Longhorn volume for the hub (~5–10 min downtime, a homelab StorageClass + key in the cluster) or (b) take the hub volume out of the Longhorn backup — ep0's copy is the backup; the access log then has no backup until it rides the ep0 push. **CC's pick: (b).** And: delete the two plain copies (the operator's call; with (b) they never rotate out). | **VERIFY** — **2026-10-24: D40 put into effect** (operator ruling 07:22): the hub PVC `hub-data` carries `recurring-job-group.longhorn.io/default: disabled` (felhom.eu `3363ffa8`, ArgoCD sync of the PVC only) and the Longhorn volume `pvc-486c9809…` reads `disabled` too (labelled live to match, unchanged after 2 min); the NFS folder holds no copy of the volume (no `pvc-486c9809*` directory, no backup config naming it; Longhorn lists no backup volume for it; 53 other volumes backed up this morning). Closes after the dated check: tonight's 04:00 job makes no new hub backup while the others run. WAITING-ON-OPERATOR **2026-10-25 dated check DONE (read-only, the restore-truth session):** `kubectl -n longhorn-system get backups.longhorn.io` lists 106 backups made today (the newest 04:08:28Z — the job ran) and none of `pvc-486c9809` — the hub volume is out of the Longhorn backup as D40 meant. | the operator's pick; a homelab change for either option | Operator: pick (a) or (b), and say whether to delete the two plain copies after hub 0.168.0 is live | CC (the dated check) | | **R-1355** | Security & access | P2 | **DooPlex's own Gitea 1.26.2 is inside 19 new notices (2026-11-02, fixed 28.3.0: `GHSA-8p3x` OAuth2 account takeover, `GHSA-37xv` and `GHSA-8g99` public-only tokens reach private data, `GHSA-32h8` Actions secrets) — on the host that holds our code and registry.** Also new on DooPlex: Vaultwarden 1.37.4 (fix 1.37.5), Uptime Kuma 2.5.6 (fix 2.5.9), Paperless 3.3.4 (fix 3.4.0). Felhom changes nothing on DooPlex. Read in `documentation/security-watch/state.json`, group DooPlex. | OPEN — filed 2026-11-03 | the homelab session | add them to `audits/http2shell-2026-10-27/HOMELAB-LIST.md`'s work; which notices reach a self-hosted Gitea with our settings is the homelab session's first read | operator (homelab session) | | **R-779** | Security & access | P4 | **[P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box.** Measured 2026-10-01 19:51 UTC (`audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt`): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. **Needs:** one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs** **Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up.** | — | — | CC + operator | @@ -279,7 +279,7 @@ stopping line that lies. | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| -| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` **2026-11-11:** the Clock list measured it again (32.6 s behind, time service not running) and leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CA03-6; class R-1384). | OPEN — filed 2026-10-26 **2026-11-10:** still so. The host clock is 32 s behind DooPlex (it was 14 s on 2026-10-26); `chrony` failed again at the 2026-11-08 05:35 boot with the same line; `NTPSynchronized=no`. Seen in a release run, where the box's log times read half a minute early (`documentation/audits/release-space-fixes-2026-11-10/agent/13-n100-clock.txt`). Not fixed by hand: the N100 takes no hand step in a release run. | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC | +| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` **2026-11-11:** the Clock list measured it again (32.6 s behind, time service not running) and leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CA03-6; class R-1384). **2026-11-11 (afternoon, read only): the cause is found.** The kernel loaded an OLD cached AppArmor profile for chrony (the one of chrony 4.6.1); chrony 4.8 writes to `/run/systemd/notify`, the old profile refuses that write, chrony treats it as fatal. Proof: loaded profile sha256 `dbf67f16…` against a fresh compile `83417e83…` on the N100, equal on demo-hp; the kernel's `DENIED` line of 2026-10-16. The order that made it: a new kernel started once (2026-10-07 12:11), the box went back to the old kernel, chrony was updated there (14:15), the new kernel started again and used its stale cache folder (`documentation/audits/n100-clock-2026-11-11/README.md`). The product side is R-1388. The hand step is in `STATUS.md` „do”; it was not run and not tested. | OPEN — filed 2026-10-26 **2026-11-10:** still so. The host clock is 32 s behind DooPlex (it was 14 s on 2026-10-26); `chrony` failed again at the 2026-11-08 05:35 boot with the same line; `NTPSynchronized=no`. Seen in a release run, where the box's log times read half a minute early (`documentation/audits/release-space-fixes-2026-11-10/agent/13-n100-clock.txt`). Not fixed by hand: the N100 takes no hand step in a release run. | — | the operator runs the one hand step on the N100 (`STATUS.md`); then read `chronyc tracking` and close. The product fix is R-1388 | CC | | **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CC02, CC06-6, CC07-6, CC22-6). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC | | **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC | | **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. **2026-11-10 (release run) — part (2) RELEASED, VOUCHED and PROVEN on three test boxes** (agent 0.173.0, vouched by the operator 15:29 UTC; `documentation/audits/release-space-fixes-2026-11-10/agent/`): the folder went 49 → 1, 48 → 1 and 33 → 1 files (about 700 / 690 / 485 MB back) at the commit of THIS update, not at the next one (the earlier sentence here and in the agent's CHANGELOG was wrong, and is corrected there). A rollback after the clean-up worked on Tester 1: a signed update to the older version was crashed once, 9 seconds into the 60-second dwell, and the wrapper put the released version back in the same second; the failed version's file stays until the next good update (by design). The kept older kernel on demo-hp booted to healthy in 313 s and the box went back (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **Still open:** (1) the kernel clean-up as a program (`11-os-updates.md` §5.11.1, not built) and a second box; (3) template archives; (4) the `.bak-*` copies — the release does not reach them; (5) the failed-run logs. A household box gets part (2) at its first good agent update after the vouch. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC | @@ -291,11 +291,12 @@ stopping line that lies. | **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | WATCHING — **2026-10-24 02:4xZ dated check read (weekend-night session, read only):** Tester 1 (8 GB host, guest 4 GB, controller 0.341.0) ran its 03:30 copy at 01:30:00–01:30:13Z (3 apps, 157 MB + 86 MB + 16 KB; no „capped copy container did not start" WARN, no leftover `felhom-tier2-copy-*`); the guest cgroup's memory „full" stall counter on the host (`/sys/fs/cgroup/lxc/9201/memory.pressure`) read **total 444 µs** over the whole 11.1 h since the host started (15:32Z), so „full" stayed far under 20 % (20 % of 10 s = 2 s); 0 OOM kills. **Not shown:** that the copy ran in the capped container (the INFO log does not name it; Docker keeps only ~200 events, from 02:36Z). **Weak load:** 0.24 GB, not the soak's large copy — so the row stays WATCHING; close after a night whose copy moves ≥ 1 GB, read from the same counter. WATCHING — **RELEASED 2026-10-23 in controller 0.340.0** (floor 06:41 UTC; on all three test boxes, start-up „no leftover”, 0 copy containers; Tester 1's manual second-drive run: 3 apps, 0 containers left). **Open only for the consequence:** dated check 2026-10-24 — Tester 1's first night copy ran in the capped container and memory „full” pressure stayed under 20 % (the box's own night log). READY — rides controller 0.340.0. READY — waits for the controller 0.340.0 release. **2026-10-22 night — cause measured, fix built and proven on 9202, waits for the controller release.** Cause: the copy's page cache is charged to the guest and evicts the apps' working set. On 9202 (4 GB RAM, 1.8 GB held by a hog, a 6 GB copy): plain rsync full stall 5 702 / 2 464 ms, peak „full” 16.8 / 6.9 %; the same copy in a 512 MB memory-capped container 374 / 320 ms, 0.9 / 1.0 %. Controller main `03c0a69` + `ae39467` (unreleased, rides 0.340.0): the copy runs capped only for clean `/mnt/` paths with no symlink (security review); a docker error (exit 125) falls back to plain rsync. A real `POST /api/backup/tier2` on rc.4: 6 capped containers, each exit 0; Immich's copy equals its source (dry-run 0 differences, 44 = 44 files). Evidence `audits/night-2026-10-22/P4-R1214/`. **2026-10-23 night (stray-copy fix, night brief Part 1).** The capped copy's container outlived the copy (a timeout or a controller restart killed only the `docker` CLI; `--delete` went on outside the R-1180 guard). Controller main `66a69d4` + `912e592` (unreleased, rides 0.340.0): a fixed name + label per copy, removed by name on any error, every leftover removed at start before the scheduler and the web server, a restore refused while a copy runs or may run; `--mount` (a path the daemon does not see is refused, not created empty) and the image ID with `--pull never` (review C4, C7). Proven on 9202 with rc.5 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB copy → the new controller removed `felhom-tier2-copy-immich-948f2280` at start, the restore probe was refused during the copy, the redone copy equals the source; a 15 s test build → the timed-out copy's container removed, the partial copy stopped growing. A second review of that code (N2–N5) and the commit's own security review (two races) → `9d9e404`, `1babfe9`: a stray clears when a later removal works, the start-up check never removes this process's own copy and holds the destination's lock, one copy per destination at a time, `RunTier2` refuses during a restore for every caller and records itself in one step with that check, every leg asks again. Re-proven on 9202 with rc.8 (`audits/night-2026-10-23/P1-straycopy/1*`). 9202 back on 0.339.0. | — | Release 0.340.0; then re-measure the 03:30 job on the 8 GB soak box and close if „full” stays low | CC | | **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a new install from the public download replaces the bootstrap and the off-site check; `pilot/TESTER2-handover.md`).** | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator | | **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC | -| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB21); the newest golden passed its 7-day limit on 2026-11-10 and no waiver is in force, so the gate refuses code pushes to `felhom.eu`. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement **2026-11-03: the Sunday 2026-11-01 bake was missed (it was in no brief): golden 0.344.0 is 8 days old, the floor 0.349.0; a CODE push to felhom.eu is refused until a bake and the vouch or a dated waiver. The night's close-out was documents only, which the gate exempts; no waiver written. First on the morning list.** | — | — | CC | +| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB21); the newest golden passed its 7-day limit on 2026-11-10 and no waiver is in force, so the gate refuses code pushes to `felhom.eu`. **2026-11-11: golden 0.352.0 baked and published; one fresh box installed from it (`documentation/tests/golden-0.352.0-2026-11-11/README.md`); the gate is green (`python3 scripts/golden_currency_gate.py`, rc 0). The operator vouched it on 2026-11-11 at 14:48:11 CET.** | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement **2026-11-03: the Sunday 2026-11-01 bake was missed (it was in no brief): golden 0.344.0 is 8 days old, the floor 0.349.0; a CODE push to felhom.eu is refused until a bake and the vouch or a dated waiver. The night's close-out was documents only, which the gate exempts; no waiver written. First on the morning list.** | — | — | CC | | **R-1358** | Box system & updates | P3 | **cloudflared 2026.10.4 misses two dependency fixes (VULN-183456 chi, 2026.10.5; VULN-184686 go-jose, 2026.10.6; the newest is 2026.10.7).** No public text on reach. Not pinned tonight: the tunnel is a box's only door, and 9202 has none to prove it on. | OPEN — filed 2026-11-03 | — | raise `CloudflaredImage` on the next controller release with a tunnel proof on Tester 1 | CC | -| **R-1380** | Box system & updates | P3 | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** Found 2026-11-11 (the End-dates list, class T2, `documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB02-3 to DB02-7). MEASURED: the agent trusts one exact Proxmox certificate (a pin), and the certificate on the three test boxes ends about two years after install — the N100's about 2028-07-16, then demo-hp and Tester 1 within three months. Proxmox makes a new certificate by itself about 14 days before the old one ends (so about 2028-07-02 on the N100). No code path gives the agent the new fingerprint, no alarm names the cause, no runbook holds the hand step, and nobody has tried it. Every box installed in the same season fails in the same fortnight. P3 only because the first date is in 2028; it is certain, not a risk. Decision D76 in `STATUS.md`. | OPEN — filed 2026-11-11 (end-dates list) | — | After D76: the agent reads the certificate file on its own host and takes the new fingerprint (design in `03-host-agent.md` first); prove on a scratch host with a forced renewal. | CC | +| **R-1380** | Box system & updates | P3 | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** Found 2026-11-11 (the End-dates list, class T2, `documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB02-3 to DB02-7). MEASURED: the agent trusts one exact Proxmox certificate (a pin), and the certificate on the three test boxes ends about two years after install — the N100's about 2028-07-16, then demo-hp and Tester 1 within three months. Proxmox makes a new certificate by itself about 14 days before the old one ends (so about 2028-07-02 on the N100). No code path gives the agent the new fingerprint, no alarm names the cause, no runbook holds the hand step, and nobody has tried it. Every box installed in the same season fails in the same fortnight. P3 only because the first date is in 2028; it is certain, not a risk. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 297). Not built.** **2026-11-11 (afternoon): tried on demo-hp (`documentation/audits/certpin-spike-2026-11-11/README.md`).** A forced new certificate (`pvecm updatecerts --force`, then a reload of the web service; Proxmox's own renewal does that reload by itself) blinds the agent at once: `TLS pin mismatch`, no reconcile, no host report to the hub, 502 to the controller; guests and apps stay up. One operator mail names a symptom (`agent_channel_unknown`), none names the cause; the hub's Hosts page still read ONLINE eleven minutes after the last report (when it turns offline was not measured). The hand repair works and is written down (`documentation/runbooks/RUNBOOK-agent-proxmox-pin-repair.md`, 2.1 s from restart to a working call). **A CA file in place of the fingerprint works with no agent code:** with `proxmox.tls.ca_file` on a readable copy of the box's root CA and the fingerprint empty, the agent went through a second new certificate with no hand step; a wrong CA file was refused. Two facts for the fix: the agent's user cannot read Proxmox's certificate files, and a set fingerprint wins over the CA file. The design with both ways is `03-host-agent.md` §12a; CC's pick differs from the ruled shape, so it is decision D80. | OPEN — filed 2026-11-11 (end-dates list) | — | the operator rules D80 (CA file, or follow the fingerprint); then the design's open parts in `03-host-agent.md` §12a (the root step that refreshes the copy; an alarm that names the cause), then build, then prove on a scratch host with a forced renewal | CC | | **R-1382** | Box system & updates | P3 | **A box that was off past an end date: nobody knows if it comes back by itself (5 answers).** Found 2026-11-11 (the End-dates list, class T4, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; read in source and documents, never tried). After months without power: the web certificate has ended (DB01-6: does Traefik renew at the first start, and what does the household see until then), Debian's package keys may be too old to verify an update (DB16-6), the tunnel client may be below what Cloudflare accepts (DB19-6), a 30-day restore copy is past its date (DB24-6), and the 14-day countdown after a stopped off-site copy ran out while the box was off (DB04-6). Tester 2 has been off since 2026-10-04 and is the natural test. | OPEN — filed 2026-11-11 (end-dates list) | — | When Tester 2 comes back (or with a scratch host and a moved clock on Tier 0, as a task): record what renews by itself and what needs a hand; put the hand steps in a runbook. | CC | -| **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. Decision D79 in `STATUS.md`. | OPEN — filed 2026-11-11 (clock list) | — | After D79: the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC | +| **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 300). Not built.** **2026-11-11 (afternoon): why the N100's clock is wrong is known (R-1301, `documentation/audits/n100-clock-2026-11-11/README.md`).** Numbers from source: a host clock more than 120 s behind refuses every signed job (`felhom-agent/internal/authz/verifier.go:21`, `:153-162`); at about 1.25 s a day the N100 gets there in about 70 days (mid-January 2027, an estimate from one measurement). | OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC | +| **R-1388** | Box system & updates | P3 | **After a kernel went back and forth, the box can load an old security profile for an updated package — on the N100 this stopped the time service for good.** Found 2026-11-11 (`documentation/audits/n100-clock-2026-11-11/README.md`; read only, on three boxes). AppArmor keeps one cache folder for each kernel. A package update writes the new profile only into the folder of the kernel that runs. If the box then starts another kernel that it started once before, that kernel's folder still holds the old profile, and AppArmor uses it (the cached copy looks newer than the package's file). MEASURED on the N100: chrony 4.8 under the profile of chrony 4.6.1; the kernel refuses its write to `/run/systemd/notify`; the unit fails at every start since 2026-10-16. On a household box the product's own kernel rollback (the crash guard) makes the same order, and it is true for every profile, not only chrony's. Not measured: a box where the rollback really ran. Also seen: Tester 1's host still has chrony 4.6.1 while the other two have 4.8 from the OS-update leg; why was not looked at. | OPEN — filed 2026-11-11 (N100 clock diagnosis) | — | Design first, in `11-os-updates.md`: after an OS-update leg and after a kernel rollback, reload the profiles with the cache ignored (`apparmor_parser -r -T -W`), or clear the cache folders of the kernels that do not run. Red-proof on a scratch host: make the order, see the stale profile, see the fix remove it. Then say why Tester 1 did not take chrony 4.8. | CC | | **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC | | **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC | | **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC | @@ -331,8 +332,8 @@ stopping line that lies. | **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC | | **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) | | **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC | -| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. | OPEN — filed 2026-11-11 (end-dates list) | — | After D77: the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC | -| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. Decision D78 in `STATUS.md`. | OPEN — filed 2026-11-11 (clock list) | — | After D78: one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC | +| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC | +| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC | | **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC | ## Hub & operator — 17 rows (P2 1, P3 5, P4 11) @@ -429,3 +430,5 @@ stopping line that lies. | **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | **OPEN — (A) BUILT 2026-10-26 evening, closes after the first nightly run (2026-10-27 03:20 UTC) is read.** Both runners proven at once (jobs 2374 + 2375 on `felhom-gates-runner` / `-2`, 17:11:41Z). felhom.eu CI, one push each: **(a) docs only** 810 s (old runner) → 351 s (2 CPUs) → see the close-out; **(b) a gate script** 351 → 203 s (`ecd82aba`, job 2376); **(c) a hub change** 351 → 118 s (`7e57e824`, job 2378). Heavy suites by path + tag + nightly; `test_repo_gates.py` and due-checks group G test the wiring (130 s each → under 3 s). Decoys through CI's entry point with the filter ON: a docs-only register break → red; an inert gate → GREEN at first (a decoy hole, fixed in `e33ec821`) → red. No Go cache: CI has no Go (the 110 s was security-scan, now 27 s). `audits/r1297-ci-trim-2026-10-26/`. (B) DONE by the operator (`09` §3 decision 267). | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC | | **R-1381** | Process & tooling | P4 | **No document says when eleven things end, or where to read the date (11 answers).** Found 2026-11-11 (the End-dates list, class T3, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; search S18 found no document that lists end dates). OS support for Debian 13 and Proxmox 9 (DB17-2, DB17-3), the tunnel client's oldest allowed version (DB19-2), the installer ISO (DH19-3), and the outside accounts and tokens (DH23-2, DH24-2, DH27-2, DH28-2, DH29-2, DH31-2, DH32-2). This run could not read these dates (no account call, by its fences). The operator reads them once: the table „Only you can read these dates” in `documentation/audits/end-dates-list-2026-11-11/README.md`. | OPEN — filed 2026-11-11 (end-dates list) | — | When the operator has read the dates: write them in one place (a short table in `documentation/operations/`), and give the due-checks block the ones inside a year. | CC | | **R-1387** | Process & tooling | P4 | **A document and the source disagree on a time, in four places (4 answers).** Found 2026-11-11 (the Clock list, class W5, `documentation/audits/clock-list-2026-11-11/TABLE.md`). (1) `documentation/architecture/03-host-agent.md` names `Restart=always` and a watchdog for the agent's unit; the unit has `Restart=on-failure` and no watchdog (CA18-6, CA18-8). Which one is the design is not stated, so this session did not edit either. (2) The hub-database copy alarm: 13 hours in one place, 26 in another (CH23-7; a DooPlex procedure, listed for the operator, not edited). (3) Old „24 h / 02:30” text for the hub's daily job (CH17-6). **Seen beside the lists, same kind, not in the count:** (4) the comment at `felhom-controller/controller/cmd/controller/main.go:1412` says 14 days where the code keeps restore copies 30 days (D62); (5) `documentation/architecture/07-backup-architecture.md:455-457` cites `main.go` lines that have moved; (6) `documentation/operations/tailscale.md` says „both nodes” where three test hosts exist, and Tailscale was not running on demo-hp and Tester 1 on 2026-11-11 (not known if that is meant); (7) the threat model and the recovery inventory do not say the same about how often the hub bearer is rotated. | OPEN — filed 2026-11-11 (clock list) | — | Operator or a task: say which side is right for (1) and (2); then one docs commit. (3) is a text fix in the same commit. | CC | +| **R-1389** | Process & tooling | P4 | **From the operator's home network the installer address gives 404.** Found 2026-11-11 (`documentation/tests/golden-0.352.0-2026-11-11/fresh-install/installer-address-404-on-lan.txt`). The home router's DNS (192.168.0.250) answers `felhom.eu` with 192.168.0.192, the site copy on DooPlex that was retired on 2026-11-07 (R-1228). So `https://felhom.eu/scripts/…` gives 404 for every machine that uses that DNS; from DooPlex (public DNS) it gives 200. A household is not touched. A test box on the home network, and Tester 2 if it sits there, cannot get the installer by its public address. | OPEN — filed 2026-11-11 (golden fresh install) | — | The operator removes the `felhom.eu` override from the home DNS (it is his router), then one `curl -sI https://felhom.eu/scripts/` from a machine on that network reads 200. | operator | +| **R-1390** | Process & tooling | P4 | **`sec put` cannot push the store any more, so a rotated token is saved on DooPlex only.** Found 2026-11-11. `sec put` ends with a plain `git push` of the `cc-secrets` clone (`cc-secrets/ccsec/cli.py:51-56`). Since this run that clone's address holds no login, and its helper answers only inside `sec run GITEA_GIT_TOKEN -- …`. A plain push now fails for lack of a login; the new entry is committed in the clone and not pushed. Until it is fixed, the push is a second command: `sec run GITEA_GIT_TOKEN -- git -C /mnt/5_hdd/felhom.eu/git/cc-secrets push origin HEAD:main`. Not tried in this run (no store change was allowed). | OPEN — filed 2026-11-11 (token fix) | — | In `cc-secrets`: `sec put` runs its push through the store's own token (or says plainly „committed, not pushed: run …”); a test that a `put` on a clone with no login in the address ends pushed or says so. | CC | diff --git a/documentation/runbooks/RUNBOOK-agent-proxmox-pin-repair.md b/documentation/runbooks/RUNBOOK-agent-proxmox-pin-repair.md new file mode 100644 index 00000000..c531629b --- /dev/null +++ b/documentation/runbooks/RUNBOOK-agent-proxmox-pin-repair.md @@ -0,0 +1,44 @@ +# RUNBOOK — the agent cannot reach Proxmox after a certificate change (pin repair) + +**PROVEN on demo-hp 2026-11-11** (`documentation/audits/certpin-spike-2026-11-11/README.md`, R-1380). One box, one +run. Design and the open fix: `architecture/03-host-agent.md` §12a. + +## When to use it + +All three are true: + +1. The agent's log has this line: `proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint` + (`journalctl -u felhom-agent -n 50 --no-pager | grep "TLS pin mismatch"`). +2. The operator got the mail `agent_channel_unknown … agentapi: GET /storage: HTTP 502`, or the hub has no new host + report from the box. +3. The guests and the apps are up. (If they are down, this is a different fault.) + +## Steps (as root on the Proxmox host) + +1. Read the certificate that Proxmox serves, and the pin: + `echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256 -enddate` + `python3 -c "import json;print(json.load(open('/etc/felhom-agent/agent.json'))['proxmox']['tls'])"` + If the two fingerprints are equal, stop: the pin is not the fault. +2. Check that the served certificate is Proxmox's own: its fingerprint must equal the file's. + `openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -fingerprint -sha256` + If they differ, run `systemctl reload pveproxy`, wait 15 seconds, read step 1 again. Do not go on while they differ. +3. Save the config: `cp -p /etc/felhom-agent/agent.json /etc/felhom-agent/agent.json.pre-pin-repair` +4. Write the served fingerprint into `proxmox.tls.fingerprint` in `/etc/felhom-agent/agent.json`. Keep the file's + owner (`felhom-agent`) and mode (`0600`). The script of the proof does this: `tools/setpin.sh fp` in the audit + folder. +5. `systemctl restart felhom-agent` — **one time.** The unit allows 4 starts in 2 minutes; a fifth leaves the agent + stopped until the 2 minutes pass. +6. Proof that it works (a positive sign, not an empty log): + `grep 'felhom-agent@pve' /var/log/pveproxy/access.log | tail -3` shows new lines that end in `200`. + On demo-hp the first one came 2.1 seconds after the restart. +7. At the hub: a new `host-report` from the box in at most 5 minutes, and the event `agent_channel_recovered`. +8. Remove the saved file (it holds the agent's token): `shred -u /etc/felhom-agent/agent.json.pre-pin-repair` + +## If the agent does not come back + +Put the saved file back (`cp -p …pre-pin-repair agent.json`), wait 2 minutes, restart one time, and read +`journalctl -u felhom-agent -n 80 --no-pager`. + +## Not covered + +A household box has no path for this today: the steps need root on the host. That is the open part of R-1380. diff --git a/documentation/runbooks/workspace-CLAUDE.md b/documentation/runbooks/workspace-CLAUDE.md index 85a5a40a..9e8d4996 100644 --- a/documentation/runbooks/workspace-CLAUDE.md +++ b/documentation/runbooks/workspace-CLAUDE.md @@ -123,6 +123,8 @@ R-96 incident record (committed 2026-07-27) — rationale, not directives. > `cc-secrets/ARCHITECTURE.md` in this workspace. Never `source` or `cat` a credentials file. > **Deleting a stored secret: operator only (D60, 2026-10-27).** CC never runs `sec rm` or removes anything from the > store, even after the operator's yes — he runs `sec rm NAME` in his own terminal. The guard refuses it. +> **A repository address never holds a credential (2026-11-11, R-925).** A clone that needs a login uses its own +> credential helper (`felhom.eu/scripts/git-credential-sec.sh`) and `sec run GITEA_GIT_TOKEN -- git …`. Never print a git address. ## Live validation — no browser here diff --git a/documentation/tests/golden-0.352.0-2026-11-11/00-boot.txt b/documentation/tests/golden-0.352.0-2026-11-11/00-boot.txt new file mode 100644 index 00000000..779167ff --- /dev/null +++ b/documentation/tests/golden-0.352.0-2026-11-11/00-boot.txt @@ -0,0 +1,8 @@ +## boot 2026-11-11T12:54:28Z +drill VM reverted to virgin and cold-booted (RUNBOOK-manual-build §4.0); pve-manager/9.2.2 +pveam update: update successful; template debian-13-standard_13.6-1_amd64.tar.zst (checksum verified) +build-golden.sh v3.2.0 from felhom-agent 91a2924ad0, sha256 645b3b65… (the same file as the 0.350.0 bake) +token + registry user copied file->file from sec (GITEA_PKG_TOKEN, GITEA_PKG_USER); unit properties grep for the token value: 0 +token in bake.log: 0 (control with the token appended: 1) +registry login in the archive: tar listing for .docker/config.json = 0 (listing control: 3 other lines found) +bake started 12:55:43Z, ended about 13:02Z diff --git a/documentation/tests/golden-0.352.0-2026-11-11/02-round-trip.txt b/documentation/tests/golden-0.352.0-2026-11-11/02-round-trip.txt new file mode 100644 index 00000000..68f7e548 --- /dev/null +++ b/documentation/tests/golden-0.352.0-2026-11-11/02-round-trip.txt @@ -0,0 +1,5 @@ +## round trip 2026-11-11T13:02:59Z +GET https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.352.0/golden.tar.zst (anonymous) +HTTP 200 bytes 639712511 +d76336054282ec138d8b0c8b9fec63319aa06f192aa421d012890b265b9abced - +bake: d76336054282ec138d8b0c8b9fec63319aa06f192aa421d012890b265b9abced diff --git a/documentation/tests/golden-0.352.0-2026-11-11/03-teardown.txt b/documentation/tests/golden-0.352.0-2026-11-11/03-teardown.txt new file mode 100644 index 00000000..47b3970d --- /dev/null +++ b/documentation/tests/golden-0.352.0-2026-11-11/03-teardown.txt @@ -0,0 +1,13 @@ +== teardown 2026-11-11T13:03:03Z + Logical volume "vm-9100-disk-0" successfully removed. + Logical volume "vm-9100-disk-1" successfully removed. +purging CT 9100 from related configurations.. +pct destroy rc=0 +shred rc=0 +ls: cannot access '/root/.gitea-token': No such file or directory +ls: cannot access '/root/bake.log': No such file or directory +qemu count 0 at 2026-11-11T13:05:04Z +qemu-img snapshot -a virgin rc=0 +Snapshot list: +ID TAG VM_SIZE DATE VM_CLOCK ICOUNT +1 virgin 0 B 2026-07-03 14:12:15 0000:00:00.000 0 diff --git a/documentation/tests/golden-0.352.0-2026-11-11/04-vouch-readback.txt b/documentation/tests/golden-0.352.0-2026-11-11/04-vouch-readback.txt new file mode 100644 index 00000000..70fc730f --- /dev/null +++ b/documentation/tests/golden-0.352.0-2026-11-11/04-vouch-readback.txt @@ -0,0 +1,3 @@ +read 2026-11-11T13:48:44Z (the hub log prints CET); the hub's Day-0 page shows the same two hashes +2026/11/11 14:48:11 [INFO] Artifact manifest set: agent=0.173.0 golden=0.352.0 min_agent="0.158.1" wrapper_sha=false bundle_sha="51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1" +2026/11/11 14:48:11 [INFO] artifact vouch timing (R-1346): registry probes 166ms, sha lookups 189ms, bundle sha 10ms, fleet check + save 336ms, total 702ms diff --git a/documentation/tests/golden-0.352.0-2026-11-11/README.md b/documentation/tests/golden-0.352.0-2026-11-11/README.md new file mode 100644 index 00000000..eaaa50c7 --- /dev/null +++ b/documentation/tests/golden-0.352.0-2026-11-11/README.md @@ -0,0 +1,76 @@ +# Golden 0.352.0 — bake, publish, one fresh install, 2026-11-11 (afternoon brief, Part 1) + +Why: golden 0.350.0 passed the 7-day limit on 2026-11-10, and the gate refused code pushes to `felhom.eu` (R-468). +Procedure: 0.350.0's README, the same recipe. Controller 0.352.0 is the global floor (MinAgent 0.158.1). +Documents read: `architecture/09-update-architecture.md` (§3 decisions 247, 250), `architecture/05-hub-architecture.md` +(Day-0 vouch, D50), `runbooks/publish-train-rules.md`, `runbooks/nightly-rotation.md`, rows R-468, R-242. + +| | Previous (`../golden-0.350.0-2026-11-03/`) | This bake | +|---|---|---| +| `build-golden.sh` | v3.2.0 | same file | +| Controller | `felhom-controller:0.350.0` | **`felhom-controller:0.352.0`** (MinAgent **0.158.1**) | +| Traefik baked | v3.7.17 | same | +| Docker engine | the six approved pairs | same | +| Template | `debian-13-standard_13.6-1_amd64.tar.zst` | same | +| Token | from `sec`, file → file | same | + +## Pass markers (`bake.log`) + +``` +[golden] build-golden.sh v3.2.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.352.0 +[golden] upload OK (HTTP 201) +GOLDEN_VERSION=0.352.0 +GOLDEN_SHA256=d76336054282ec138d8b0c8b9fec63319aa06f192aa421d012890b265b9abced +``` + +Round trip: the registry's file, downloaded with no login and hashed, equals the bake's sha; 639,712,511 bytes +(`02-round-trip.txt`). Build guest and teardown: `00-boot.txt`, `03-teardown.txt`. +**Golden gate:** `python3 scripts/golden_currency_gate.py` → OK, rc 0. A baked golden with no vouch passes it; no +waiver was written. + +## The fresh install (`fresh-install/`) + +**The path, exactly.** The hub offers only the vouched golden (0.350.0). So the box ran the public installer 1.36.1 +(sha256 `e18cdd17…aa1f1`) with **one added line** after line 479, which names golden 0.352.0 and its sha256 +(`installer-the-one-line.diff.txt`). All other steps are the product's own: the hub pairing, the agent download, the +guest build, the setup-code mail, the claim page, the app page. + +| Step | Result | Proof | +|---|---|---| +| VM 9384 `golden1111-box` on demo-hp, disks on `nvme-scratch` (never the thin pool), Proxmox from the Felhom ISO | installed | `vm-create.txt`, `install/` | +| Installer run | host `Tester4-d35551` at the hub, agent **0.173.0**, ONLINE, guests 1/1 | `app.txt` (last line) | +| Setup-code mail | arrived 13:07:20Z | the test mailbox | +| Claim page | `POST /claim` → 302 at 13:19:51Z | `claim.txt` | +| Controller version on the box | **0.352.0** | `claim.txt` | +| One app: BentoPDF, through the page's own deploy request | 202 at 13:20:41Z; `running` at 13:21:31Z; its address answers 200; a name with no app answers 404 | `app.txt` | + +**The two reads the brief asked for** + +1. Time service on the fresh host: `chrony` active, 0.000001 s off, read at 13:05:52Z — **before** the Felhom + installer ran. Not read after it (see below). +2. Proxmox certificate, read from the network: subject `CN=box1111.felhom.test`, issuer `CN=Proxmox Virtual + Environment, O=PVE Cluster Manager CA`, valid 2026-11-10 13:05:27 to **2028-11-09 13:05:27** GMT, sha256 + `2D:C1:FE:7C:09:EA:80:F5:8D:89:15:DB:A4:B5:6B:4F:C5:07:0D:95:E5:C5:06:83:2F:B8:EB:7F:BB:79:AB:46`. + **The agent's pin was not read.** Indirect proof only: the agent is ONLINE at the hub and built the guest, and both + need a working call to Proxmox. + +**Why not read: CC locked itself out of the host.** The installer closes the host on purpose (new root password into +the hub's vault, SSH by key only). CC put no key on the host before the install. That was CC's mistake, not a product +fault. Lesson, now a line in the proof-VM traps: put your key on the host before you run the installer. + +**Seen on the way (not the golden's fault):** from the operator's LAN the name `felhom.eu` answers from the retired +site at 192.168.0.192, so the installer address gives 404 there; from DooPlex it gives 200 +(`installer-address-404-on-lan.txt`). Filed as a row. The hub's list showed "0/0 guests" for the new host at 13:21:40Z +and "ONLINE 1/1" at 13:28:30Z: the guest shows at the host's next report (`fresh-install/app.txt`). + +## Vouch — done by the operator, 2026-11-11 14:48:11 CET (`04-vouch-readback.txt`) + +- golden **0.352.0**, sha256 `d76336054282ec138d8b0c8b9fec63319aa06f192aa421d012890b265b9abced` +- agent **0.173.0**, sha256 `752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea`, bundle + `51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1` (both unchanged) +- min_agent **0.158.1** (unchanged) + +## Teardown + +`fresh-install/teardown.txt`: VM 9384 stopped and destroyed with its two disks; the helper script removed from demo-hp. +The hub host `Tester4-d35551` and appliance `3fb5fed3-504c-4c2c-a63f-8108d5c15b6e` are the operator's to remove. diff --git a/documentation/tests/golden-0.352.0-2026-11-11/bake.log b/documentation/tests/golden-0.352.0-2026-11-11/bake.log new file mode 100644 index 00000000..7ade52fc --- /dev/null +++ b/documentation/tests/golden-0.352.0-2026-11-11/bake.log @@ -0,0 +1,341 @@ +[golden] build-golden.sh v3.2.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.352.0 +[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) … + Logical volume "vm-9100-disk-0" created. + Logical volume pve/vm-9100-disk-0 changed. +Creating filesystem with 8388608 4k blocks and 2097152 inodes +Filesystem UUID: 809cd9a8-e042-431b-abcd-d8eb0a100d94 +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, + 4096000, 7962624 + Logical volume "vm-9100-disk-1" created. + Logical volume pve/vm-9100-disk-1 changed. +Creating filesystem with 6291456 4k blocks and 1572864 inodes +Filesystem UUID: 433a7dea-e960-424b-be3c-20f7076e7eda +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, +extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst' +Total bytes read: 553512960 (528MiB, 106MiB/s) +Detected container architecture: amd64 +Creating SSH host key 'ssh_host_rsa_key' - this may take some time ... +done: SHA256:HJsSglsPEot2RL07p81VwRRw4OVfWhi7NVOJnBScUM8 root@felhom-golden +Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ... +done: SHA256:Lt6rmdjcR5gQ5awqXI/n3JgloNv3jJSb2OTosPz8RVE root@felhom-golden +Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ... +done: SHA256:PLM8d/fxjmQEXBgLNuc/cRf9hfI5de1Fh9P2CxsfJoY root@felhom-golden +[golden] starting + installing Docker (official repo, trixie channel) … +[golden] Docker engine set PINNED to the approved release: containerd.io=2.4.2-3~debian.13~trixie docker-buildx-plugin=0.38.1-1~debian.13~trixie docker-ce=5:29.9.2-1~debian.13~trixie docker-ce-cli=5:29.9.2-1~debian.13~trixie docker-ce-rootless-extras=5:29.9.2-1~debian.13~trixie docker-compose-plugin=5.6.1-1~debian.13~trixie +apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct! +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = (unset), + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to the standard locale ("C"). +locale: Cannot set LC_CTYPE to default locale: No such file or directory +locale: Cannot set LC_MESSAGES to default locale: No such file or directory +locale: Cannot set LC_ALL to default locale: No such file or directory +apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct! +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = (unset), + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to the standard locale ("C"). +locale: Cannot set LC_CTYPE to default locale: No such file or directory +locale: Cannot set LC_MESSAGES to default locale: No such file or directory +locale: Cannot set LC_ALL to default locale: No such file or directory + installed: containerd.io 2.4.2-3~debian.13~trixie + installed: docker-buildx-plugin 0.38.1-1~debian.13~trixie + installed: docker-ce 5:29.9.2-1~debian.13~trixie + installed: docker-ce-cli 5:29.9.2-1~debian.13~trixie + installed: docker-ce-rootless-extras 5:29.9.2-1~debian.13~trixie + installed: docker-compose-plugin 5.6.1-1~debian.13~trixie +[golden] no approved guest release given - the template versions stay; first-night count vs an approved release: n/a +[golden] pending Debian upgrades in the baked guest (what a FUTURE approval may bring): 61 +[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation … +[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds … +[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) … +Unable to find image 'hello-world:latest' locally +latest: Pulling from library/hello-world +4f55086f7dd0: Pulling fs layer +4f55086f7dd0: Download complete +4f55086f7dd0: Pull complete +Digest: sha256:c567b231f65614688711a75f31518a064258a51d442117a3e771a3a8fdba78ef +Status: Downloaded newer image for hello-world:latest + docker OK (overlay2; data-root /var/lib/docker) + live-restore: on + /var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4 + /mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4 + both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576 +[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.352.0 (no registry cred at deploy) … + +WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'. +Configure a credential helper to remove this warning. See +https://docs.docker.com/go/credential-store/ + +0.352.0: Pulling from admin/felhom-controller +ecc510c1e359: Pulling fs layer +c3f83389ce1b: Pulling fs layer +e81c1fce9826: Pulling fs layer +63a1406b1961: Pulling fs layer +d747c9b6922d: Pulling fs layer +f0be24959d17: Pulling fs layer +153556725a0f: Pulling fs layer +63a1406b1961: Waiting +d747c9b6922d: Waiting +f0be24959d17: Waiting +153556725a0f: Waiting +e81c1fce9826: Verifying Checksum +e81c1fce9826: Download complete +ecc510c1e359: Verifying Checksum +ecc510c1e359: Download complete +63a1406b1961: Verifying Checksum +63a1406b1961: Download complete +d747c9b6922d: Verifying Checksum +d747c9b6922d: Download complete +c3f83389ce1b: Verifying Checksum +c3f83389ce1b: Download complete +f0be24959d17: Verifying Checksum +f0be24959d17: Download complete +153556725a0f: Verifying Checksum +153556725a0f: Download complete +ecc510c1e359: Pull complete +c3f83389ce1b: Pull complete +e81c1fce9826: Pull complete +63a1406b1961: Pull complete +d747c9b6922d: Pull complete +f0be24959d17: Pull complete +153556725a0f: Pull complete +Digest: sha256:3c785dd396b091f19d4c777db9d7a206e75ab018b6238b5d887e1a4c6cfd3b29 +Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.352.0 +gitea.dooplex.hu/admin/felhom-controller:0.352.0 +[golden] asking the controller which infra images it manages … +[golden] baking infra images (4): traefik:v3.7.17 cloudflare/cloudflared:2026.10.4 gtstef/filebrowser:1.5.8-stable gitea.dooplex.hu/admin/felhom-samba:1.2.0 … +v3.7.17: Pulling from library/traefik +881fc932939e: Pulling fs layer +c901ba6ff9e0: Pulling fs layer +4f376259b112: Pulling fs layer +7705a4c1b3f8: Pulling fs layer +7705a4c1b3f8: Waiting +881fc932939e: Verifying Checksum +881fc932939e: Download complete +c901ba6ff9e0: Verifying Checksum +c901ba6ff9e0: Download complete +881fc932939e: Pull complete +7705a4c1b3f8: Verifying Checksum +7705a4c1b3f8: Download complete +4f376259b112: Verifying Checksum +4f376259b112: Download complete +c901ba6ff9e0: Pull complete +4f376259b112: Pull complete +7705a4c1b3f8: Pull complete +Digest: sha256:f4bb143d1135a579b630ed491e17ec50f6f7c9c6a30585ff3d385471d2a86f5a +Status: Downloaded newer image for traefik:v3.7.17 +docker.io/library/traefik:v3.7.17 +2026.10.4: Pulling from cloudflare/cloudflared +2cc7ee286bf3: Pulling fs layer +c172f21841df: Pulling fs layer +218cf840d0d9: Pulling fs layer +f6069939f718: Pulling fs layer +d6b1b89eccac: Pulling fs layer +2780920e5dbf: Pulling fs layer +7c12895b777b: Pulling fs layer +3214acf345c0: Pulling fs layer +52630fc75a18: Pulling fs layer +dd64bf2dd177: Pulling fs layer +b839dfae01f6: Pulling fs layer +ebddc55facdc: Pulling fs layer +a0921f9d8de9: Pulling fs layer +b96fe2995f90: Pulling fs layer +810e2b405a08: Pulling fs layer +bd8962e29291: Pulling fs layer +cac2ae0193cb: Pulling fs layer +eb8da9d09a5e: Pulling fs layer +dd64bf2dd177: Waiting +b839dfae01f6: Waiting +ebddc55facdc: Waiting +a0921f9d8de9: Waiting +b96fe2995f90: Waiting +810e2b405a08: Waiting +bd8962e29291: Waiting +cac2ae0193cb: Waiting +eb8da9d09a5e: Waiting +2780920e5dbf: Waiting +7c12895b777b: Waiting +3214acf345c0: Waiting +52630fc75a18: Waiting +f6069939f718: Waiting +d6b1b89eccac: Waiting +2cc7ee286bf3: Download complete +c172f21841df: Verifying Checksum +c172f21841df: Download complete +218cf840d0d9: Verifying Checksum +218cf840d0d9: Download complete +f6069939f718: Verifying Checksum +f6069939f718: Download complete +2cc7ee286bf3: Pull complete +d6b1b89eccac: Verifying Checksum +d6b1b89eccac: Download complete +2780920e5dbf: Verifying Checksum +2780920e5dbf: Download complete +7c12895b777b: Verifying Checksum +7c12895b777b: Download complete +3214acf345c0: Verifying Checksum +3214acf345c0: Download complete +52630fc75a18: Verifying Checksum +52630fc75a18: Download complete +dd64bf2dd177: Verifying Checksum +dd64bf2dd177: Download complete +b839dfae01f6: Verifying Checksum +b839dfae01f6: Download complete +ebddc55facdc: Verifying Checksum +ebddc55facdc: Download complete +a0921f9d8de9: Download complete +810e2b405a08: Verifying Checksum +810e2b405a08: Download complete +bd8962e29291: Download complete +b96fe2995f90: Verifying Checksum +b96fe2995f90: Download complete +cac2ae0193cb: Verifying Checksum +cac2ae0193cb: Download complete +eb8da9d09a5e: Verifying Checksum +eb8da9d09a5e: Download complete +c172f21841df: Pull complete +218cf840d0d9: Pull complete +f6069939f718: Pull complete +d6b1b89eccac: Pull complete +2780920e5dbf: Pull complete +7c12895b777b: Pull complete +3214acf345c0: Pull complete +52630fc75a18: Pull complete +dd64bf2dd177: Pull complete +b839dfae01f6: Pull complete +ebddc55facdc: Pull complete +a0921f9d8de9: Pull complete +b96fe2995f90: Pull complete +810e2b405a08: Pull complete +bd8962e29291: Pull complete +cac2ae0193cb: Pull complete +eb8da9d09a5e: Pull complete +Digest: sha256:c6c7ae1d5dfdcb094ad344ed68ccffc26635c9c6be80965664c33baf064b0515 +Status: Downloaded newer image for cloudflare/cloudflared:2026.10.4 +docker.io/cloudflare/cloudflared:2026.10.4 +1.5.8-stable: Pulling from gtstef/filebrowser +e2de96513ba9: Pulling fs layer +93c39e448959: Pulling fs layer +b7597036c324: Pulling fs layer +c6591d351da9: Pulling fs layer +4f4fb700ef54: Pulling fs layer +74aec9809d97: Pulling fs layer +9f7b5accdf20: Pulling fs layer +19114688b75e: Pulling fs layer +2067d5df1f91: Pulling fs layer +5669e416ca05: Pulling fs layer +08c322a8a1d4: Pulling fs layer +19114688b75e: Waiting +2067d5df1f91: Waiting +5669e416ca05: Waiting +08c322a8a1d4: Waiting +c6591d351da9: Waiting +4f4fb700ef54: Waiting +74aec9809d97: Waiting +9f7b5accdf20: Waiting +e2de96513ba9: Verifying Checksum +e2de96513ba9: Download complete +c6591d351da9: Verifying Checksum +c6591d351da9: Download complete +4f4fb700ef54: Verifying Checksum +4f4fb700ef54: Download complete +93c39e448959: Verifying Checksum +93c39e448959: Download complete +b7597036c324: Verifying Checksum +b7597036c324: Download complete +e2de96513ba9: Pull complete +9f7b5accdf20: Verifying Checksum +9f7b5accdf20: Download complete +19114688b75e: Verifying Checksum +19114688b75e: Download complete +2067d5df1f91: Verifying Checksum +2067d5df1f91: Download complete +5669e416ca05: Verifying Checksum +5669e416ca05: Download complete +74aec9809d97: Verifying Checksum +74aec9809d97: Download complete +08c322a8a1d4: Verifying Checksum +08c322a8a1d4: Download complete +93c39e448959: Pull complete +b7597036c324: Pull complete +c6591d351da9: Pull complete +4f4fb700ef54: Pull complete +74aec9809d97: Pull complete +9f7b5accdf20: Pull complete +19114688b75e: Pull complete +2067d5df1f91: Pull complete +5669e416ca05: Pull complete +08c322a8a1d4: Pull complete +Digest: sha256:68455d4953bad8e984e1bc5d112ed1e9c918a0a04f104c13c9ece50790308894 +Status: Downloaded newer image for gtstef/filebrowser:1.5.8-stable +docker.io/gtstef/filebrowser:1.5.8-stable +1.2.0: Pulling from admin/felhom-samba +e2de96513ba9: Already exists +0ddbc9f3a6b6: Pulling fs layer +ce57a3f93416: Pulling fs layer +fb94eeec2fe1: Pulling fs layer +ce57a3f93416: Verifying Checksum +ce57a3f93416: Download complete +fb94eeec2fe1: Verifying Checksum +fb94eeec2fe1: Download complete +0ddbc9f3a6b6: Verifying Checksum +0ddbc9f3a6b6: Download complete +0ddbc9f3a6b6: Pull complete +ce57a3f93416: Pull complete +fb94eeec2fe1: Pull complete +Digest: sha256:da96f25033827fbb77d0ee398e86fe5b744e2f29be9de898994533340c97acf9 +Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.2.0 +gitea.dooplex.hu/admin/felhom-samba:1.2.0 +[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'. +[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'. +[golden] baking the first-boot SSH host-key regeneration unit (F3) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'. +[golden] identity-clean + minimize … +[golden] stop + archive … +INFO: including mount point rootfs ('/') in backup +INFO: including mount point mp0 ('/var/lib/felhom') in backup +INFO: archive file size: 610MB +INFO: Finished Backup of VM 9100 (00:00:27) +[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_11_11-14_00_21.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive) +[golden] publishing golden (639712511 bytes, sha256 d76336054282ec13…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.352.0/golden.tar.zst +[golden] pre-delete existing: HTTP 404 (404/204 expected) +[golden] upload OK (HTTP 201) +GOLDEN_VERSION=0.352.0 +GOLDEN_SHA256=d76336054282ec138d8b0c8b9fec63319aa06f192aa421d012890b265b9abced +[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.352.0 / d76336054282ec138d8b0c8b9fec63319aa06f192aa421d012890b265b9abced +[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge) diff --git a/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/app.txt b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/app.txt new file mode 100644 index 00000000..aa33d5d2 --- /dev/null +++ b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/app.txt @@ -0,0 +1,13 @@ +2026-11-11T13:20:41Z +{"ok":true,"message":"Telepítés elindítva – az állapot a kártyán követhető"} + +HTTP 202 +13:20:51 {'deployed': True, 'state': 'deploying'} +13:21:01 {'deployed': True, 'state': 'deploying'} +13:21:11 {'deployed': True, 'state': 'starting'} +13:21:21 {'deployed': True, 'state': 'starting'} +13:21:31 {'deployed': True, 'state': 'running'} +route: 200 at 2026-11-11T13:21:40Z +negative control (no such app name): 404 +Tester4-d35551 Tester4 0.173.0 floor: declared MinAgent 9.2.2 7.0.2-6-pve ONLINE 0/0 0% 18% 16% unknown (no running customer guest) 16% +2026-11-11T13:28:30Z Tester4-d35551 Tester4 0.173.0 floor: declared MinAgent 9.2.2 7.0.2-6-pve ONLINE 1/1 0% 21% 21% absent (no cloudflared container in the guest (no tunnel set up on this box)) 21% local-lvm diff --git a/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/claim.txt b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/claim.txt new file mode 100644 index 00000000..5304ff61 --- /dev/null +++ b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/claim.txt @@ -0,0 +1,3 @@ +GET /claim HTTP/2 200 +csrf_len 64 cookie_present yes +POST /claim HTTP/2 302 location: / diff --git a/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/02-eula.png b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/02-eula.png new file mode 100644 index 00000000..9a08c186 Binary files /dev/null and b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/02-eula.png differ diff --git a/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/03-disk.png b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/03-disk.png new file mode 100644 index 00000000..4990ca20 Binary files /dev/null and b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/03-disk.png differ diff --git a/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/04-location.png b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/04-location.png new file mode 100644 index 00000000..ded47ca5 Binary files /dev/null and b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/04-location.png differ diff --git a/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/05-password.png b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/05-password.png new file mode 100644 index 00000000..2bd5e10a Binary files /dev/null and b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/05-password.png differ diff --git a/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/06-network.png b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/06-network.png new file mode 100644 index 00000000..7aaa8239 Binary files /dev/null and b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/06-network.png differ diff --git a/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/07-summary.png b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/07-summary.png new file mode 100644 index 00000000..ec147f06 Binary files /dev/null and b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/07-summary.png differ diff --git a/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/install-pressed.txt b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/install-pressed.txt new file mode 100644 index 00000000..c9f4731c --- /dev/null +++ b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/install/install-pressed.txt @@ -0,0 +1 @@ +Install pressed 13:02:31Z diff --git a/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/installer-address-404-on-lan.txt b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/installer-address-404-on-lan.txt new file mode 100644 index 00000000..bdcf9f88 --- /dev/null +++ b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/installer-address-404-on-lan.txt @@ -0,0 +1,17 @@ +== the installer address from the box and from DooPlex, 2026-11-11T13:06:42Z +-- box +2606:4700:3030::6815:12da felhom.eu +2606:4700:3037::ac43:b765 felhom.eu +search felhom.test +nameserver 192.168.0.250 +404 192.168.0.192 scripts +404 192.168.0.192 root +-- dooplex +2a06:98c1:3121::3 felhom.eu +2a06:98c1:3120::3 felhom.eu +nameserver 8.8.8.8 +nameserver 1.1.1.1 +200 188.114.97.3 scripts +200 188.114.97.3 root +server: cloudflare +cf-ray: a58d840b9a5f28d0-VIE diff --git a/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/installer-the-one-line.diff.txt b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/installer-the-one-line.diff.txt new file mode 100644 index 00000000..6ecfed01 --- /dev/null +++ b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/installer-the-one-line.diff.txt @@ -0,0 +1,8 @@ +e18cdd17823911e7d87da12fcb8d48b6827f1706a84b3168114fe19fe30aa1f1 /root/hi-orig.sh +197:SCRIPT_VERSION="1.36.1" # the SINGLE version source (F-1): -h and the run banners follow it. +line: 479 +479a480 +> ART_GOLDEN_VER=0.352.0; ART_GOLDEN_SHA=d76336054282ec138d8b0c8b9fec63319aa06f192aa421d012890b265b9abced +syntax rc=0 +2026-11-11T13:06:42Z +Running as unit: install1111.service; invocation ID: 8f11a38ee7c44269a2f521571357c14d diff --git a/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/teardown.txt b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/teardown.txt new file mode 100644 index 00000000..f99b1f49 --- /dev/null +++ b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/teardown.txt @@ -0,0 +1,40 @@ +== teardown 2026-11-11T13:48:56Z +name: golden1111-box +scsi0: nvme-scratch:9384/vm-9384-disk-0.raw,size=64G +scsi1: nvme-scratch:9384/vm-9384-disk-1.raw,size=48G +scsihw: virtio-scsi-single +stop rc=0 +purging VM 9384 from related configurations.. +destroy rc=0 +VMID NAME STATUS +341 night1004-tester1 running +341 +9202 +9401 +341.conf +removed '/root/vm-create-1111.sh' +9201-pkgs-after-partd.txt +9201-pkgs-before-partd.txt +e-crash.sh +e-pass.log +felhom-agent.sudoers.bak-pre-osapply +felhom-os-apply.bak-0.140.0 +felhom-os-apply.bak-0.141.1 +grub.default.bak-2026-10-04 +spike-bootnext-setup.sh +spike-bootnext-undo.sh +spike-cleanup.sh +spike-setup.sh +spike-undo.sh +spike-watchdog-setup.sh +spike-watchdog-undo.sh +Name Type Status +felhom-pbs pbs active +local dir active +local-lvm lvmthin active +nvme-scratch dir active + +4096 Nov 11 . +4096 Aug 21 .. +1705324544 Nov 11 felhom-installer-1.30.0-pve9.2-1.iso +removed '/var/lib/vz/template/iso/felhom-installer-1.30.0-pve9.2-1.iso' diff --git a/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/vm-create.txt b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/vm-create.txt new file mode 100644 index 00000000..b9b2248b --- /dev/null +++ b/documentation/tests/golden-0.352.0-2026-11-11/fresh-install/vm-create.txt @@ -0,0 +1,24 @@ +update VM 9384: -scsi0 nvme-scratch:64,format=raw -scsi1 nvme-scratch:48,format=raw +Formatting '/mnt/hdd_1/images/9384/vm-9384-disk-0.raw', fmt=raw size=68719476736 preallocation=off +scsi0: successfully created disk 'nvme-scratch:9384/vm-9384-disk-0.raw,size=64G' +Formatting '/mnt/hdd_1/images/9384/vm-9384-disk-1.raw', fmt=raw size=51539607552 preallocation=off +scsi1: successfully created disk 'nvme-scratch:9384/vm-9384-disk-1.raw,size=48G' +update VM 9384: -ide2 local:iso/felhom-installer-1.30.0-pve9.2-1.iso,media=cdrom +update VM 9384: -boot order=scsi0;ide2 +boot: order=scsi0;ide2 +cores: 6 +cpu: host +ide2: local:iso/felhom-installer-1.30.0-pve9.2-1.iso,media=cdrom,size=1665356K +memory: 10240 +meta: creation-qemu=11.1.3,ctime=1794401854 +name: golden1111-box +net0: virtio=BC:24:11:98:E4:72,bridge=vmbr0 +ostype: l26 +scsi0: nvme-scratch:9384/vm-9384-disk-0.raw,size=64G +scsi1: nvme-scratch:9384/vm-9384-disk-1.raw,size=48G +scsihw: virtio-scsi-single +smbios1: uuid=3fb5fed3-504c-4c2c-a63f-8108d5c15b6e +vga: std +vmgenid: cc62c7e4-c0d6-4766-abbf-8b12ef8f83c5 +STARTED +2026-11-11T12:57:41Z diff --git a/scripts/git-credential-sec.sh b/scripts/git-credential-sec.sh new file mode 100755 index 00000000..5710e1d8 --- /dev/null +++ b/scripts/git-credential-sec.sh @@ -0,0 +1,23 @@ +#!/bin/sh +# git-credential-sec.sh — git credential helper for CC's clones on DooPlex (Part C, 2026-11-11; R-925). +# +# A repository address never holds a credential. Git asks this helper for the password at the moment it needs one, +# and the helper answers from the environment that the secret store sets for one command: +# +# sec run GITEA_GIT_TOKEN -- git push origin main +# +# Per clone (no secret in any of the three settings): +# git config --local credential.helper "" +# git config --local --add credential.helper /mnt/5_hdd/felhom.eu/git/felhom.eu/scripts/git-credential-sec.sh +# git config --local credential.username +# +# Without `sec run` the helper gives no password and tells git to stop asking (no prompt, no other helper, no +# clear-text credentials file). A public repository still fetches, because the server asks for nothing. +# The helper writes the value only to git's pipe: never to a file, the terminal or a log. +[ "${1:-}" = get ] || exit 0 +if [ -z "${GITEA_GIT_TOKEN:-}" ]; then + printf 'git-credential-sec: no token in the environment. Run: sec run GITEA_GIT_TOKEN -- git \n' >&2 + printf 'quit=true\n' + exit 0 +fi +printf 'password=%s\n' "$GITEA_GIT_TOKEN" diff --git a/skills/felhom-build-deploy/SKILL.md b/skills/felhom-build-deploy/SKILL.md index 7982d5a8..e52ad040 100644 --- a/skills/felhom-build-deploy/SKILL.md +++ b/skills/felhom-build-deploy/SKILL.md @@ -93,7 +93,8 @@ Runs as the NON-ROOT `felhom-agent` user: `/usr/local/bin/felhom-agent --config > **Release and delivery today (corrected 2026-10-13, R-998 session — the copy-and-restart block below is the > pre-R-840 path):** release with `scripts/release-agent.sh ` (push the code first, the CHANGELOG entry after; -> package creds = the gitea entry of `~/.docker/config.json`, read into env, never printed). A box takes it ONLY by a +> package creds = the gitea entry of `~/.docker/config.json`, read into env, never printed). **Since 2026-11-11 the felhom-agent +> clone's address holds no login:** a push, a pull or the release script's tag push runs inside `sec run GITEA_GIT_TOKEN -- …`. A box takes it ONLY by a > signed job per box: `felhom-opsign -op agent_update -host -key-id felhom-op-1 -key > -agent-version -sha256 -upload http://:8080 -hub-key "$REPORT_API_KEY"`, and when > `configs/` changed also `-op agent_config_update -agent-version -bundle-sha256 ` (root files ride @@ -190,6 +191,9 @@ curl -fsSL -o /tmp/rt.iso https://iso.felhom.eu/felhom-installer--pve. > never appeared in the command. `%{url_effective}` and `%{referer}` do the same. Print > `%{http_code}` alone; read the destination from `-D-` headers if you need it. The verb is **`/discard`**, POST only (`hub/internal/web/server.go:345`); `/delete` 404s. +- **Put your SSH key on the proof host BEFORE you run the Felhom installer.** The installer closes the host on + purpose (new root password into the hub's vault, SSH by key only). With no key you are locked out and cannot read + the agent's config or logs (2026-11-11, golden 0.352.0's fresh install). - Venue: `demo-hp`, scratch `dir` storage at **`/mnt/nvme-1tb` root** (a subdirectory reads `disconnected` forever — the agent's `exactMount` check). Never `local-lvm`. Remove the storage at teardown.