Golden 0.352.0 vouched; token fix finished; why the N100 keeps no time; certificate pin spike on demo-hp
gates / gates (push) Successful in 3m34s

- Golden 0.352.0: bake, publish, one fresh install, the operator's vouch (R-468).
- No clone holds a login in its address; the credential helper script is in the repository (R-925, R-1390).
- N100 clock: a stale cached AppArmor profile stops chrony (R-1301, new R-1388). Read only.
- Certificate pin spike on demo-hp (R-1380): the break, the proven hand repair (runbook), the CA-file test,
  03-host-agent.md 12a, decision D80.
- Rulings of 13:17 written down (09 decisions 297-302). Register 245 -> 248 (R-1388, R-1389, R-1390).
- Docs, evidence and one helper script. No release, no floor change, no hub deploy.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-11 15:15:20 +01:00
1 parent 978666dde3
commit d4c7e84c22
74 files changed
+3376 -14

No files matched your search

@@ -0,0 +1,122 @@
# Report — the golden, the token fix, the N100's clock, the certificate pin (2026-11-11, afternoon)
Session start 2026-11-11T12:50:33Z. One STOP (the vouch). No release, no floor change, no hub deploy, no agent,
controller or wrapper code.
## 1. Claims in the prompt that turned out wrong
- **"Proxmox has a documented way to force a new certificate" — true, with one more step.** `pvecm updatecerts
--force` writes the new certificate, but the web service serves the old one until `systemctl reload pveproxy`.
Proxmox's own nightly renewal does that reload by itself.
- **"A setting may make the agent follow" — true, with two limits.** The key `proxmox.tls.ca_file` exists and works.
But a set fingerprint wins over it, and the agent's user cannot read Proxmox's CA file; the test used a copy.
- **"The fresh box of Part 1 can show if the clock problem is the product's" — wrong.** The cause needs a kernel that
went back and forth. A fresh box has no such history. It showed only that a new install keeps time.
- **The ruled fix shape of R-1380 ("the agent reads the certificate file on its own host")** cannot work as written:
the agent's user has no right to read that file. It needs a root step. Decision D80 is the result.
## 2. Five lines
1. **Golden 0.352.0 is baked, published, tried on one fresh box and vouched by the operator** (14:48:11 CET). The
gate is green (`python3 scripts/golden_currency_gate.py`, rc 0).
2. **No clone holds a login in its address** (16 clones read, 0 with one). 13 clones still use the clear-text file in
the home folder through the global `store` helper.
3. **The N100 keeps no time because the kernel loaded an old cached security profile for the time service.** One
hand step repairs it; it is in `STATUS.md`, not run.
4. **On demo-hp a new Proxmox certificate blinded the agent at once.** Guests and apps stayed up. The hand repair
took 2.1 s from the restart to a working call.
5. **With a CA file in place of the fingerprint the agent went through a second new certificate with no hand step.**
## 3. Parts
| Part | State | Where |
|---|---|---|
| 0 — the rulings of 13:17 (D76–D79, all yes) | done | `09-update-architecture.md` §3 decisions 297–302; the four rows left `STATUS.md` |
| 1 — golden: bake, publish, fresh install, vouch | done | `documentation/tests/golden-0.352.0-2026-11-11/README.md` |
| 2 — token fix: the two other clones, the rule line, the skill line | done | §4 below |
| 3 — why the N100 keeps no time | done (diagnosis only) | `documentation/audits/n100-clock-2026-11-11/README.md` |
| 4 — certificate spike on demo-hp | done | `documentation/audits/certpin-spike-2026-11-11/README.md` |
**Part 1, changed from the brief:** the hub offers only the vouched golden, so the fresh box ran the public installer
with one added line that names golden 0.352.0 and its sha256. All other steps were the product's own. **Not read:**
the agent's pin on the fresh box. CC put no SSH key on the host before the installer closed it. That was CC's mistake;
the lesson is now a line in the build skill.
## 4. The token fix, and what still depends on the clear-text file
Fixed in this run: `git/cc-secrets` and the plug-meter clone of `felhom-agent`. With the morning's fix that is three
clones; each uses its own helper (`scripts/git-credential-sec.sh`, now in the repository) and the store entry
`GITEA_GIT_TOKEN`. Measured after: 0 of 16 clones hold a credential in the address.
Still on the clear-text file (3 lines, mode 600, hosts `gitea.dooplex.hu` and `github.com`): the 13 other clones,
through the global `store` helper. **Proposal, nothing changed:** (1) give each of the 13 clones the same helper;
(2) put the `github.com` login into the store under its own name; (3) the operator then unsets the global helper and
removes the file. Step 3 is the operator's, because it removes a credential.
No token was made, rotated or deleted. A side effect, filed: `sec put` can no longer push the store by itself
(R-1390); the second command is in `STATUS.md`.
## 5. Versions, digests, CI
- Golden 0.352.0, sha256 `d76336054282ec138d8b0c8b9fec63319aa06f192aa421d012890b265b9abced`, 639,712,511 bytes;
round trip from the registry equal. Controller in it: 0.352.0 (MinAgent 0.158.1).
- Vouched manifest: agent 0.173.0 sha256 `752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea`
(unchanged), golden 0.352.0, min agent 0.158.1.
- Baselines at start, all equal to `origin/main`: felhom.eu `978666dde3`, felhom-controller `a9189c9dc4`,
felhom-agent `91a2924ad0`, app-catalog `f4c5c797d9`. Only felhom.eu has a commit from this run.
- CI of this commit: in the chat note, by hash.
## 6. Register
**Before 245, after 248, opened 3, closed 0.** Opened: R-1388 (stale security profile after a kernel went back and
forth), R-1389 (installer address 404 on the home network), R-1390 (`sec put` cannot push). Dated lines: R-468,
R-925, R-1301, R-1380, R-1384. One new decision: D80. Capability map: no row; no box proves a new capability.
## 7. Machines, and how each was left
| Machine | What changed | Left as |
|---|---|---|
| DooPlex | the bake and its registry copy; the address and helper settings of two clones; the files of this commit | bake guest torn down; nothing else |
| demo-hp host | Part 1: VM 9384 on `nvme-scratch`, an uploaded ISO, a helper script. Part 4: two forced certificates, three reloads of the web service, the agent's `proxmox.tls` block, eight agent restarts | VM, disks, ISO and script removed. Pin = served certificate. Agent active, 77 of 77 capabilities, calls answer 200, hub report at 14:02:53Z. The CA copy and the saved config are removed. **The node certificate is new** (ends 2028-11-09; before: 2028-08-19) |
| The N100 | nothing (reads only) | as found: its time service is still failed |
| Tester 1 | nothing (reads only) | agent active, guest running (read 14:04:07Z) |
| ep0 | nothing by CC. **The hub re-issued Tester4's backup token there during the fresh install** — the product's own act | the operator removes host `Tester4-d35551` at the hub |
| The hub | host `Tester4-d35551` and appliance `3fb5fed3-504c-4c2c-a63f-8108d5c15b6e` (the fresh box) | both wait for the operator |
**In Part 4 the agent on demo-hp was blind for 8 min 56 s (planned) and stopped for 92 s (not planned).** The second
was CC's mistake: too many restarts in 2 minutes met systemd's start limit. One operator mail went out
(`agent_channel_unknown`, 13:52:05Z), and its recovery event followed.
## 8. Documents read, by name
Workspace `CLAUDE.md`, `felhom.eu/CLAUDE.md`, `.claude/rules/live-probes.md`, the `felhom-build-deploy` skill,
`runbooks/publish-train-rules.md`, `RUNBOOK-manual-build.md` §4.2, `runbooks/nightly-rotation.md`, the golden-0.350.0
README, `cc-secrets/ARCHITECTURE.md`, the morning report §3. Architecture: `01-topology-and-trust.md`,
`03-host-agent.md`, `04-control-plane-authorization.md`, `05-hub-architecture.md`, `09-update-architecture.md`,
`11-os-updates.md`. Lists: Clock CA03, CA04, CA09; End dates DB02. **No architecture document named the agent's
Proxmox pin** (search of 01, 03, 04: 0 lines); it is now `03-host-agent.md` §12a.
## 9. Edits to instruction files and the poster
- Workspace `CLAUDE.md`, lines 126–127, and its tracked copy `documentation/runbooks/workspace-CLAUDE.md`. Before: no
line. After: "A repository address never holds a credential (2026-11-11, R-925). A clone that needs a login uses
its own credential helper (`felhom.eu/scripts/git-credential-sec.sh`) and `sec run GITEA_GIT_TOKEN -- git …`. Never
print a git address." Why: the brief's Part 2 asked for it. It adds a limit; it loosens none.
- `skills/felhom-build-deploy/SKILL.md`: two lines added (the felhom-agent clone needs `sec run GITEA_GIT_TOKEN`; put
your SSH key on a proof host before the installer). `check_skills.py` passes.
- System poster: no fact in its facts file changed. Nothing to do.
## 10. Observations
- The stale profile after a kernel went back and forth — FILED: R-1388.
- Tester 1's host has the older chrony — FILED: in R-1388.
- The installer address gives 404 on the home network — FILED: R-1389.
- `sec put` cannot push the store — FILED: R-1390.
- No alarm names the certificate cause; the hub reads ONLINE with no new report — FILED: R-1380 (dated line).
- The agent cannot read Proxmox's certificate files; the fingerprint wins over the CA file — FILED: R-1380 and
`03-host-agent.md` §12a.
- The agent unit allows 4 starts in 2 minutes — NOT-A-FINDING: the unit's design; the runbook now says "one restart".
- The pin scripts rewrote `agent.json` in a new layout — NOT-A-FINDING: the content is equal (`evidence/24`).
- The End-dates list still gives demo-hp's old end date (2028-08-19) — NOT-A-FINDING: a dated list; the spike README
holds the new date.
- `felhom-web-design` skill is not installed (a warning of `check_skills.py`) — NOT-A-FINDING: not from this run.
+3 -6
View File
@@ -2,7 +2,7 @@
**Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).**
**Updated 2026-11-11 (midday).** Two read-only lists are ready: every job on a timer (70) and everything that ends on a date (39); nothing ends in the next 14 days (`documentation/audits/end-dates-list-2026-11-11/README.md`). The golden passed its 7-day limit on 2026-11-10, so code pushes to `felhom.eu` are refused. Four decisions wait (D76–D79).
**Updated 2026-11-11 (afternoon).** Golden 0.352.0 is vouched, so code pushes to `felhom.eu` work again. The N100 keeps no time: one hand step repairs it (below). A new Proxmox certificate blinds the agent; the hand repair is proven on demo-hp and one decision waits (D80).
## What works
- **The Plex attack did not reach Felhom's secrets** (R-1241).
@@ -32,10 +32,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|---|---|---|---|---|---|---|
| D76 | R-1380 | Shall the agent follow Proxmox's new certificate by itself? | **Yes** (one agent change, long before 2028; no box then depends on a calendar) | a hand step in a runbook and a calendar date for each box, for ever | in July 2028 the N100's agent loses guest control and backups, with no alarm that says why; the other boxes follow | — |
| D77 | R-1379 | Shall the product warn you 14 days before an end date (a box's web certificate, a hub token)? | **Yes** (a renewal that fails is silent today) | you keep the dates in a calendar by hand | the list of 2026-11-11 is the only guard | — |
| D78 | R-1383 | Shall a timed job that gives up or is skipped tell you (four cases: the guest-power watchdog, a skipped hub daily check, a lost operator mail, a refused signed job)? | **Yes** (small changes in the agent and the hub) | the logs stay the only trace | you learn of a dead guest from the household | — |
| D79 | R-1384 | Shall the hub check each box's clock and alarm above one minute of error? | **Yes** (a box with a wrong clock can look healthy and cannot take an agent update) | R-1301 (repair the N100's time service) stays the only guard | the N100 stays 32 s behind, and nobody is told when it gets worse | — |
| D80 | R-1380 | How shall the agent trust Proxmox after its certificate changes: a copy of the box's own root CA, or a helper that follows the new fingerprint (your ruling of 11-11 13:17)? | **The root CA copy** (proven on demo-hp: the agent went through a new certificate with no hand step; no new privileged code) | follow the fingerprint: a root helper, a sudo rule and re-pin logic to write and test; trust stays on one exact certificate | nothing is built; the first box goes blind about 2028-07-02 | the fix of R-1380 |
| D66 | R-1365 | Should a security-watch source that stays down mail again every 7 days, as an open notice does? Today it mails once, then only the Monday summary lists it. | **Yes** (a real outage then cannot be forgotten; the seven pins that can never be read no longer count, so it stays quiet when all is well) | no: one line in a long Monday mail stays the only reminder | a source that stays down mails once; the Monday list still shows it | — |
| D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — |
| D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — |
@@ -43,7 +40,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
| D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — |
## Waiting on you — do
**2026-11-11.** 1. Bake and vouch a golden, or write a waiver (R-468). If not: no code push to `felhom.eu`. 2. Read the dates of „Only you can read these dates” (`documentation/audits/end-dates-list-2026-11-11/README.md`, R-1381). If not: a domain or a token can end with no warning. 3. Rotate the Gitea token of login `kisfenyo` into the store as `GITEA_GIT_TOKEN` (R-925); it was printed on 11-10. 4. Yes or no: may CC fix the two other clone addresses (`cc-secrets`, the plug-meter clone)? Only `felhom-agent` is fixed. 5. Search Console read (R-1035, now 11-18).
**2026-11-11.** 1. N100, as root, one time: `apparmor_parser -r -T -W /etc/apparmor.d/usr.sbin.chronyd && systemctl restart chrony && chronyc tracking` (R-1301; not tested). If not: about mid-January 2027 it refuses every agent update. 2. Hub Hosts page: delete host `Tester4-d35551`, Discard appliance `3fb5fed3…` (today's test box). If not: a dead entry alarms. 3. Rotate login `kisfenyo`'s Gitea token into the store as `GITEA_GIT_TOKEN` (R-925); the push after `sec put` is in R-1390. 4. Home DNS: remove the `felhom.eu` override (R-1389). 5. Read the dates only you can read (R-1381). 6. Search Console read (R-1035, 11-18).
**Still needed? (yes/no)** — answer per line.
1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). (since 10-16)
@@ -933,6 +933,45 @@ tunnel token, encrypted storage creds (NFS/CIFS/PBS), and the **live PBS key**.
and the secret footprint that left the controller now concentrate here — which is the whole
argument for §3's root-minimization and a small, auditable agent.
### 12a. How the agent trusts Proxmox's certificate, and what happens when it changes (R-1380)
`[FACT — measured on demo-hp 2026-11-11]` (`documentation/audits/certpin-spike-2026-11-11/README.md`)
- The agent calls Proxmox at `https://127.0.0.1:8006`. It trusts **one exact certificate**: the installer reads the
served certificate and writes its sha256 into `proxmox.tls.fingerprint` (`scripts/felhom-host-install.sh`). The
order in the agent is: skip-verify, then fingerprint, then `proxmox.tls.ca_file`, then the system's roots
(`felhom-agent/internal/proxmox/tls.go`). **A set fingerprint wins over a CA file.**
- Proxmox makes a new node certificate by itself (`pveupdate`, daily timer; `PVE/Cluster/Setup.pm`, pve-cluster
9.1.9): the certificate lasts at most 2 × 365 days and is renewed when it ends within 14 days. `pveupdate` then
reloads the web service, so the new certificate is served the same night.
- From that moment every agent call to Proxmox fails with `TLS pin mismatch`. Measured: no reconcile, no guest list,
no storage view, no host report to the hub; the controller gets 502 from the local API. Guests and apps stay up.
One operator mail names the symptom (`agent_channel_unknown`), none names the cause. The hand repair is
`runbooks/RUNBOOK-agent-proxmox-pin-repair.md` (2.1 s from restart to a working call).
- The agent's user **cannot read** `/etc/pve/pve-root-ca.pem` or the node certificate (`0640 root:www-data`).
- With the fingerprint empty and `ca_file` set to a readable copy of Proxmox's root CA, the agent connected, and it
went on through a second forced certificate with no restart and no hand step. A CA file that did not sign the
certificate was refused (`x509: certificate signed by unknown authority`).
- Proxmox's root CA lasts 3650 days. Its code renews it when less than two years are left, **with the same key and
the same name**; a node certificate is not made from a CA that ends in less than 14 days. Read from the code, not
forced.
`[DESIGN — proposed, not built]` Two ways. The operator's ruling of 2026-11-11 13:17 (`09` §3 decision 297) names the
first; the spike points to the second, so the choice is back with the operator (`STATUS.md` D80).
| | A — the agent follows the fingerprint | B — the agent trusts the box's own root CA (**CC's pick**) |
|---|---|---|
| How | A root helper reads the node certificate after a mismatch and gives the agent the new fingerprint | The installer puts a copy of `/etc/pve/pve-root-ca.pem` where the agent can read it, sets `ca_file`, leaves the fingerprint empty |
| New agent code | yes: the helper, a sudo rule, the re-pin logic, its tests | none for the trust itself (the key exists and is proven) |
| Trust | one exact certificate | every certificate this box's own CA signs; the CA's key is root-only on the same box |
| Next hand step | none, if the helper is right | none until the copy's own end date (10 years from install); the copy must be refreshed after Proxmox renews the CA |
| Boxes that exist | one root step each (the signed config bundle) | the same |
Why B: it is the standard way to trust a private CA, it is proven on a box, and it removes the failure with no new
privileged code. What B still needs before it is built: the root step that refreshes the copy (and a test that an
old copy raises an alarm that names the cause), and a rule for what the agent does if both keys are set.
Whichever way is built, the mismatch must raise an alarm that names the cause.
## 13. Open items / what this unblocks
Resolved here: tunnel placement (host, agent-managed, own systemd service), the
@@ -1265,6 +1265,28 @@ Recorded before the work (`audits/space-list-2026-11-10/README.md` §5 held the
Hetzner and the mail service are read from source and documents only: no hub database read, no account call.
*Operator, 2026-11-10 12:22.*
### 2026-11-11 (13:17, operator, planning chat) — four rulings on the End-dates and Clock lists; the clones; the gap review pauses
Recorded before any build. **None of these is built.** The questions were D76–D79 in `STATUS.md`
(`audits/end-dates-list-2026-11-11/README.md`, `audits/clock-list-2026-11-11/README.md`).
297. **D76 = yes: the agent follows Proxmox's new certificate by itself** — after the failure is seen on demo-hp
(`audits/certpin-spike-2026-11-11/README.md`). **Cost of the other option:** a hand step in a runbook and a
calendar date for each box, for ever. The two ways to build it are in `03-host-agent.md` (the part on the Proxmox
certificate pin). R-1380. *Operator, 2026-11-11 13:17.*
298. **D77 = yes: the product warns 14 days before an end date** (a box's web certificate, a hub token). **Cost of the
other option:** the dates stay in a calendar kept by hand. R-1379. *Operator, 2026-11-11 13:17.*
299. **D78 = yes: a timed job that gives up or is skipped tells the operator — the four cases:** the guest-power
watchdog gave up; a hub daily check was skipped; an operator mail was lost; a signed job was refused. **Cost of
the other option:** the logs stay the only trace. R-1383. *Operator, 2026-11-11 13:17.*
300. **D79 = yes: the hub checks each box's clock and alarms above one minute of error.** **Cost of the other option:**
the repair of one box's time service (R-1301) stays the only guard. R-1384. *Operator, 2026-11-11 13:17.*
301. **The two other clone addresses are fixed the same way as `felhom-agent`'s** (no login in a repository address;
`audits/no-token-in-address-2026-11-11/`). **The token rotation is the operator's and comes later; CC does not ask
again.** R-925. *Operator, 2026-11-11 13:17.*
302. **The gap review pauses after four lists** (crossings, endings, end dates, clock): the operator chose „build”.
The lists „outside services”, „sizes” and „fleet” are not run. *Operator, 2026-11-11 13:17.*
### 2026-11-10 (15:19, operator, planning chat) — five rulings on the Endings list and the system disk
Recorded before any build. **None of these is built.** The questions were D71–D75 in `STATUS.md`
@@ -0,0 +1,68 @@
# The Proxmox certificate changes and the agent's pin does not follow — spike on demo-hp, 2026-11-11
Row R-1380. Box: demo-hp only. No agent code, no release. Files: `evidence/` (26 files), `tools/` (the scripts).
Documents read: `architecture/01-topology-and-trust.md`, `03-host-agent.md`, `04-control-plane-authorization.md`;
the End-dates list entry DB02 and its `evidence/box/`. **No architecture document named this pin before today**
(a search of 01, 03 and 04 found no line; now `03-host-agent.md` §12a).
## What was done, in order (times in UTC)
| Time | Step | Result | File |
|---|---|---|---|
| 13:49:16 | Before | served certificate = file = agent's pin (`07:B5:72:5D…07:BB`, ends 2028-08-19); agent calls answer 200 | `01`, `05` |
| — | Who else trusts it | port 8006's log: only the agent, from 127.0.0.1 (75,374 lines). No file outside the box holds the fingerprint | `02` |
| 13:49:23 | `pvecm updatecerts --force` | new file (`21:D6:B5:E0…84:12`); **Proxmox still serves the old one, the agent still works** | `06`, `07` |
| 13:50:23 | `systemctl reload pveproxy` | the new certificate is served about 10 s later | `08` |
| 13:50:39 | The agent | first `TLS pin mismatch` line; every call to Proxmox fails from here | `09`, `10` |
| 13:59:25 | Hand repair | new fingerprint into `agent.json`, restart: **a working call 2.1 s later** | `16`, `17` |
| 13:59:39 | CA file in place of the fingerprint | the agent connects (2.2 s) | `18`, `19` |
| 13:59:54 | Second forced certificate (`1F:75:D3:51…0E:74`) + reload | **the agent goes on, no restart, no hand step, 0 TLS lines** | `20` |
| 14:01:02 | Control: a CA file that did not sign it | `x509: certificate signed by unknown authority` (so the CA file is really checked) | `25` |
| 14:02:46 | Back to the standard | pin = served certificate; 77 of 77 capabilities; hub report at 14:02:53 | `22`, `24`, `26` |
## What broke while the pin was wrong (8 min 56 s, by choice)
- **The agent's line:** `proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint`, as a
WARN in each loop. It does not say "the certificate changed" and does not say what to do.
- **Failed:** the guest list, the storage view, the guest config, the pool read, the node status. So: reconcile
(`pass failed`), the controller supervisor, the guest-power watchdog, the guest network sweep, the stale-lock
recovery, and the local API's `/storage` and `/disks` (502 to the controller). From the code, not tried: backup,
restore, snapshot, start/stop, resize all use the same client, so they fail too.
- **The hub:** the agent sent **no host report** (`hub: collect failed; skipping this cycle's report`). The Hosts page
still read `ONLINE 1/2` eleven minutes after the last report. Not measured: when it turns to offline.
- **The alarm:** one mail to the operator after 94 s, `agent_channel_unknown (warning) — Controller→agent channel
down: agentapi: GET /storage: HTTP 502`. It names a symptom, not the cause. A recovery event came 39 s after the
repair.
- **Stayed up:** both guests, Tester 1's VM, all 26 containers in the guest, the enrolled drive.
## Three facts the fix must know
1. **Proxmox's own renewal reloads the web service by itself** (`pveupdate`, daily timer): node certificate at most
2 × 365 days, renewed when it ends within 14 days. So on a real box the break comes the same night (`03`).
2. **The agent cannot read Proxmox's certificate files.** It runs as user `felhom-agent`; `/etc/pve/pve-root-ca.pem`
and the node certificate are `0640 root:www-data`. The CA test used a copy in `/etc/felhom-agent/` (`18`). The
ruled fix shape ("the agent reads the certificate file on its own host") needs a root step for the same reason.
3. **The fingerprint wins.** Config key `proxmox.tls.ca_file` exists, but it is used only when
`proxmox.tls.fingerprint` is empty (`felhom-agent/internal/proxmox/tls.go`).
The root CA lasts 10 years (demo-hp: to 2036-08-17). Proxmox renews it when less than two years are left (about
August 2034) and **keeps its key and its name**, so new node certificates still pass a copy of the old CA file until
that copy's own end date. Read from Proxmox's code (`03`), not forced.
## The two ways, and the pick → `03-host-agent.md` §12a, decision D80 in `STATUS.md`
**CA file** (CC's pick): proven here, no agent code. **Follow the fingerprint**: needs new agent code and a root
helper. Both need one root step to reach the boxes that exist.
## CC's mistakes in this run
- The first control ran under 1 s (the waiter matched an older call): void. The repeat met systemd's start limit
(4 starts in 2 minutes): **the agent was stopped for 92 s** (14:01:14 to 14:02:46). The failure hook ran and did
nothing ("no pending update"). The proof for the control comes from the 1-second run's own log (`23`, `24`, `25`).
- The scripts rewrote `agent.json` with a new layout. The content is equal to the saved copy (`24`: `rest equal: True`).
## How demo-hp was left
Fingerprint pin equal to the served certificate (`1F:75:D3:51…0E:74`, ends 2028-11-09). Agent active, 0 restarts,
calls answer 200. The CA copy and the saved config are removed. **The certificate itself is new**: that is the one
lasting change, and it moves demo-hp's end date from 2028-08-19 to 2028-11-09.
@@ -0,0 +1,187 @@
== demo-hp 2026-11-11T13:26:29Z pve pve-manager/9.2.26/41a0e2e09b9e6f9d (running kernel: 7.0.14-28-pve)
== served certificate (network read, 127.0.0.1:8006)
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Aug 20 15:44:42 2026 GMT
notAfter=Aug 19 15:44:42 2028 GMT
sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB
X509v3 Subject Alternative Name:
IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1, DNS:localhost, IP Address:192.168.0.104, DNS:demo-hp, DNS:demo-hp.local
== files
-rw------- 1 root www-data 3272 2026-08-21 17:44:42.000000000 +0200 /etc/pve/priv/pve-root-ca.key
-rw-r----- 1 root www-data 2094 2026-08-21 17:44:42.000000000 +0200 /etc/pve/pve-root-ca.pem
/etc/pve/local/:
total 2
drwxr-xr-x 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 .
drwxr-xr-x 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 ..
-rw-r----- 1 root www-data 83 2026-11-11 14:26:26.000000000 +0100 lrm_status
drwxr-xr-x 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 lxc
drwxr-xr-x 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 openvz
drwx------ 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 priv
-rw-r----- 1 root www-data 1704 2026-08-21 17:44:42.000000000 +0200 pve-ssl.key
-rw-r----- 1 root www-data 1785 2026-08-21 17:44:42.000000000 +0200 pve-ssl.pem
drwxr-xr-x 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 qemu-server
-rw-r----- 1 root www-data 560 2026-11-10 15:43:03.000000000 +0100 ssh_known_hosts
-- /etc/pve/local/pve-ssl.pem
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Aug 20 15:44:42 2026 GMT
notAfter=Aug 19 15:44:42 2028 GMT
sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB
-- /etc/pve/pve-root-ca.pem
subject=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Aug 20 15:44:42 2026 GMT
notAfter=Aug 17 15:44:42 2036 GMT
sha256 Fingerprint=E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7
ls: cannot access '/etc/pve/local/pveproxy-ssl.pem': No such file or directory
== agent
felhom-agent 0.173.0
active
Restart=on-failure
NRestarts=0
ActiveEnterTimestamp=Tue 2026-11-10 15:58:17 CET
proxmox keys: ['endpoint', 'node', 'tls', 'token']
endpoint : https://127.0.0.1:8006
node : demo-hp
token : <present>
tls: {'fingerprint': '07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB', 'insecure_skip_verify': False}
total 28
drwxr-xr-x 2 root root 4096 2026-09-28 16:25:22.412907166 +0200 .
drwxr-xr-x 102 root root 4096 2026-11-08 02:49:07.920613811 +0100 ..
-rw------- 1 felhom-agent felhom-agent 2454 2026-09-28 16:25:22.443523132 +0200 agent.json
-rw------- 1 felhom-agent felhom-agent 2498 2026-09-24 15:33:45.307471226 +0200 agent.json.night-0925-off
-rw------- 1 felhom-agent felhom-agent 2451 2026-09-24 21:56:34.103084862 +0200 agent.json.pre-a4-retention
-rw------- 1 felhom-agent felhom-agent 2451 2026-09-24 21:57:43.930322856 +0200 agent.json.pre-d44
-rw------- 1 root root 2451 2026-08-21 18:09:15.470472229 +0200 agent.json.pre-r672
8:# allowlist in /etc/sudoers.d/felhom-agent (configs/felhom-agent.sudoers). The User= here and the
39:User=felhom-agent
41:ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json
== agent log, last proxmox lines
4422
pam_unix(sudo:session): session closed for user root
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- cat /etc/network/interfaces
pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
pam_unix(sudo:session): session closed for user root
pam_unix(sudo:session): session closed for user root
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- pgrep -x dhclient
pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
pam_unix(sudo:session): session closed for user root
== who calls port 8006 (pveproxy access log, client addresses, last 2 days)
75375 ::ffff:127.0.0.1
-- users seen
75374 ::ffff:127.0.0.1 felhom-agent@pve!agent
1 ::ffff:127.0.0.1 -
== listening / other TLS users of this cert
LISTEN 0 4096 127.0.0.1:85 0.0.0.0:* users:(("pvedaemon worke",pid=235161,fd=6),("pvedaemon worke",pid=168847,fd=6),("pvedaemon w
LISTEN 0 4096 *:8006 *:* users:(("pveproxy worker",pid=316496,fd=6),("pveproxy worker",pid=308404,fd=6),("pveproxy wo
LISTEN 0 4096 *:3128 *:* users:(("spiceproxy work",pid=1733482,fd=6),("spiceproxy",pid=1399,fd=6))
== in flight
running tasks: []
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
341 night1004-tester1 running 8192 200.00 1950
9384 golden1111-box running 10240 64.00 313884
VMID Status Lock Name
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
== tools' own help
USAGE: pvecm updatecerts [OPTIONS]
Update node certificates (and generate all needed files/directories).
--force <boolean>
Force generation of new SSL certificate.
--silent <boolean>
Ignore errors (i.e. when cluster has no quorum).
--unmerge-known-hosts <boolean> (default=0)
Unmerge legacy SSH known hosts.
USAGE: pvenode cert delete [<restart>]
DELETE custom certificate chain and key.
<restart> <boolean> (default=0)
Restart pveproxy.
USAGE: pvenode cert info [FORMAT_OPTIONS]
Get information about node's certificates.
USAGE: pvenode cert set <certificates> [<key>] [OPTIONS] [FORMAT_OPTIONS]
Upload or update custom certificate chain and key.
<certificates> <string>
PEM encoded certificate (chain).
<key> <string>
PEM encoded private key.
--force <boolean> (default=0)
Overwrite existing custom or ACME certificate files.
--restart <boolean> (default=0)
Restart pveproxy.
== renew rule in code
83:my $pveca_cert_days = 3650;
85:my $pvessl_cert_max_days = 2 * 365;
86:my $pvessl_cert_days_before_ca = 2;
438: '-days',
439: $pveca_cert_days,
478:# certificate chain up to the renewed one, and clients trusting the previous one accept new node
479:# certificates until it expires. Returns the expiry of the previous one if it got renewed.
480:sub renew_pveca_cert {
483: my $renewal_reason = sub {
484: my $min_days = $pvessl_cert_max_days + $pvessl_cert_days_before_ca;
485: if (PVE::Certificate::check_expiry($pveca_cert_fn, time() + $min_days * 24 * 60 * 60)) {
486: return 'expires within two years';
501: my $reason = $renewal_reason->() or return undef;
503: my $old_expiry = PVE::Cluster::cfs_lock_domain(
507: return undef if !$renewal_reason->(); # another node renewed it in the meantime
519: '-days', $pveca_cert_days,
528: die "subject of renewed certificate differs\n";
535: die "renewing pve root certificate failed:\n$err";
543: if ($old_expiry) {
544: my $deadline = localtime($old_expiry);
545: print "Renewed cluster CA certificate, as the previous one $reason. Clients that trust"
546: . " the previous one must import the renewed $pveca_cert_fn before $deadline\n";
549: return $old_expiry;
579:# and its expiry. A certificate from another issuer was placed there by an admin and must be kept.
685: # get ca expiry
687: my $daysleft = int(($cainfo->{notafter} - time()) / (24 * 60 * 60));
689: if ($daysleft < 14) {
690: die "CA expires in less than 2 weeks, unable to generate certificate.\n";
693: $daysleft -= $pvessl_cert_days_before_ca;
695: if ($daysleft >= $pvessl_cert_max_days) {
/usr/bin/pveupdate:86: my $lead_time = PVE::Certificate::get_cert_renew_lead_time($cert);
/usr/bin/pveupdate:88: if (PVE::Certificate::check_expiry($cert, time() + $lead_time)) {
/usr/bin/pveupdate:89: PVE::API2::ACME->renew_certificate({ node => $nodename });
/usr/bin/pveupdate:93: 'Custom certificate does not expire soon, skipping ACME renewal.',
/usr/bin/pveupdate:99: 'ACME config found for node, but no custom certificate exists. Skipping ACME renewal until initial certificate has been deployed.',
/usr/bin/pveupdate:104:syslog('err', "Renewing ACME certificate failed: $@") if $@;
/usr/bin/pveupdate:106:eval { PVE::Cluster::Setup::renew_pveca_cert() };
/usr/bin/pveupdate:107:syslog('err', "Renewing cluster CA certificate failed: $@") if $@;
/usr/bin/pveupdate:131: my $renew = sub {
/usr/bin/pveupdate:142: print "Renewed PVE certificate is valid until $expiry\n";
/usr/bin/pveupdate:144: print "Restarting pveproxy after renewing certificate\n";
/usr/bin/pveupdate:152: $renew->("was issued by a replaced cluster CA, renewing...", 1);
/usr/bin/pveupdate:153: } elsif (PVE::Certificate::check_expiry($certpath)) {
/usr/bin/pveupdate:155: $renew->("expired, renewing...");
/usr/bin/pveupdate:156: } elsif (PVE::Certificate::check_expiry($certpath, time() + 14 * 24 * 60 * 60)) {
/usr/bin/pveupdate:158: $renew->("expires soon, renewing...");
/usr/bin/pveupdate:159: } elsif (!PVE::Certificate::check_expiry($certpath, time() + 2 * 365 * 24 * 60 * 60)) {
/usr/bin/pveupdate:161: $renew->(
/usr/bin/pveupdate:162: "expires in more than 2 years, renewing to reduce certificate life-span for client compatibility..."
/usr/bin/pveupdate:166:syslog('err', "Checking/Renewing SSL certificate failed: $@") if $@;
uid=999(felhom-agent) gid=990(felhom-agent) groups=990(felhom-agent),999(systemd-journal)
agent CANNOT read the root CA file
agent CANNOT read the leaf file
755 root:www-data /etc/pve
640 root:www-data /etc/pve/pve-root-ca.pem
@@ -0,0 +1,20 @@
== who else holds demo-hp's fingerprint? 2026-11-11T13:26:49Z
-- workspace files (colon form, plain form), names only, audits' evidence left out
(end of list)
-- positive control: the same search with audits in
felhom.eu/documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-hp-host-dates.txt
felhom.eu/documentation/audits/evidence-fixes-first-tester-2026-09-30/partC/agent-0138-live.txt
-- hub code: does the hub store or use a box's Proxmox fingerprint?
felhom.eu/hub/internal/osupdates/service.go:704: if rel, _ := s.Store.LatestOSRelease(LayerPVE); rel != nil && rel.Fingerprint == fp {
(end)
-- agent: where the pin is written
felhom-agent/internal/localapi/cert.go:29:// is absent. It returns the tls.Certificate to serve and the leaf's SHA-256 fingerprint (the
felhom-agent/internal/localapi/cert.go:38:func EnsureLeaf(certPath, keyPath, bridgeHost string) (cert tls.Certificate, fingerprint string, generated bool, err error) {
felhom-agent/internal/config/config.go:916: if v := os.Getenv("FELHOM_AGENT_PROXMOX_TLS_FINGERPRINT"); v != "" {
felhom-agent/internal/config/config.go:917: cfg.Proxmox.TLS.Fingerprint = v
felhom-agent/internal/proxmox/tls.go:55: return fmt.Errorf("proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint")
felhom-agent/cmd/felhom-agent/main.go:340: Fingerprint: cfg.Proxmox.TLS.Fingerprint,
felhom.eu/scripts/felhom-host-install.sh:3182: # TLS pin: the SERVED leaf cert fingerprint (not pvesh node info — may differ)
felhom.eu/scripts/felhom-host-install.sh:3185: [[ -n "$fp" ]] || log_warn " could not compute TLS fingerprint (leaving empty — agent will use system trust)"
felhom.eu/scripts/felhom-host-install.sh:3206: log_dry "write $AGENT_CONFIG (0600): proxmox{endpoint,node=$NODE,token=<secret>,tls.fingerprint=$fp} hub{url=$HUB_URL,host_id=$HOST_ID,a
felhom.eu/scripts/felhom-host-install.sh:3335: "tls": {"fingerprint": os.environ['FP'], "insecure_skip_verify": False},
@@ -0,0 +1,318 @@
2026-11-11T13:27:23Z
libpve-cluster-perl 9.1.9
pve-cluster 9.1.9
== Setup.pm 470-560
gen_pveca_key();
$generate_pveca_cert->($pveca_key_fn, $pveca_cert_fn);
return 1;
}
# Keeping the key and the exact encoded subject lets node certificates issued by the previous CA
# certificate chain up to the renewed one, and clients trusting the previous one accept new node
# certificates until it expires. Returns the expiry of the previous one if it got renewed.
sub renew_pveca_cert {
return undef if !-f $pveca_key_fn || !-f $pveca_cert_fn;
my $renewal_reason = sub {
my $min_days = $pvessl_cert_max_days + $pvessl_cert_days_before_ca;
if (PVE::Certificate::check_expiry($pveca_cert_fn, time() + $min_days * 24 * 60 * 60)) {
return 'expires within two years';
}
my $key_usage;
my $cmd = ['openssl', 'x509', '-noout', '-ext', 'keyUsage', '-in', $pveca_cert_fn];
PVE::Tools::run_command(
$cmd,
outfunc => sub { $key_usage = 1 },
errfunc => sub { },
timeout => $pveca_lock_cmd_timeout,
);
return 'lacks the keyUsage extension that strict TLS clients require' if !$key_usage;
return undef;
};
my $reason = $renewal_reason->() or return undef;
my $old_expiry = PVE::Cluster::cfs_lock_domain(
'pve-root-ca',
undef,
sub {
return undef if !$renewal_reason->(); # another node renewed it in the meantime
my $old = PVE::Certificate::get_certificate_info($pveca_cert_fn);
my $tmp_fn = "$pveca_cert_fn.tmp.$$";
my $cmd = [
[
'openssl', 'x509', '-x509toreq', # keeps the exact encoded subject
'-in', $pveca_cert_fn, '-key', $pveca_key_fn,
],
[
'faketime', 'yesterday', # NOTE: wrap openssl with faketime to prevent bug #904
'openssl', 'req', '-x509', '-in', '/dev/stdin', '-key', $pveca_key_fn,
'-days', $pveca_cert_days,
'-addext', 'keyUsage=critical,keyCertSign,cRLSign',
'-out', $tmp_fn,
],
];
eval {
run_silent_cmd($cmd, timeout => $pveca_lock_cmd_timeout);
my $new = PVE::Certificate::get_certificate_info($tmp_fn);
if ($new->{subject} ne $old->{subject}) {
die "subject of renewed certificate differs\n";
}
rename($tmp_fn, $pveca_cert_fn)
or die "replacing '$pveca_cert_fn' failed - $!\n";
};
if (my $err = $@) {
unlink $tmp_fn;
die "renewing pve root certificate failed:\n$err";
}
return $old->{notafter};
},
);
die $@ if $@;
if ($old_expiry) {
my $deadline = localtime($old_expiry);
print "Renewed cluster CA certificate, as the previous one $reason. Clients that trust"
. " the previous one must import the renewed $pveca_cert_fn before $deadline\n";
}
return $old_expiry;
}
sub rotate_pveca {
my $tmp_key_fn = "$pveca_key_fn.tmp.$$";
my $tmp_cert_fn = "$pveca_cert_fn.tmp.$$";
eval {
$generate_pveca_key->($tmp_key_fn);
$generate_pveca_cert->($tmp_key_fn, $tmp_cert_fn);
PVE::Cluster::cfs_lock_domain(
'pve-root-ca',
== Setup.pm 575-720
}
}
# Returns who issued the node certificate, the 'current-ca', a 'replaced-ca' or an 'other' issuer,
# and its expiry. A certificate from another issuer was placed there by an admin and must be kept.
sub pve_ssl_cert_status {
my ($nodename) = @_;
my $pvessl_cert_fn = "$pmxcfs_base_dir/nodes/$nodename/pve-ssl.pem";
return undef if !-f $pvessl_cert_fn;
my $cert = PVE::Certificate::get_certificate_info($pvessl_cert_fn);
my $issuer = $cert->{issuer};
my $issued_by = 'other';
if ($issuer eq PVE::Certificate::get_certificate_info($pveca_cert_fn)->{subject}) {
$issued_by = 'current-ca';
} elsif ($issuer =~ m!^/CN=Proxmox Virtual Environment/OU=[^/]+/O=PVE Cluster Manager CA$!) {
$issued_by = 'replaced-ca';
}
return { 'issued-by' => $issued_by, notafter => $cert->{notafter} };
}
sub gen_pve_ssl_key {
my ($nodename) = @_;
die "no node name specified" if !$nodename;
my $pvessl_key_fn = "$pmxcfs_base_dir/nodes/$nodename/pve-ssl.key";
return if -f $pvessl_key_fn;
eval { run_silent_cmd(['openssl', 'genrsa', '-out', $pvessl_key_fn, '2048']); };
die "unable to generate pve ssl key for node '$nodename':\n$@" if $@;
}
sub gen_pve_www_key {
return if -f $pvewww_key_fn;
eval { run_silent_cmd(['openssl', 'genrsa', '-out', $pvewww_key_fn, '2048']); };
die "unable to generate pve www key:\n$@" if $@;
}
sub update_serial {
my ($serial) = @_;
PVE::Tools::file_set_contents($pveca_srl_fn, $serial);
}
sub gen_pve_ssl_cert {
my ($force, $nodename, $ip) = @_;
die "no node name specified" if !$nodename;
die "no IP specified" if !$ip;
my $pvessl_cert_fn = "$pmxcfs_base_dir/nodes/$nodename/pve-ssl.pem";
return if !$force && -f $pvessl_cert_fn;
my $names = "IP:127.0.0.1,IP:::1,DNS:localhost,IP:${ip},DNS:${nodename}";
my $fqdn = $nodename;
my $rc = PVE::INotify::read_file('resolvconf');
if ($rc && $rc->{search}) {
$fqdn .= ".$rc->{search}";
$names .= ",DNS:$fqdn";
}
my $reqfn = "/run/pve-cluster/pvecertreq-$$.tmp";
unlink $reqfn;
my $pvessl_key_fn = "$pmxcfs_base_dir/nodes/$nodename/pve-ssl.key";
eval {
run_silent_cmd([
'openssl',
'req',
'-batch',
'-new',
'-key',
$pvessl_key_fn,
'-out',
$reqfn,
'-subj',
"/OU=PVE Cluster Node/O=Proxmox Virtual Environment/CN=$fqdn",
'-addext',
'basicConstraints=CA:FALSE',
'-addext',
'extendedKeyUsage=serverAuth',
'-addext',
"subjectAltName=$names",
]);
};
if (my $err = $@) {
unlink $reqfn;
die "unable to generate pve certificate request:\n$err";
}
# the lock also covers the serial file that all nodes share
PVE::Cluster::cfs_lock_domain(
'pve-root-ca',
undef,
sub {
update_serial("0000000000000000") if !-f $pveca_srl_fn;
# get ca expiry
my $cainfo = PVE::Certificate::get_certificate_info($pveca_cert_fn);
my $daysleft = int(($cainfo->{notafter} - time()) / (24 * 60 * 60));
if ($daysleft < 14) {
die "CA expires in less than 2 weeks, unable to generate certificate.\n";
}
$daysleft -= $pvessl_cert_days_before_ca;
if ($daysleft >= $pvessl_cert_max_days) {
$daysleft = $pvessl_cert_max_days;
}
my $cmd = [
'faketime', 'yesterday', # NOTE: wrap openssl with faketime to prevent bug #904
'openssl', 'x509', '-req', '-in', $reqfn, '-days', $daysleft, '-out',
$pvessl_cert_fn,
'-CAkey', $pveca_key_fn, '-CA', $pveca_cert_fn, '-CAserial', $pveca_srl_fn,
'-copy_extensions', 'copyall',
];
run_silent_cmd($cmd, timeout => $pveca_lock_cmd_timeout);
},
);
my $err = $@;
unlink $reqfn or $!{ENOENT} or warn "failed to clean up '$reqfn' - $!";
die "unable to generate pve ssl certificate:\n$err" if $err;
}
sub gen_pve_node_files {
my ($nodename, $ip, $opt_force) = @_;
gen_local_dirs($nodename);
== pveupdate 100-170
);
}
}
};
syslog('err', "Renewing ACME certificate failed: $@") if $@;
eval { PVE::Cluster::Setup::renew_pveca_cert() };
syslog('err', "Renewing cluster CA certificate failed: $@") if $@;
eval {
my $certpath = PVE::CertHelpers::default_cert_path_prefix($nodename) . ".pem";
my $capath = "/etc/pve/pve-root-ca.pem";
my $check_issued_by_ca = sub {
# get CA info
my $cainfo = PVE::Certificate::get_certificate_info($capath);
# get cert and check issuer and chain metadata
my $certinfo = PVE::Certificate::get_certificate_info($certpath);
if ($certinfo->{issuer} ne $cainfo->{subject}) {
die "SSL certificate ($certpath) is not issued by root CA ($capath)!\n";
}
# check if cert is really signed by the ca
# TODO: replace by low level ssleay interface if version 1.86 is available
my $cmd = [
'/usr/bin/openssl', 'verify', '-no_check_time', '-CAfile', $capath, '--', $certpath,
];
PVE::Tools::run_command($cmd);
};
my $renew = sub {
my ($msg, $replaced_ca) = @_;
# one from a replaced cluster CA fails these checks, but is still ours to replace
$check_issued_by_ca->() if !$replaced_ca;
print "PVE certificate $msg\n";
# create new certificate
my $ip = PVE::Cluster::remote_node_ip($nodename);
PVE::Cluster::Setup::gen_pve_ssl_cert(1, $nodename, $ip);
my $expiry = localtime(PVE::Certificate::get_expiration_as_epoch($certpath));
print "Renewed PVE certificate is valid until $expiry\n";
print "Restarting pveproxy after renewing certificate\n";
PVE::Tools::run_command(['systemctl', 'reload-or-restart', 'pveproxy']);
};
my $status = eval { PVE::Cluster::Setup::pve_ssl_cert_status($nodename) };
syslog('err', "Checking SSL certificate failed: $@") if $@;
if ($status && $status->{'issued-by'} eq 'replaced-ca') {
$renew->("was issued by a replaced cluster CA, renewing...", 1);
} elsif (PVE::Certificate::check_expiry($certpath)) {
# already expired
$renew->("expired, renewing...");
} elsif (PVE::Certificate::check_expiry($certpath, time() + 14 * 24 * 60 * 60)) {
# expires in next 2 weeks
$renew->("expires soon, renewing...");
} elsif (!PVE::Certificate::check_expiry($certpath, time() + 2 * 365 * 24 * 60 * 60)) {
# expires in more than 2 years
$renew->(
"expires in more than 2 years, renewing to reduce certificate life-span for client compatibility..."
);
}
};
syslog('err', "Checking/Renewing SSL certificate failed: $@") if $@;
sub cleanup_tasks {
my $taskdir = "/var/log/pve/tasks";
== timer
NEXT LEFT LAST PASSED UNIT ACTIVATES
Thu 2026-11-12 01:46:13 CET 11h Wed 2026-11-11 02:22:58 CET 12h ago pve-daily-update.timer pve-daily-update.service
@@ -0,0 +1,7 @@
== preconditions 2026-11-11T13:49:13Z
demo-hp running tasks: []
(end of process list)
config saved on the box (0600, stays on the box)
57f9650d1ca60fcc
tester1 running tasks: []
(end)
@@ -0,0 +1,20 @@
== state 2026-11-11T13:49:16Z
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Aug 20 15:44:42 2026 GMT
notAfter=Aug 19 15:44:42 2028 GMT
sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB
agent tls config: {'fingerprint': '07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB', 'insecure_skip_verify': False}
agent: active since Tue 2026-11-10 15:58:17 CET restarts=0
-- agent calls in pveproxy's access log, last 4 (time, call, answer code)
[11/11/2026:14:49:07 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
[11/11/2026:14:49:07 +0100] "GET /api2/json/storage 200
[11/11/2026:14:49:08 +0100] "GET /api2/json/nodes/demo-hp/storage 200
[11/11/2026:14:49:08 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6)
0
-- guests
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
341 night1004-tester1 running
@@ -0,0 +1,10 @@
== force 2026-11-11T13:49:23Z
before: sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB
(re)generate node files
generate new node certificate
merge authorized SSH keys
updatecerts rc=0
file now: sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12 notAfter=Nov 9 13:49:24 2028 GMT
served 5 s later, no restart: sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB
pveproxy: Tue 2026-11-10 15:43:04 CET reloads:
done 2026-11-11T13:49:29Z
@@ -0,0 +1,20 @@
== state 2026-11-11T13:50:14Z
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Aug 20 15:44:42 2026 GMT
notAfter=Aug 19 15:44:42 2028 GMT
sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB
agent tls config: {'fingerprint': '07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB', 'insecure_skip_verify': False}
agent: active since Tue 2026-11-10 15:58:17 CET restarts=0
-- agent calls in pveproxy's access log, last 4 (time, call, answer code)
[11/11/2026:14:50:07 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
[11/11/2026:14:50:07 +0100] "GET /api2/json/storage 200
[11/11/2026:14:50:08 +0100] "GET /api2/json/nodes/demo-hp/storage 200
[11/11/2026:14:50:08 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6)
0
-- guests
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
341 night1004-tester1 running
@@ -0,0 +1,24 @@
== the step Proxmox's own nightly job does after a renewal: systemctl reload-or-restart pveproxy; 2026-11-11T13:50:23Z
reload rc=0
served: sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB notAfter=Aug 19 15:44:42 2028 GMT
2026-11-11T13:50:28Z
2026-11-11T13:50:34Z
served now: sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12
2026-11-11T14:50:24+01:00 demo-hp systemd[1]: Reloaded pveproxy.service - PVE API Proxy Server.
2026-11-11T14:50:25+01:00 demo-hp pveproxy[1293]: restarting server
2026-11-11T14:50:25+01:00 demo-hp pveproxy[1293]: starting 3 worker(s)
2026-11-11T14:50:25+01:00 demo-hp pveproxy[1293]: worker 494020 started
2026-11-11T14:50:25+01:00 demo-hp pveproxy[1293]: worker 494021 started
2026-11-11T14:50:25+01:00 demo-hp pveproxy[1293]: worker 494022 started
2026-11-11T14:50:30+01:00 demo-hp pveproxy[316496]: worker exit
2026-11-11T14:50:30+01:00 demo-hp pveproxy[308404]: worker exit
2026-11-11T14:50:30+01:00 demo-hp pveproxy[203540]: worker exit
2026-11-11T14:50:31+01:00 demo-hp pveproxy[1293]: worker 316496 finished
2026-11-11T14:50:31+01:00 demo-hp pveproxy[1293]: worker 203540 finished
2026-11-11T14:50:31+01:00 demo-hp pveproxy[1293]: worker 308404 finished
PID STARTED CMD
1293 Tue Nov 10 15:43:04 2026 pveproxy
1293 Tue Nov 10 15:43:04 2026 pveproxy
494020 Wed Nov 11 14:50:25 2026 pveproxy worker
494021 Wed Nov 11 14:50:25 2026 pveproxy worker
494022 Wed Nov 11 14:50:25 2026 pveproxy worker
@@ -0,0 +1,26 @@
== state 2026-11-11T13:52:21Z
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Nov 10 13:49:24 2026 GMT
notAfter=Nov 9 13:49:24 2028 GMT
sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12
agent tls config: {'fingerprint': '07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB', 'insecure_skip_verify': False}
agent: active since Tue 2026-11-10 15:58:17 CET restarts=0
-- agent calls in pveproxy's access log, last 4 (time, call, answer code)
[11/11/2026:14:50:19 +0100] "GET /api2/json/pools/felhom 200
[11/11/2026:14:50:19 +0100] "GET /api2/json/nodes/demo-hp/lxc 200
[11/11/2026:14:50:19 +0100] "GET /api2/json/pools/felhom 200
[11/11/2026:14:50:19 +0100] "GET /api2/json/nodes/demo-hp/storage 200
-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6)
22
time=2026-11-11T14:52:05.144+01:00 level=WARN msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" vmid=9201 target=felhom-pbs err="proxmox: GET /nodes/demo-hp/storage/felhom-pbs/content: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/storage/felhom-pbs/content\": pr
time=2026-11-11T14:52:05.505+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:52:19.211+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:52:19.230+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:52:19.393+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:52:19.394+01:00 level=WARN msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
-- guests
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
341 night1004-tester1 running
@@ -0,0 +1,37 @@
== all agent log lines since the new certificate was served (no sudo noise), first 25, cut
2026-11-11T14:50:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:50:39.237+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:50:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:50:39.323+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
2026-11-11T14:50:49+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:50:49.395+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:50:59+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:50:59.230+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:50:59+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:50:59.310+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
2026-11-11T14:51:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:05.501+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:51:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:19.212+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:51:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:19.231+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:51:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:19.313+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
2026-11-11T14:51:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:19.395+01:00 level=WARN msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:51:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:19.395+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:51:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:39.233+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:51:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:39.310+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
2026-11-11T14:51:49+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:49.395+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:51:59+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:59.231+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:51:59+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:59.313+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.111+01:00 level=WARN msg="local-api: storage view unavailable for the tier presence report" target=local err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.116+01:00 level=WARN msg="local-api: storage view unavailable for the tier presence report" target=felhom-pbs err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.124+01:00 level=WARN msg="local-api: storage view unavailable for the backup-target presence check — assuming present" target=local err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.129+01:00 level=WARN msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" vmid=9201 target=local err="proxmox: GET /nodes/demo-hp/storage/local/content: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/storage/local/content\": proxmox: TLS pin mismatch: server cert sha256 does n
2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.136+01:00 level=WARN msg="local-api: storage view unavailable for the backup-target presence check — assuming present" target=felhom-pbs err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.144+01:00 level=WARN msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" vmid=9201 target=felhom-pbs err="proxmox: GET /nodes/demo-hp/storage/felhom-pbs/content: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/storage/felhom-pbs/content\": proxmox: TLS pin mismatch: server cer
2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.144+01:00 level=INFO msg="local-api: backup storage unreadable — due-check uses the last success saved on disk (R-894)" vmid=9201 target=felhom-pbs saved=2026-11-05T20:50:35Z
2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.505+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:52:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:19.211+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
== count by message
6 msg="reconcile: storage view unavailable — using mount-table only"
6 msg="reconcile: enrolled drive bound under shared parent (live, no reboot)"
4 msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)"
2 msg="local-api: storage view unavailable for the tier presence report"
2 msg="local-api: storage view unavailable for the backup-target presence check — assuming present"
2 msg="local-api: /storage guest config"
2 msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record"
2 msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)"
2 msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)"
1 msg="local-api: backup storage unreadable — due-check uses the last success saved on disk (R-894)"
@@ -0,0 +1,17 @@
2026-11-11T13:54:07Z
HUB: .173.0 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 23% 37% running (connected) 37% local demo-hp-bb76ea Demo HP 0.173.0 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 73% 45% running (connected) 59% local-lvm tester-1-d70be4 Tester 1 0.173.0 9.2.2 7.0.14-22-pve ONLINE 1/2 8% 29% 33% running (connected) 33% local Felhom Hub 0.180.0
--
== guest 9201 containers: running count / not running
26
== agent report lines to the hub since 14:50
time=2026-11-11T14:52:05.111+01:00 level=WARN msg="local-api: storage view unavailable for the tier presence report" target=local err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configure
time=2026-11-11T14:52:05.116+01:00 level=WARN msg="local-api: storage view unavailable for the tier presence report" target=felhom-pbs err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match conf
== agent local API (what the controller asks) last lines
2026/11/11 13:54:05 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping bookstack — last check 3m0s ago, effective interval 5m0s, healthy=true
2026/11/11 13:54:05 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping home-assistant — last check 3m0s ago, effective interval 5m0s, healthy=true
2026/11/11 13:54:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health
2026/11/11 13:54:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting
2026/11/11 13:54:05 client.go:1288: [DEBUG] [agentapi] GET /storage -> 502 (4ms)
2026/11/11 13:54:05 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 5ms)
2026/11/11 13:54:06 client.go:1288: [DEBUG] [agentapi] GET /disks -> 502 (6ms)
2026/11/11 13:54:06 client.go:1288: [DEBUG] [agentapi] GET /disks -> 502 (5ms)
@@ -0,0 +1,9 @@
36
Forbidden: missing scope hosts:read
pin 0
TLS 0
certificate 0
proxmox: 0
unavailable 0
degraded 0
Storage 0
@@ -0,0 +1,14 @@
2026-11-11T13:54:34Z
== hub log lines naming demo-hp since 13:45Z (count, then last 15, cut)
9
2026/11/11 14:43:30 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 22057 bytes)
2026/11/11 14:43:30 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1)
2026/11/11 14:47:07 [INFO] DR-recipe app-half stored for customer demo-hp (v1)
2026/11/11 14:47:07 [INFO] Received report from demo-hp (10182 bytes)
2026/11/11 14:48:15 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 22040 bytes)
2026/11/11 14:48:15 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1)
2026/11/11 14:52:05 [INFO] Event from demo-hp: agent_channel_unknown (warning) — Controller→agent channel down: agentapi: GET /storage: HTTP 502
2026/11/11 14:52:05 [INFO] Operator email sent for demo-hp/agent_channel_unknown
2026/11/11 14:54:27 [WARN] access token "cc-releases-2" refused: missing scope hosts:read — GET /hosts/demo-hp-bb76ea → 403
== any hub line with pin/tls/certificate
0
@@ -0,0 +1,21 @@
2026-11-11T13:59:16Z
== hub log lines naming demo-hp, last 12 min
2026/11/11 14:48:15 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 22040 bytes)
2026/11/11 14:48:15 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1)
2026/11/11 14:52:05 [INFO] Event from demo-hp: agent_channel_unknown (warning) — Controller→agent channel down: agentapi: GET /storage: HTTP 502
2026/11/11 14:52:05 [INFO] Operator email sent for demo-hp/agent_channel_unknown
HUB LINE: demo-hp-bb76ea Demo HP 0.173.0 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 73% 45% running (connected) 59% local-lvm tester-1-d70be4 Tester
== agent lines about the hub report since 14:50 CET
26 msg="reconcile: storage view unavailable — using mount-table only"
26 msg="reconcile: enrolled drive bound under shared parent (live, no reboot)"
17 msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)"
9 msg="local-api: /storage guest config"
8 msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)"
8 msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)"
4 msg="local-api: storage view unavailable for the tier presence report"
4 msg="local-api: storage view unavailable for the backup-target presence check — assuming present"
4 msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record"
2 msg="local-api: backup storage unreadable — due-check uses the last success saved on disk (R-894)"
1 msg="stale-lock: guest list unavailable — skipping recovery"
1 msg="reconcile: pass failed"
1 msg="hub: collect failed; skipping this cycle's report"
@@ -0,0 +1,3 @@
time=2026-11-11T14:58:18.520+01:00 level=ERROR msg="reconcile: pass failed" err="reconcile: actual state: proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:58:19.394+01:00 level=WARN msg="stale-lock: guest list unavailable — skipping recovery" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:58:24.067+01:00 level=WARN msg="hub: collect failed; skipping this cycle's report" err="hub: NodeStatus failed (no useful report): proxmox: GET /nodes/demo-hp/status: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/status\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
@@ -0,0 +1,7 @@
== repair start 2026-11-11T13:59:25Z
== setpin fp start 2026-11-11T13:59:25Z
written tls: {'fingerprint': '21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12', 'insecure_skip_verify': False}
first agent call after restart: [11/11/2026:14:59:27 "GET /api2/json/pools/felhom 200
seconds from start to a working call: 2.149922736 (loops: 2; 60 = no call seen)
agent: active
== repair end 2026-11-11T13:59:28Z
@@ -0,0 +1,22 @@
== state 2026-11-11T13:59:29Z
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Nov 10 13:49:24 2026 GMT
notAfter=Nov 9 13:49:24 2028 GMT
sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12
agent tls config: {'fingerprint': '21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12', 'insecure_skip_verify': False}
agent: active since Wed 2026-11-11 14:59:25 CET restarts=0
-- agent calls in pveproxy's access log, last 4 (time, call, answer code)
[11/11/2026:14:59:27 +0100] "GET /api2/json/storage 200
[11/11/2026:14:59:27 +0100] "GET /api2/json/storage 200
[11/11/2026:14:59:28 +0100] "GET /api2/json/nodes/demo-hp/lxc 200
[11/11/2026:14:59:28 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6)
33
time=2026-11-11T14:58:59.229+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:05.502+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.213+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.232+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.394+01:00 level=WARN msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.394+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
-- guests
@@ -0,0 +1,15 @@
== CA test start 2026-11-11T13:59:39Z
-rw-r----- 1 root felhom-agent 2094 Nov 11 14:59 /etc/felhom-agent/pve-root-ca.pem
-rw-r----- 1 root www-data 2094 Aug 21 17:44 /etc/pve/pve-root-ca.pem
agent user can read the copy
agent user CANNOT read the original
subject=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Aug 20 15:44:42 2026 GMT
notAfter=Aug 17 15:44:42 2036 GMT
sha256 Fingerprint=E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7
== setpin ca start 2026-11-11T13:59:39Z
written tls: {'ca_file': '/etc/felhom-agent/pve-root-ca.pem', 'fingerprint': '', 'insecure_skip_verify': False}
first agent call after restart: [11/11/2026:14:59:41 "GET /api2/json/storage 200
seconds from start to a working call: 2.160335125 (loops: 2; 60 = no call seen)
agent: active
@@ -0,0 +1,26 @@
== state 2026-11-11T13:59:43Z
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Nov 10 13:49:24 2026 GMT
notAfter=Nov 9 13:49:24 2028 GMT
sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12
agent tls config: {'ca_file': '/etc/felhom-agent/pve-root-ca.pem', 'fingerprint': '', 'insecure_skip_verify': False}
agent: active since Wed 2026-11-11 14:59:39 CET restarts=0
-- agent calls in pveproxy's access log, last 4 (time, call, answer code)
[11/11/2026:14:59:41 +0100] "GET /api2/json/storage 200
[11/11/2026:14:59:41 +0100] "GET /api2/json/storage 200
[11/11/2026:14:59:41 +0100] "GET /api2/json/nodes/demo-hp/lxc 200
[11/11/2026:14:59:41 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6)
32
time=2026-11-11T14:58:59.229+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:05.502+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.213+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.232+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.394+01:00 level=WARN msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.394+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
-- guests
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
341 night1004-tester1 running
@@ -0,0 +1,43 @@
== force 2 start 2026-11-11T13:59:54Z
== force 2026-11-11T13:59:54Z
before: sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12
(re)generate node files
generate new node certificate
merge authorized SSH keys
updatecerts rc=0
file now: sha256 Fingerprint=1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74 notAfter=Nov 9 13:59:54 2028 GMT
served 5 s later, no restart: sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12
pveproxy: Tue 2026-11-10 15:43:04 CET reloads:
done 2026-11-11T13:59:59Z
reload rc=0 at 2026-11-11T14:00:01Z
== state 2026-11-11T14:00:46Z
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Nov 10 13:59:54 2026 GMT
notAfter=Nov 9 13:59:54 2028 GMT
sha256 Fingerprint=1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74
agent tls config: {'ca_file': '/etc/felhom-agent/pve-root-ca.pem', 'fingerprint': '', 'insecure_skip_verify': False}
agent: active since Wed 2026-11-11 14:59:39 CET restarts=0
-- agent calls in pveproxy's access log, last 4 (time, call, answer code)
[11/11/2026:15:00:41 +0100] "GET /api2/json/nodes/demo-hp/lxc 200
[11/11/2026:15:00:41 +0100] "GET /api2/json/pools/felhom 200
[11/11/2026:15:00:41 +0100] "GET /api2/json/pools/felhom 200
[11/11/2026:15:00:42 +0100] "GET /api2/json/nodes/demo-hp/storage 200
-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6)
18
time=2026-11-11T14:58:59.229+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:05.502+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.213+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.232+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.394+01:00 level=WARN msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.394+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
-- guests
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
341 night1004-tester1 running
== agent start time and restarts (must be unchanged: 14:59:39)
NRestarts=0
ActiveEnterTimestamp=Wed 2026-11-11 14:59:39 CET
== agent lines with x509/pin/tls since 14:59:45
0
@@ -0,0 +1,13 @@
== negative control: a CA file that did NOT sign the certificate. Start 2026-11-11T14:01:01Z
== setpin ca start 2026-11-11T14:01:01Z
written tls: {'ca_file': '/etc/ssl/certs/ISRG_Root_X1.pem', 'fingerprint': '', 'insecure_skip_verify': False}
first agent call after restart: [11/11/2026:15:01:02 "GET /api2/json/nodes/demo-hp/storage 500
seconds from start to a working call: .112020654 (loops: 0; 60 = no call seen)
agent: active
x509 lines: 0
== back to the standard: fingerprint pin. 2026-11-11T14:01:02Z
== setpin fp start 2026-11-11T14:01:02Z
written tls: {'fingerprint': '1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74', 'insecure_skip_verify': False}
first agent call after restart: [11/11/2026:15:01:02 "GET /api2/json/nodes/demo-hp/storage 500
seconds from start to a working call: .718520877 (loops: 0; 60 = no call seen)
agent: active
@@ -0,0 +1,36 @@
== the first control was void: the waiter matched a call of the same second; the agent ran under 1 s on the wrong file. Second try.
== state now 2026-11-11T14:01:14Z
[11/11/2026:15:01:07 "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
[11/11/2026:15:01:08 "GET /api2/json/nodes/demo-hp/storage 200
[11/11/2026:15:01:08 "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
::ffff:127.0.0.1 - felhom-agent@pve!agent [11/11/2026:15:01:02 +0100] "GET /api2/json/nodes/demo-hp/storage HTTP/1.1" 500 -
== wrong CA, 30 s
Job for felhom-agent.service failed because start of the service was attempted too often.
See "systemctl status felhom-agent.service" and "journalctl -xeu felhom-agent.service" for details.
To force a start use "systemctl reset-failed felhom-agent.service"
followed by "systemctl start felhom-agent.service" again.
== setpin ca start 2026-11-11T14:01:14Z
written tls: {'ca_file': '/etc/ssl/certs/ISRG_Root_X1.pem', 'fingerprint': '', 'insecure_skip_verify': False}
seconds from start to a working call: 61.364869823 (loops: 60; 60 = no call seen)
x509 lines in 32 s: 0
agent calls answered 200 in the last 25 s: 0
== back to the standard 2026-11-11T14:02:46Z
== setpin fp start 2026-11-11T14:02:46Z
written tls: {'fingerprint': '1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74', 'insecure_skip_verify': False}
first agent call after restart: [11/11/2026:15:02:48 "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
seconds from start to a working call: 2.148034679 (loops: 2; 60 = no call seen)
agent: active
== state 2026-11-11T14:03:09Z
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Nov 10 13:59:54 2026 GMT
notAfter=Nov 9 13:59:54 2028 GMT
sha256 Fingerprint=1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74
agent tls config: {'fingerprint': '1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74', 'insecure_skip_verify': False}
agent: active since Wed 2026-11-11 15:02:46 CET restarts=0
-- agent calls in pveproxy's access log, last 4 (time, call, answer code)
[11/11/2026:15:03:08 +0100] "GET /api2/json/nodes/demo-hp/storage 200
[11/11/2026:15:03:08 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
[11/11/2026:15:03:08 +0100] "GET /api2/json/storage 200
[11/11/2026:15:03:09 +0100] "GET /api2/json/nodes/demo-hp/storage 200
-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6)
@@ -0,0 +1,50 @@
2026-11-11T14:03:22Z
Restart=on-failure
RestartUSec=5s
Result=success
NRestarts=0
ActiveState=active
SubState=running
StartLimitIntervalUSec=2min
StartLimitBurst=4
2026-11-11T15:01:02+01:00 demo-hp systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)...
2026-11-11T15:01:02+01:00 demo-hp systemd[1]: felhom-agent.service: Deactivated successfully.
2026-11-11T15:01:02+01:00 demo-hp systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-11T15:01:02+01:00 demo-hp systemd[1]: felhom-agent.service: Consumed 28.460s CPU time, 348.1M memory peak.
2026-11-11T15:01:02+01:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-11T15:01:02+01:00 demo-hp systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)...
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.135+01:00 level=WARN msg="hub: collect failed; skipping this cycle's report" err="hub: NodeStatus failed (no useful rep
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="reconcile: pass failed" err="reconcile: actual state: proxmox: GET /nodes/demo-hp/lxc: Get \
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.136+01:00 level=WARN msg="reconcile: shared parent ensure failed — binds may not propagate live" err="shared-parent:
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.138+01:00 level=ERROR msg="reconcile: AttachDrive failed" vmid=9201 where=/mnt/hdd_1 err="guest-attach: namespace /mnt/
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.139+01:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.139+01:00 level=WARN msg="selfheal: node check FAILED" heal=networking detail="networking.service active=false, default
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.139+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_windo
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.139+01:00 level=ERROR msg="node self-heal: networking start attempt failed" attempt=1 err="context canceled: "
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.139+01:00 level=WARN msg="lanresolver: EnsureDnsmasq failed — will retry" err="enable dnsmasq: context canceled"
2026-11-11T15:01:03+01:00 demo-hp systemd[1]: felhom-agent.service: Deactivated successfully.
2026-11-11T15:01:03+01:00 demo-hp systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-11T15:01:03+01:00 demo-hp systemd[1]: felhom-agent.service: Consumed 1.212s CPU time, 23.5M memory peak.
2026-11-11T15:01:03+01:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-11T15:01:05+01:00 demo-hp felhom-agent[527856]: time=2026-11-11T15:01:05.230+01:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3
2026-11-11T15:01:05+01:00 demo-hp felhom-agent[527856]: time=2026-11-11T15:01:05.230+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_windo
2026-11-11T15:01:14+01:00 demo-hp felhom-agent[527856]: time=2026-11-11T15:01:14.815+01:00 level=WARN msg="pbs: verify task wait failed; reporting current snapshot state" datastore=felhom-offsite err=
2026-11-11T15:01:14+01:00 demo-hp felhom-agent[527856]: time=2026-11-11T15:01:14.815+01:00 level=WARN msg="pbs: snapshot list failed" datastore=felhom-offsite err="pbs: GET /admin/datastore/felhom-off
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)...
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent.service: Deactivated successfully.
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent.service: Consumed 15.975s CPU time, 327.3M memory peak.
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent.service: Start request repeated too quickly.
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent.service: Failed with result 'start-limit-hit'.
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: Failed to start felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent.service: Triggering OnFailure= dependencies.
2026-11-11T15:02:46+01:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-11T15:02:49+01:00 demo-hp felhom-agent[535281]: time=2026-11-11T15:02:49.047+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_windo
2026-11-11T15:02:49+01:00 demo-hp felhom-agent[535281]: time=2026-11-11T15:02:49.047+01:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3
== tests in the agent repo for the CA path
5:func TestNormalizeFingerprint(t *testing.T) {
20:func TestNormalizeFingerprint_Bad(t *testing.T) {
28:func TestTLSConfig_Build(t *testing.T) {
@@ -0,0 +1,26 @@
2026-11-11T14:03:37Z
== the 1-second run on the WRONG CA file (pid 527553): its proxmox errors
time=2026-11-11T15:01:03.135+01:00 level=WARN msg="hub: collect failed; skipping this cycle's report" err="hub: NodeStatus failed (no useful report): proxmox: GET /nodes/demo-hp/status: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/status\": terminated signal received"
time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="reconcile: pass failed" err="reconcile: actual state: proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": terminated signal received"
== x509 count for that pid: 3
== OnFailure unit
OnFailure=felhom-agent-rollback.service
Result=success
ExecMainStartTimestamp=Wed 2026-11-11 15:01:14 CET
ActiveState=inactive
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: Starting felhom-agent-rollback.service - Felhom agent self-update auto-rollback (pending-guarded; fires per crash, no-ops without a pending update)...
2026-11-11T15:01:14+01:00 demo-hp felhom-selfupdate-guarded[529488]: felhom-selfupdate-guarded: rollback: no pending update — no-op
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent-rollback.service: Deactivated successfully.
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: Finished felhom-agent-rollback.service - Felhom agent self-update auto-rollback (pending-guarded; fires per crash, no-ops without a pending update).
== now: capability / attach lines since 15:02:46
time=2026-11-11T15:02:48.053+01:00 level=INFO msg="capabilities self-check" ok=77 total=77 degraded=0 inactive=0
time=2026-11-11T15:02:48.851+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
time=2026-11-11T15:03:09.617+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
time=2026-11-11T15:03:29.562+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
(end)
== drive in guest 9201
/mnt/felhom-drives/hdd_1 /dev/nvme0n1[/felhom-data] ext4
/mnt/felhom-drives/hdd_1 /dev/nvme0n1[/felhom-data] ext4
== config: same as the backup but for tls?
rest equal: True | tls keys now: ['fingerprint', 'insecure_skip_verify'] | before: ['fingerprint', 'insecure_skip_verify']
-rw------- felhom-agent felhom-agent
@@ -0,0 +1,17 @@
2026-11-11T14:03:44Z
time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=pve:pool-read feature="stale-lock recovery scoping (pool ownership check)" reason="proxmox: GET /pools/felhom: Get \"https://127.0.0.1:8006/api2/json/pools/felhom\": tls: failed to verify certificate: x509: certificate signed by unknown authority" critical=false
time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=pve:store-grant:local feature="backup tier local readable by the agent (archive listing, restore-test candidacy)" reason="could not read own permissions: proxmox: GET /access/permissions?path=%2Fstorage%2Flocal: Get \"https://127.0.0.1:8006/api2/json/access/permissions?path=%2Fstorage%
time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=pve:store-grant:felhom-pbs feature="backup tier felhom-pbs readable by the agent (archive listing, restore-test candidacy)" reason="could not read own permissions: proxmox: GET /access/permissions?path=%2Fstorage%2Ffelhom-pbs: Get \"https://127.0.0.1:8006/api2/json/access/permissions?p
== hub: demo-hp lines since the repair
2026/11/11 14:59:32 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 21349 bytes)
2026/11/11 14:59:32 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1)
2026/11/11 14:59:47 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 21349 bytes)
2026/11/11 14:59:47 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1)
2026/11/11 15:00:06 [INFO] Event from demo-hp: agent_channel_recovered (info) — Controller→agent channel recovered — local-API reachable again.
2026/11/11 15:01:10 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 21351 bytes)
2026/11/11 15:01:10 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1)
2026/11/11 15:02:07 [INFO] DR-recipe app-half stored for customer demo-hp (v1)
2026/11/11 15:02:07 [INFO] Received report from demo-hp (14341 bytes)
2026/11/11 15:02:07 [INFO] managed floor SERVED for demo-hp: floor 0.352.0, agent requirement "0.158.1" from manifest (golden 0.352.0)
2026/11/11 15:02:53 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 21349 bytes)
2026/11/11 15:02:53 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1)
@@ -0,0 +1,18 @@
2026-11-11T14:03:55Z
0
(0 = both files gone)
active
active
NRestarts=0
ActiveEnterTimestamp=Wed 2026-11-11 15:02:46 CET
[11/11/2026:15:03:49 "GET /api2/json/pools/felhom 200
[11/11/2026:15:03:49 "GET /api2/json/nodes/demo-hp/storage 200
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
341 night1004-tester1 running
26
Host key verification failed.
active
9201 running
2026-11-11T14:04:07Z
@@ -0,0 +1,41 @@
main() {
echo "== $(hostname) $(date -u +%FT%TZ) pve $(pveversion | head -1)"
echo "== served certificate (network read, 127.0.0.1:8006)"
echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -subject -issuer -dates -fingerprint -sha256 -ext subjectAltName 2>/dev/null
echo "== files"
ls -la --time-style=full-iso /etc/pve/local/ /etc/pve/pve-root-ca.pem /etc/pve/priv/pve-root-ca.key 2>&1 | cut -c1-140
for f in /etc/pve/local/pve-ssl.pem /etc/pve/pve-root-ca.pem; do echo "-- $f"; openssl x509 -in $f -noout -subject -issuer -dates -fingerprint -sha256; done
ls /etc/pve/local/pveproxy-ssl.pem 2>&1
echo "== agent"
/usr/local/bin/felhom-agent --version; systemctl is-active felhom-agent; systemctl show felhom-agent -p ActiveEnterTimestamp -p NRestarts -p Restart
python3 - <<'PY'
import json
c=json.load(open('/etc/felhom-agent/agent.json'))
p=c.get('proxmox',{})
print('proxmox keys:',sorted(p.keys()))
for k,v in p.items():
if k=='tls': print('tls:',v)
elif 'secret' in k.lower() or 'token' in k.lower(): print(k,': <present>' if v else ': <empty>')
else: print(k,':',v)
PY
ls -la --time-style=full-iso /etc/felhom-agent/ | cut -c1-120
systemctl cat felhom-agent | grep -i -n 'Environment\|ExecStart\|User='
echo "== agent log, last proxmox lines"
journalctl -u felhom-agent --since '-20 min' --no-pager -o cat | grep -i -c .
journalctl -u felhom-agent --since '-20 min' --no-pager -o cat | cut -c1-230 | tail -8
echo "== who calls port 8006 (pveproxy access log, client addresses, last 2 days)"
for f in /var/log/pveproxy/access.log /var/log/pveproxy/access.log.1; do [ -f $f ] && awk '{print $1}' $f; done | sort | uniq -c | sort -rn | head
echo "-- users seen"
for f in /var/log/pveproxy/access.log /var/log/pveproxy/access.log.1; do [ -f $f ] && awk '{print $1, $3}' $f; done | sort | uniq -c | sort -rn | head
echo "== listening / other TLS users of this cert"
ss -ltnp | grep -E ':8006|:3128|:8007|:85 ' | cut -c1-150
echo "== in flight"
pvesh get /cluster/tasks --output-format json 2>/dev/null | python3 -c "import json,sys; t=json.load(sys.stdin); r=[x for x in t if not x.get('endtime')]; print('running tasks:',[(x['type'],x.get('id')) for x in r])"
qm list; pct list
echo "== tools' own help"
pvecm help updatecerts 2>&1 | head -20
pvenode help cert 2>&1 | head -30
echo "== renew rule in code"
grep -n -i 'expir\|renew\|check_expiry\|2 \* 365\|730\|days' /usr/share/perl5/PVE/Cluster/Setup.pm | head -30
grep -rn -i 'renew\|check_expiry\|updatecerts' /usr/share/perl5/PVE/Service/pveproxy.pm /usr/bin/pve-daily-update /usr/share/perl5/PVE/CLI/pveupdate.pm /usr/bin/pveupdate 2>/dev/null | head -20
}; main </dev/null
@@ -0,0 +1,11 @@
# force a new node certificate the documented way: pvecm updatecerts --force
main() {
echo "== force $(date -u +%FT%TZ)"
echo "before: $(echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256)"
pvecm updatecerts --force; echo "updatecerts rc=$?"
echo "file now: $(openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -fingerprint -sha256 -enddate | tr '\n' ' ')"
sleep 5
echo "served 5 s later, no restart: $(echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256)"
echo "pveproxy: $(systemctl show pveproxy -p ActiveEnterTimestamp --value) reloads: $(journalctl -u pveproxy --since '-1 min' --no-pager -o cat | tail -3 | tr '\n' ';')"
echo "done $(date -u +%FT%TZ)"
}; main </dev/null
@@ -0,0 +1,19 @@
# usage: setpin.sh fp | setpin.sh ca — writes the agent's proxmox.tls block, restarts the agent
main() {
mode=$1; t0=$(date +%s.%N); echo "== setpin $mode start $(date -u +%FT%TZ)"
FP=$(echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256 | cut -d= -f2)
MODE=$mode FP=$FP python3 - <<'PY'
import json,os
f='/etc/felhom-agent/agent.json'; c=json.load(open(f)); st=os.stat(f)
if os.environ['MODE']=='fp': c['proxmox']['tls']={'fingerprint':os.environ['FP'],'insecure_skip_verify':False}
else: c['proxmox']['tls']={'ca_file':'/etc/ssl/certs/ISRG_Root_X1.pem','fingerprint':'','insecure_skip_verify':False}
tmp=f+'.tmp'; fd=os.open(tmp,os.O_WRONLY|os.O_CREAT|os.O_TRUNC,0o600)
with os.fdopen(fd,'w') as o: json.dump(c,o,indent=2); o.write('\n')
os.chown(tmp,st.st_uid,st.st_gid); os.rename(tmp,f); print('written tls:',c['proxmox']['tls'])
PY
systemctl restart felhom-agent
n=0; mark=$(date '+%d/%m/%Y:%H:%M:%S'); until [ $n -ge 60 ]; do l=$(grep 'felhom-agent@pve' /var/log/pveproxy/access.log | tail -1); ts=$(echo "$l" | awk '{print $4}' | tr -d '[');
if [ "$(date -d "$(echo $ts | sed 's#\([0-9]*\)/\([0-9]*\)/\([0-9]*\):#\3-\2-\1 #')" +%s 2>/dev/null || echo 0)" -ge "${t0%.*}" ]; then echo "first agent call after restart: $(echo "$l" | awk '{print $4,$6,$7,$9}')"; break; fi; n=$((n+1)); sleep 1; done
t1=$(date +%s.%N); echo "seconds from start to a working call: $(echo "$t1 - $t0" | bc) (loops: $n; 60 = no call seen)"
echo "agent: $(systemctl is-active felhom-agent)"
}; main "$@" </dev/null
@@ -0,0 +1,19 @@
# usage: setpin.sh fp | setpin.sh ca — writes the agent's proxmox.tls block, restarts the agent
main() {
mode=$1; t0=$(date +%s.%N); echo "== setpin $mode start $(date -u +%FT%TZ)"
FP=$(echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256 | cut -d= -f2)
MODE=$mode FP=$FP python3 - <<'PY'
import json,os
f='/etc/felhom-agent/agent.json'; c=json.load(open(f)); st=os.stat(f)
if os.environ['MODE']=='fp': c['proxmox']['tls']={'fingerprint':os.environ['FP'],'insecure_skip_verify':False}
else: c['proxmox']['tls']={'ca_file':'/etc/felhom-agent/pve-root-ca.pem','fingerprint':'','insecure_skip_verify':False}
tmp=f+'.tmp'; fd=os.open(tmp,os.O_WRONLY|os.O_CREAT|os.O_TRUNC,0o600)
with os.fdopen(fd,'w') as o: json.dump(c,o,indent=2); o.write('\n')
os.chown(tmp,st.st_uid,st.st_gid); os.rename(tmp,f); print('written tls:',c['proxmox']['tls'])
PY
systemctl restart felhom-agent
n=0; mark=$(date '+%d/%m/%Y:%H:%M:%S'); until [ $n -ge 60 ]; do l=$(grep 'felhom-agent@pve' /var/log/pveproxy/access.log | tail -1); ts=$(echo "$l" | awk '{print $4}' | tr -d '[');
if [ "$(date -d "$(echo $ts | sed 's#\([0-9]*\)/\([0-9]*\)/\([0-9]*\):#\3-\2-\1 #')" +%s 2>/dev/null || echo 0)" -ge "${t0%.*}" ]; then echo "first agent call after restart: $(echo "$l" | awk '{print $4,$6,$7,$9}')"; break; fi; n=$((n+1)); sleep 1; done
t1=$(date +%s.%N); echo "seconds from start to a working call: $(echo "$t1 - $t0" | bc) (loops: $n; 60 = no call seen)"
echo "agent: $(systemctl is-active felhom-agent)"
}; main "$@" </dev/null
@@ -0,0 +1,15 @@
# read-only state: served cert, agent pin, agent calls in the access log, agent log lines
main() {
echo "== state $(date -u +%FT%TZ)"
echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -subject -issuer -dates -fingerprint -sha256 2>/dev/null
python3 -c "
import json;c=json.load(open('/etc/felhom-agent/agent.json'));print('agent tls config:',c['proxmox']['tls'])"
echo "agent: $(systemctl is-active felhom-agent) since $(systemctl show felhom-agent -p ActiveEnterTimestamp --value) restarts=$(systemctl show felhom-agent -p NRestarts --value)"
echo "-- agent calls in pveproxy's access log, last 4 (time, call, answer code)"
grep 'felhom-agent@pve' /var/log/pveproxy/access.log | tail -4 | awk '{print $4,$5,$6,$7,$9}'
echo "-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6)"
journalctl -u felhom-agent --since '-3 min' --no-pager -o cat | grep -v 'pam_unix\|COMMAND=' | grep -i -c 'pin mismatch\|x509\|tls'
journalctl -u felhom-agent --since '-3 min' --no-pager -o cat | grep -v 'pam_unix\|COMMAND=' | grep -i 'pin mismatch\|x509\|tls' | cut -c1-330 | tail -6
echo "-- guests"
pct list | awk 'NR>1{print $1,$2,$NF}'; qm list | awk 'NR>1{print $1,$2,$3}'
}; main </dev/null
@@ -0,0 +1,77 @@
# Why the N100 keeps no time — 2026-11-11
Read only. Nothing on the N100, on Tester 1 or on demo-hp was changed. Files: `evidence/`, `tools/` (the read scripts).
Documents read: `architecture/11-os-updates.md`, `architecture/03-host-agent.md`, the Clock list
(`audits/clock-list-2026-11-11/`, entries CA03, CA04, CA09), rows R-1301, R-1384, R-1206.
## The cause, in three lines
1. The time service (`chrony`) tells systemd "I am ready" through the socket `/run/systemd/notify`.
2. On the N100 the kernel **refuses that one write**, because the security profile (AppArmor) it loaded for chrony is
an **old copy from a cache** — the copy made for the older chrony, which did not talk to that socket.
3. chrony treats the refused write as fatal and stops. So the unit fails 2 seconds into every start.
## The evidence (each line is a read, not a theory)
| What | N100 | demo-hp | Tester 1 host | File |
|---|---|---|---|---|
| chrony package | `4.8-4~bpo13+2` | the same | `4.6.1-3+deb13u1` (old) | `*-stale-proof.txt` |
| unit, drop-ins, `/etc/default/chrony`, config, profile file on disk | identical (profile sha256 `214e509a…`) | identical | not compared | `n100-host.txt`, `demo-hp-host.txt` |
| unit state | **failed**, 8 fatal lines = 4 starts | active, 0 fatal lines | active, 0 | `*-host-2.txt` |
| profile the kernel has loaded (sha256) | **`dbf67f16…`** | `83417e83…` | `dbf67f16…` | `*-stale-proof.txt` |
| a fresh compile of the profile file on disk | `83417e83…` | `83417e83…` | `dbf67f16…` | the same |
| loaded = fresh compile? | **no** | yes | yes | the same |
| the kernel's own line | `apparmor="DENIED" operation="sendmsg" profile="/usr/sbin/chronyd" name="/run/systemd/notify" … denied_mask="w"` (2026-10-16 05:09:25) | none | none | `n100-host-2.txt:13` |
`dbf67f16…` is the profile of chrony 4.6.1 (Tester 1, which still runs 4.6.1, compiles exactly that). `83417e83…` is
the profile of chrony 4.8. **The N100 runs chrony 4.8 under the profile of chrony 4.6.1.**
chrony's source agrees with the log: `main.c` `notify_system_manager()` ends in
`LOG_FATAL("Could not send notification to $NOTIFY_SOCKET")` on any failed send, and sends once at start and once at
exit. That is the same line twice for each start.
## How the old copy got there (dates from the box)
AppArmor keeps one cache folder for each kernel feature set. Both boxes have two: `ae1a4050.0` (kernel 7.0.2) and
`f6da0254.0` (the 7.0.14 kernels).
| When | N100 | demo-hp |
|---|---|---|
| the 7.0.14 folder was made, with the old chrony profile in it | **2026-10-07 12:11** (a kernel test of that day booted the new kernel once, then went back) | 2026-10-04 09:44 |
| chrony went to 4.8 (apt, requested by `felhom-agent`, the OS-update leg) | 2026-10-07 14:15, **on the old kernel** → the new profile went into the 7.0.2 folder only | 2026-10-10 04:40, on a 7.0.14 kernel → the new profile went into the 7.0.14 folder |
| next start on a 7.0.14 kernel | the 7.0.14 folder still holds the old profile; it is loaded | the new profile is loaded |
Why AppArmor did not see that the copy was old: the cached file carries the time 2026-07-18 18:15:53; the new profile
file from the package carries 2026-07-01 22:28:16. The cache looks newer than its source, so it is used. (The time
rule is AppArmor's documented cache check; this run did not read its source.)
A control on the method: the same check names `lxc-containers` as "differs" on all three boxes, the healthy ones too.
That one is not a signal. `usr.sbin.chronyd` differs on the N100 only.
## The three questions
1. **Is the cause the product's?** In part. The product did not write the profile or the unit. But the product's
OS-update leg installed the new chrony, and the bad order is: *start a new kernel once → go back to the old kernel →
a package with a changed profile is updated → start the new kernel again.* On the N100 a test made that order. On a
household box the **kernel rollback** (the crash guard goes back to the last good kernel) makes the same order. It
is true for every AppArmor profile, not only chrony's.
2. **When did it start?** The oldest failed start in the journal is 2026-10-16 05:09:25. The cause dates from
2026-10-07 14:15. The journal has no start between those two dates.
3. **The smallest repair.** One hand step on the N100 (in `STATUS.md`, not done, **not tested on a box**):
`apparmor_parser -r -T -W /etc/apparmor.d/usr.sbin.chronyd && systemctl restart chrony && chronyc tracking`.
For a household box no path delivers it today. Proposed, not built: after an OS-update leg `felhom-os-apply`
reloads the profiles with the cache ignored (`-T -W`), and the kernel rollback does the same. Filed as a row.
**The fresh box of Part 1 cannot show this.** It has one cache folder and no history. It showed only that a new
install keeps time (chrony active, 0.000001 s off, before the Felhom installer ran).
## What a wrong clock breaks (numbers from source)
- **Signed jobs** (the only way an agent update arrives): valid from issue minus 2 minutes (`DefaultClockSkew`,
`felhom-agent/internal/authz/verifier.go:21`) to expiry; a window over 24 hours is refused (`MaxWindow`, `:26`). A
host clock **more than 120 s behind** refuses every job as "not yet valid" (`verifier.go:153-162`).
- The N100 was 32.6 s behind this morning, after 26 days with no time service: about 1.25 s a day. At that rate it
passes 120 s in **about 70 days (mid-January 2027)**. This is an estimate from one measurement.
- The guest has no clock of its own; it reads the host's. So the backup age, the off-site age and the proof age that
the hub trusts, the night slots and the crash guard all read this clock (Clock list, class W2, 30 cells, R-1384).
- Who notices today: nobody. The hub alarms only on a tunnel handshake from the future.
@@ -0,0 +1,77 @@
#### demo-hp 2026-11-11T13:16:04Z
## fresh compile of the on-disk profile, to stdout only (no load, no cache write)
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -
apparmor_parser: Too many actions given on the command line.
## loaded raw policy sha256
83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1
## cache dirs
total 16
drwxr-xr-x 4 root root 4096 2026-10-04 09:44:36.114533363 +0200 .
drwxr-xr-x 13 root root 4096 2026-08-21 17:42:55.201857862 +0200 ..
drwx------ 2 root root 4096 2026-08-21 17:44:37.686947147 +0200 ae1a4050.0
drwx------ 2 root root 4096 2026-11-08 02:48:51.017396841 +0100 f6da0254.0
/var/cache/apparmor/ae1a4050.0/ features-sha=0e6b37757d22af1e files=106
-rw-r--r-- 1 root root 3177 2026-08-21 17:44:37.159696460 +0200 /var/cache/apparmor/ae1a4050.0//.features
-rw------- 1 root root 295529 2026-08-21 17:42:38.360866076 +0200 /var/cache/apparmor/ae1a4050.0//usr.sbin.chronyd
/var/cache/apparmor/f6da0254.0/ features-sha=611413ba507a5c19 files=106
-rw-r--r-- 1 root root 3219 2026-10-04 09:44:36.115533363 +0200 /var/cache/apparmor/f6da0254.0//.features
-rw------- 1 root root 296497 2026-08-21 17:42:38.360866076 +0200 /var/cache/apparmor/f6da0254.0//usr.sbin.chronyd
## kernel features now
yes yes yes yes yes
611413ba507a5c19
## newest source file the profile depends on
2025-07-21 12:21:49.000000000 /etc/apparmor.d/abi/kernel-5.4-vanilla
2025-07-21 12:21:49.000000000 /etc/apparmor.d/abstractions/base
2025-07-21 12:21:49.000000000 /etc/apparmor.d/abstractions/nameservice
2025-07-21 12:21:49.000000000 /etc/apparmor.d/tunables/global
2026-05-21 13:32:03.000000000 /etc/apparmor.d/local/usr.sbin.chronyd
2026-07-01 22:28:16.000000000 /etc/apparmor.d/usr.sbin.chronyd
## files in /etc/apparmor.d changed 2026-07-18 or 2026-08-21
/etc/apparmor.d/lxc
/etc/apparmor.d/abi
/etc/apparmor.d/abstractions
/etc/apparmor.d/abstractions/lxc
/etc/apparmor.d/abstractions/ubuntu-browsers.d
/etc/apparmor.d/abstractions/apparmor_api
/etc/apparmor.d/local
/etc/apparmor.d/local/usr.bin.tcpdump
/etc/apparmor.d/local/lxc-containers
/etc/apparmor.d/local/usr.sbin.dhclient
## versions
apparmor 4.1.1-pmx1
chrony 4.8-4~bpo13+2
AppArmor parser version 4.1.1
## dpkg log
2026-08-21 15:41:16 install apparmor:amd64 <none> 4.1.1-pmx1
2026-08-21 15:41:58 install proxmox-kernel-7.0.2-6-pve-signed:amd64 <none> 7.0.2-6
2026-10-04 09:42:20 install proxmox-kernel-7.0.14-20-pve-signed:amd64 <none> 7.0.14-20
2026-10-10 04:40:04 upgrade chrony:amd64 4.6.1-3+deb13u1 4.8-4~bpo13+2
2026-10-10 04:41:43 install proxmox-kernel-7.0.14-22-pve-signed:amd64 <none> 7.0.14-22
2026-10-11 04:42:12 install proxmox-kernel-7.0.14-23-pve-signed:amd64 <none> 7.0.14-23
2026-10-16 04:48:58 install proxmox-kernel-7.0.14-24-pve-signed:amd64 <none> 7.0.14-24
2026-10-21 04:45:13 install proxmox-kernel-7.0.14-25-pve-signed:amd64 <none> 7.0.14-25
2026-10-29 05:06:43 install proxmox-kernel-7.0.14-26-pve-signed:amd64 <none> 7.0.14-26
2026-11-10 02:50:13 install proxmox-kernel-7.0.14-28-pve-signed:amd64 <none> 7.0.14-28
## kernels booted
IDX BOOT ID FIRST ENTRY LAST ENTRY
-5 182af91122a941ec97ac7ee247a4d470 Sun 2026-10-04 21:42:08 CEST Sun 2026-10-04 21:43:33 CEST
-4 bb68aee4240244718d8a7f39d999a3a8 Wed 2026-11-04 21:06:10 CET Tue 2026-11-10 02:53:28 CET
-3 b6e0edea2b5647dfa36f5bb3825adc3a Tue 2026-11-10 02:53:57 CET Tue 2026-11-10 13:56:35 CET
-2 d1167d9431bf459abf2a569e8997a569 Tue 2026-11-10 13:57:04 CET Tue 2026-11-10 15:32:31 CET
-1 e2cc9e3e60f14bde8949f01d7ba35ccf Tue 2026-11-10 15:32:59 CET Tue 2026-11-10 15:42:26 CET
0 bed106f79e804eb1ace314a26da7f049 Tue 2026-11-10 15:42:54 CET Wed 2026-11-11 14:16:04 CET
br-1359a945d146: port 1(vethf5b125d) entered disabled state
Linux version 7.0.14-28-pve (build@proxmox) (gcc (Debian 14.
Linux version 7.0.14-28-pve (build@proxmox) (gcc (Debian 14.
Linux version 7.0.14-26-pve (build@proxmox) (gcc (Debian 14.
Linux version 7.0.14-28-pve (build@proxmox) (gcc (Debian 14.
## chronyd confinement now
/usr/sbin/chronyd (enforce) 780 ? 00:00:00 chronyd
/usr/sbin/chronyd (enforce) 903 ? 00:00:00 chronyd
## what installed chrony 4.8
Start-Date: 2026-10-10 04:39:59
Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --only-upgrade --no-install-recommends frr=10.6.1-1+pve3 libradosstriper1=19.2.6-pve4 librgw2=19.2.6-pve4 ceph-common=19.2.6-pve4 librbd1=19.2.6-pve4 libra
Requested-By: felhom-agent (999)
Upgrade: librados2:amd64 (19.2.3-pve4, 19.2.6-pve4), pve-docs:amd64 (9.2.1, 9.2.13), pve-edk2-firmware-ovmf:amd64 (4.2025.05-2, 4.2026.08-1), ceph-fuse:amd64 (19.2.3-pve4, 19.2.6-pve4), proxmox-widget-toolkit:amd64 (5.2.2, 5.2.10), corosync:amd64 (3.1.10-pve2,
Start-Date: 2026-10-10 04:41:43
@@ -0,0 +1,232 @@
#### demo-hp 2026-11-11T13:11:00Z
net.unix.max_dgram_qlen = 512
kernel.printk_ratelimit = 5
## cmdline
BOOT_IMAGE=/boot/vmlinuz-7.0.14-28-pve root=/dev/mapper/pve-root ro quiet
## apparmor denied this boot: 4
Nov 10 15:43:11 demo-hp kernel: audit: type=1400 audit(1794321791.730:129): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=2094 comm="vgs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
Nov 10 15:43:11 demo-hp kernel: audit: type=1400 audit(1794321791.782:130): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=2095 comm="lvs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
Nov 10 15:43:11 demo-hp kernel: audit: type=1400 audit(1794321791.923:131): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=2111 comm="vgs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
Nov 10 15:43:11 demo-hp kernel: audit: type=1400 audit(1794321791.986:132): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=2112 comm="lvs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
## audit lines with chrony, all boots
## the second around the chrony start, this boot
start: 2026-11-10T15:42:59+01:00
[ 8.825037] demo-hp kernel: input: HD-Audio Generic HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:08.1/0000:05:00.1/sound/card0/input9
[ 8.826027] demo-hp kernel: input: HD-Audio Generic HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:08.1/0000:05:00.1/sound/card0/input10
[ 8.826114] demo-hp kernel: input: HD-Audio Generic HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:08.1/0000:05:00.1/sound/card0/input11
[ 8.827023] demo-hp kernel: input: HD-Audio Generic HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:08.1/0000:05:00.1/sound/card0/input12
[ 8.831933] demo-hp systemd[1]: Reached target sound.target - Sound Card.
[ 8.837028] demo-hp kernel: amdgpu: Virtual CRAT table created for CPU
[ 8.837135] demo-hp kernel: amdgpu: Topology: Add CPU node
[ 8.837161] demo-hp kernel: amdgpu 0000:05:00.0: enabling device (0106 -> 0107)
[ 8.837625] demo-hp kernel: amdgpu 0000:05:00.0: initializing kernel modesetting (RAVEN 0x1002:0x15DD 0x103C:0x8522 0x82).
[ 8.838111] demo-hp kernel: amdgpu 0000:05:00.0: register mmio base: 0xFCB00000
[ 8.838441] demo-hp kernel: amdgpu 0000:05:00.0: register mmio size: 524288
[ 8.840015] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 0 <common_v2_0_0> (soc15_common)
[ 8.840395] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 1 <gmc_v9_0_0> (gmc_v9_0)
[ 8.840735] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 2 <ih_v4_0_0> (vega10_ih)
[ 8.841064] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 3 <psp_v10_0_0> (psp)
[ 8.841387] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 4 <smu_v1_0_0> (powerplay)
[ 8.841727] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 5 <dce_v1_0_0> (dm)
[ 8.842073] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 6 <gfx_v9_0_0> (gfx_v9_0)
[ 8.842434] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 7 <sdma_v4_0_0> (sdma_v4_0)
[ 8.842772] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 8 <vcn_v1_0_0> (vcn_v1_0)
[ 8.843155] demo-hp kernel: amdgpu 0000:05:00.0: Fetched VBIOS from VFCT
[ 8.843500] demo-hp kernel: amdgpu 0000:05:00.0: [drm] ATOM BIOS: 113-RAVEN-111
[ 8.843847] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: ALC256: picked fixup (pin match)
[ 8.844233] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: autoconfig for ALC256: line_outs=1 (0x14/0x0/0x0/0x0/0x0) type:speaker
[ 8.844547] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[ 8.844874] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: hp_outs=1 (0x21/0x0/0x0/0x0/0x0)
[ 8.845180] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: mono: mono_out=0x0
[ 8.845473] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: inputs:
[ 8.845756] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: Mic=0x19
[ 8.870030] demo-hp kernel: Console: switching to colour dummy device 80x25
[ 8.871013] demo-hp kernel: amdgpu 0000:05:00.0: vgaarb: deactivate vga console
[ 8.871457] demo-hp kernel: amdgpu 0000:05:00.0: Trusted Memory Zone (TMZ) feature enabled
[ 8.871771] demo-hp kernel: amdgpu 0000:05:00.0: vm size is 262144 GB, 4 levels, block size is 9-bit, fragment size is 9-bit
[ 8.872113] demo-hp kernel: amdgpu 0000:05:00.0: VRAM: 2048M 0x000000F400000000 - 0x000000F47FFFFFFF (2048M used)
[ 8.872452] demo-hp kernel: amdgpu 0000:05:00.0: GART: 1024M 0x0000000000000000 - 0x000000003FFFFFFF
[ 8.872757] demo-hp kernel: amdgpu 0000:05:00.0: [drm] Detected VRAM RAM=2048M, BAR=2048M
[ 8.873106] demo-hp kernel: amdgpu 0000:05:00.0: [drm] RAM width 128bits DDR4
[ 8.873432] demo-hp kernel: amdgpu 0000:05:00.0: 2048M of VRAM memory ready
[ 8.873746] demo-hp kernel: amdgpu 0000:05:00.0: 14997M of GTT memory ready.
[ 8.874109] demo-hp kernel: amdgpu 0000:05:00.0: [drm] GART: num cpu pages 262144, num gpu pages 262144
[ 8.874443] demo-hp kernel: amdgpu 0000:05:00.0: [drm] PCIE GART of 1024M enabled.
[ 8.874765] demo-hp kernel: amdgpu 0000:05:00.0: [drm] PTB located at 0x000000F400A00000
[ 8.875104] demo-hp kernel: amdgpu: hwmgr_sw_init smu backed is smu10_smu
[ 8.875136] demo-hp kernel: amdgpu 0000:05:00.0: [VCN instance 0] Found VCN firmware Version ENC: 1.15 DEC: 3 VEP: 0 Revision: 0
[ 8.882025] demo-hp kernel: input: HD-Audio Generic Mic as /devices/pci0000:00/0000:00:08.1/0000:05:00.6/sound/card1/input13
[ 8.882116] demo-hp kernel: input: HD-Audio Generic Front Headphone as /devices/pci0000:00/0000:00:08.1/0000:05:00.6/sound/card1/input14
[ 8.895018] demo-hp kernel: amdgpu 0000:05:00.0: reserve 0x400000 from 0xf47fc00000 for PSP TMR
[ 8.974285] demo-hp kernel: amdgpu 0000:05:00.0: RAS: optional ras ta ucode is not available
[ 8.982279] demo-hp kernel: amdgpu 0000:05:00.0: RAP: optional rap ta ucode is not available
[ 8.982905] demo-hp kernel: amdgpu 0000:05:00.0: SECUREDISPLAY: optional securedisplay ta ucode is not available
[ 8.990029] demo-hp kernel: [drm] DM_PPLIB: values for F clock
[ 8.990101] demo-hp kernel: [drm] DM_PPLIB: 400000 in kHz, 3599 in mV
[ 8.990150] demo-hp kernel: [drm] DM_PPLIB: 933000 in kHz, 3724 in mV
[ 8.990192] demo-hp kernel: [drm] DM_PPLIB: 1067000 in kHz, 3924 in mV
[ 8.990225] demo-hp kernel: [drm] DM_PPLIB: 1200000 in kHz, 4074 in mV
[ 8.990252] demo-hp kernel: [drm] DM_PPLIB: values for DCF clock
[ 8.990295] demo-hp kernel: [drm] DM_PPLIB: 300000 in kHz, 3599 in mV
[ 8.990330] demo-hp kernel: [drm] DM_PPLIB: 600000 in kHz, 3724 in mV
[ 8.990379] demo-hp kernel: [drm] DM_PPLIB: 626000 in kHz, 3924 in mV
[ 8.990415] demo-hp kernel: [drm] DM_PPLIB: 654000 in kHz, 4074 in mV
[ 8.990720] demo-hp kernel: amdgpu 0000:05:00.0: [drm] Display Core v3.2.369 initialized on DCN 1.0
[ 8.994046] demo-hp kernel: snd_hda_intel 0000:05:00.1: bound 0000:05:00.0 (ops amdgpu_dm_audio_component_bind_ops [amdgpu])
[ 9.036409] demo-hp kernel: amdgpu 0000:05:00.0: kiq ring mec 2 pipe 1 q 0
[ 9.053051] demo-hp kernel: kfd kfd: Allocated 3969056 bytes on gart
[ 9.053414] demo-hp kernel: kfd kfd: Total number of KFD nodes to be created: 1
[ 9.053588] demo-hp kernel: amdgpu: Virtual CRAT table created for GPU
[ 9.053620] demo-hp kernel: amdgpu: Topology: Add GPU node [0x1002:0x15dd]
[ 9.053643] demo-hp kernel: kfd kfd: added device 1002:15dd
[ 9.053792] demo-hp kernel: amdgpu 0000:05:00.0: SE 1, SH per SE 1, CU per SH 11, active_cu_number 8
[ 9.054099] demo-hp kernel: amdgpu 0000:05:00.0: ring gfx uses VM inv eng 0 on hub 0
[ 9.054373] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.0.0 uses VM inv eng 1 on hub 0
[ 9.054628] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.1.0 uses VM inv eng 4 on hub 0
[ 9.054881] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.2.0 uses VM inv eng 5 on hub 0
[ 9.055139] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.3.0 uses VM inv eng 6 on hub 0
[ 9.055384] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.0.1 uses VM inv eng 7 on hub 0
[ 9.055641] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.1.1 uses VM inv eng 8 on hub 0
[ 9.055897] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.2.1 uses VM inv eng 9 on hub 0
[ 9.056158] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.3.1 uses VM inv eng 10 on hub 0
[ 9.056432] demo-hp kernel: amdgpu 0000:05:00.0: ring kiq_0.2.1.0 uses VM inv eng 11 on hub 0
[ 9.056680] demo-hp kernel: amdgpu 0000:05:00.0: ring sdma0 uses VM inv eng 0 on hub 8
[ 9.056932] demo-hp kernel: amdgpu 0000:05:00.0: ring vcn_dec uses VM inv eng 1 on hub 8
[ 9.057198] demo-hp kernel: amdgpu 0000:05:00.0: ring vcn_enc0 uses VM inv eng 4 on hub 8
[ 9.057469] demo-hp kernel: amdgpu 0000:05:00.0: ring vcn_enc1 uses VM inv eng 5 on hub 8
[ 9.057727] demo-hp kernel: amdgpu 0000:05:00.0: ring jpeg_dec uses VM inv eng 6 on hub 8
[ 9.060024] demo-hp kernel: amdgpu: pp_dpm_get_sclk_od was not implemented.
[ 9.060097] demo-hp kernel: amdgpu: pp_dpm_get_mclk_od was not implemented.
[ 9.060121] demo-hp kernel: amdgpu 0000:05:00.0: Runtime PM not available
[ 9.061030] demo-hp kernel: amdgpu 0000:05:00.0: [drm] Registered 4 planes with drm panic
[ 9.068049] demo-hp kernel: [drm] Initialized amdgpu 3.64.0 for 0000:05:00.0 on minor 1
[ 9.081054] demo-hp kernel: amdgpu 0000:05:00.0: [drm] Failed to setup vendor infoframe on connector DP-1: -22
[ 9.086060] demo-hp kernel: fbcon: amdgpudrmfb (fb0) is primary device
[ 9.131047] demo-hp kernel: Console: switching to colour frame buffer device 240x67
[ 9.147045] demo-hp kernel: amdgpu 0000:05:00.0: [drm] fb0: amdgpudrmfb frame buffer device
[ 9.170506] demo-hp systemd[1]: Finished ifupdown2-pre.service - Helper to synchronize boot up for ifupdown.
[ 9.171027] demo-hp systemd[1]: Finished systemd-udev-settle.service - Wait for udev To Complete Device Initialization.
[ 9.171890] demo-hp systemd[1]: zfs-import-cache.service - Import ZFS pools by cache file skipped, unmet condition check ConditionFileNotEmpty=/etc/zfs/zpool.cache
[ 9.172055] demo-hp systemd[1]: Reached target zfs-import.target - ZFS pool import target.
[ 9.179591] demo-hp systemd[1]: Starting zfs-mount.service - Mount ZFS filesystems...
[ 9.181286] demo-hp systemd[1]: Starting zfs-volume-wait.service - Wait for ZFS Volume (zvol) links in /dev...
[ 9.207137] demo-hp zvol_wait[623]: No zvols found, nothing to do.
[ 9.207944] demo-hp systemd[1]: Finished zfs-mount.service - Mount ZFS filesystems.
[ 9.208529] demo-hp systemd[1]: Finished zfs-volume-wait.service - Wait for ZFS Volume (zvol) links in /dev.
[ 9.209473] demo-hp systemd[1]: Reached target local-fs.target - Local File Systems.
[ 9.209606] demo-hp systemd[1]: Reached target zfs-volumes.target - ZFS volumes are ready.
[ 9.211049] demo-hp systemd[1]: Listening on systemd-sysext.socket - System Extension Image Management.
[ 9.212846] demo-hp systemd[1]: Starting apparmor.service - Load AppArmor profiles...
[ 9.214592] demo-hp systemd[1]: Starting console-setup.service - Set console font and keymap...
[ 9.216370] demo-hp systemd[1]: Starting felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit)...
[ 9.218403] demo-hp systemd[1]: Starting networking.service - Network initialization...
[ 9.220586] demo-hp systemd[1]: Starting pvebanner.service - Proxmox VE Login Banner...
[ 9.222876] demo-hp systemd[1]: Starting pvefw-logger.service - Proxmox VE firewall logger...
[ 9.225689] demo-hp systemd[1]: Starting pvenetcommit.service - Commit Proxmox VE network changes...
[ 9.228763] demo-hp systemd[1]: Starting systemd-binfmt.service - Set Up Additional Binary Formats...
[ 9.230821] demo-hp systemd[1]: Starting systemd-tmpfiles-setup.service - Create System Files and Directories...
[ 9.233511] demo-hp systemd[1]: Finished console-setup.service - Set console font and keymap.
[ 9.242393] demo-hp apparmor.systemd[630]: Restarting AppArmor
[ 9.242393] demo-hp apparmor.systemd[630]: Reloading AppArmor profiles
[ 9.248120] demo-hp systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 640 (systemd-binfmt)
[ 9.250462] demo-hp systemd-tmpfiles[641]: /etc/tmpfiles.d/felhom-privsep.conf:15: Duplicate line for path "/run/sshd", ignoring.
[ 9.250689] demo-hp systemd-tmpfiles[641]: /usr/lib/tmpfiles.d/legacy.conf:14: Duplicate line for path "/run/lock", ignoring.
[ 9.251172] demo-hp systemd[1]: Mounting proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System...
[ 9.251836] demo-hp systemd[1]: Finished pvenetcommit.service - Commit Proxmox VE network changes.
[ 9.253408] demo-hp networking[633]: networking: Configuring network interfaces
[ 9.272681] demo-hp pvefw-logger[668]: starting pvefw logger
[ 9.274360] demo-hp systemd[1]: Started pvefw-logger.service - Proxmox VE firewall logger.
[ 9.276036] demo-hp kernel: audit: type=1400 audit(1794321779.088:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="Discord" pid=660 comm="apparmor_parser"
[ 9.276172] demo-hp kernel: audit: type=1400 audit(1794321779.088:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name=4D6F6E676F444220436F6D70617373 pid=661 comm="apparmor_pars
[ 9.276224] demo-hp kernel: audit: type=1400 audit(1794321779.088:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="QtWebEngineProcess" pid=662 comm="apparmor_parser"
[ 9.277127] demo-hp kernel: audit: type=1400 audit(1794321779.089:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="1password" pid=659 comm="apparmor_parser"
[ 9.277255] demo-hp kernel: audit: type=1400 audit(1794321779.090:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="balena-etcher" pid=664 comm="apparmor_parser"
[ 9.277292] demo-hp kernel: audit: type=1400 audit(1794321779.090:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="buildah" pid=666 comm="apparmor_parser"
[ 9.277340] demo-hp kernel: audit: type=1400 audit(1794321779.090:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="brave" pid=665 comm="apparmor_parser"
[ 9.279225] demo-hp kernel: audit: type=1400 audit(1794321779.091:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="cam" pid=672 comm="apparmor_parser"
[ 9.279392] demo-hp kernel: audit: type=1400 audit(1794321779.092:12): apparmor="STATUS" operation="profile_load" profile="unconfined" name="ch-checkns" pid=673 comm="apparmor_parser"
[ 9.279919] demo-hp systemd[1]: Finished systemd-tmpfiles-setup.service - Create System Files and Directories.
[ 9.282035] demo-hp kernel: audit: type=1400 audit(1794321779.094:13): apparmor="STATUS" operation="profile_load" profile="unconfined" name="busybox" pid=671 comm="apparmor_parser"
[ 9.284316] demo-hp systemd[1]: Mounting run-rpc_pipefs.mount - RPC Pipe File System...
[ 9.284460] demo-hp systemd[1]: ldconfig.service - Rebuild Dynamic Linker Cache skipped, no trigger condition checks were met.
[ 9.287648] demo-hp systemd[1]: Starting rpcbind.service - RPC bind portmap service...
[ 9.287715] demo-hp systemd[1]: systemd-firstboot.service - First Boot Wizard skipped, unmet condition check ConditionFirstBoot=yes
[ 9.287753] demo-hp systemd[1]: first-boot-complete.target - First Boot Complete skipped, unmet condition check ConditionFirstBoot=yes
[ 9.287841] demo-hp systemd[1]: systemd-journal-catalog-update.service - Rebuild Journal Catalog skipped, unmet condition check ConditionNeedsUpdate=/var
[ 9.287922] demo-hp systemd[1]: systemd-machine-id-commit.service - Save Transient machine-id to Disk skipped, unmet condition check ConditionPathIsMountPoint=/etc/machine-id
[ 9.288596] demo-hp systemd[1]: systemd-update-done.service - Update is Completed skipped, no trigger condition checks were met.
[ 9.290860] demo-hp systemd[1]: Mounted proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System.
[ 9.303595] demo-hp systemd[1]: Finished systemd-binfmt.service - Set Up Additional Binary Formats.
[ 9.317151] demo-hp systemd[1]: Started rpcbind.service - RPC bind portmap service.
[ 9.317289] demo-hp systemd[1]: Reached target rpcbind.target - RPC Port Mapper.
[ 9.364582] demo-hp felhom-crash-guard[632]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10
[ 9.375481] demo-hp felhom-crash-guard[775]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10
## chrony lines per boot (all boots)
-- boot -5
-- boot -4
Nov 10 02:50:56 demo-hp chronyd[774]: chronyd exiting
Nov 10 02:50:56 demo-hp systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Nov 10 02:50:56 demo-hp systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
-- boot -3
Nov 10 02:54:02 demo-hp systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Nov 10 02:54:02 demo-hp systemd[1]: Started chrony.service - chrony, an NTP client/server.
Nov 10 13:54:26 demo-hp chronyd[778]: chronyd exiting
Nov 10 13:54:26 demo-hp systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Nov 10 13:54:26 demo-hp systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
-- boot -2
Nov 10 13:57:09 demo-hp systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Nov 10 13:57:09 demo-hp systemd[1]: Started chrony.service - chrony, an NTP client/server.
Nov 10 15:30:24 demo-hp chronyd[778]: chronyd exiting
Nov 10 15:30:24 demo-hp systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Nov 10 15:30:24 demo-hp systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
-- boot -1
Nov 10 15:33:04 demo-hp systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Nov 10 15:33:04 demo-hp systemd[1]: Started chrony.service - chrony, an NTP client/server.
Nov 10 15:40:09 demo-hp chronyd[780]: chronyd exiting
Nov 10 15:40:09 demo-hp systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Nov 10 15:40:09 demo-hp systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
-- boot 0
Nov 10 15:42:59 demo-hp systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Nov 10 15:42:59 demo-hp systemd[1]: Started chrony.service - chrony, an NTP client/server.
## sandbox env of pid1
total 0
drwxr-xr-x 20 root root 540 Nov 11 14:07 .
drwxr-xr-x 36 root root 1520 Nov 11 14:11 ..
drwxr-xr-x 2 root root 40 Nov 10 15:42 ask-password
drwxr-xr-x 6 root root 260 Nov 10 15:58 generator
dr-xr-xr-x 3 root root 160 Nov 10 15:42 inaccessible
drwxr-xr-x 2 root root 40 Nov 10 15:42 incoming
drwxr-xr-x 2 root root 40 Nov 10 15:42 inhibit
srw-rw-rw- 1 root root 0 Nov 10 15:42 io.systemd.Credentials
srw-rw-rw- 1 root root 0 Nov 10 15:42 io.systemd.Hostname
srw-rw-rw- 1 root root 0 Nov 10 15:42 io.systemd.ManagedOOM
srw------- 1 root root 0 Nov 10 15:42 io.systemd.sysext
drwxr-xr-x 3 root root 200 Nov 10 15:42 journal
drwxr-xr-x 2 root root 40 Nov 10 15:42 machines
dr-xr-xr-x 2 root root 40 Nov 10 15:42 mount-rootfs
drwxr-xr-x 4 systemd-network systemd-network 80 Nov 10 15:42 netif
srwxrwxrwx 1 root root 0 Nov 10 15:42 notify
srwx------ 1 root root 0 Nov 10 15:42 private
drwxr-xr-x 7 root root 140 Nov 11 14:08 propagate
drwxr-xr-x 2 root root 60 Nov 10 15:42 seats
drwxr-xr-x 2 root root 120 Nov 11 14:11 sessions
drwxr-xr-x 2 root root 40 Nov 10 15:42 shutdown
drwxr-xr-x 2 root root 40 Nov 10 15:42 system
-r--r--r-- 1 root root 0 Nov 10 15:58 systemd-units-load
drwxr-xr-x 2 root root 100 Nov 11 14:10 transient
drwxr-xr-x 2 root root 1940 Nov 11 14:11 units
drwxr-xr-x 2 root root 60 Nov 10 15:42 userdb
drwxr-xr-x 2 root root 60 Nov 11 14:11 users
## tailscale
inactive
## rtc
ls: cannot access '/dev/pps*': No such file or directory
/dev/ptp0
/dev/ptp1
/dev/ptp2
/dev/ptp3
/dev/rtc
/dev/rtc0
## cpu
8
model name : AMD Ryzen Embedded V1756B with Radeon Vega Gfx
@@ -0,0 +1,273 @@
#### host demo-hp 2026-11-11T13:07:29Z local 14:07:29+0100
## versions
pve-manager/9.2.26/41a0e2e09b9e6f9d (running kernel: 7.0.14-28-pve)
ii chrony 4.8-4~bpo13+2
ii systemd 257.13-1~deb13u1
un systemd-timesyncd <none>
7.0.14-28-pve
## unit state
Type=notify
PIDFile=/run/chrony/chronyd.pid
NotifyAccess=main
Result=success
NRestarts=0
ExecMainCode=0
ExecMainStatus=0
ActiveState=active
SubState=running
FragmentPath=/usr/lib/systemd/system/chrony.service
DropInPaths=
UnitFileState=enabled
ActiveEnterTimestamp=Tue 2026-11-10 15:42:59 CET
InactiveEnterTimestamp=
## unit cat
# /usr/lib/systemd/system/chrony.service
[Unit]
Description=chrony, an NTP client/server
Documentation=man:chronyd(8) man:chronyc(1) man:chrony.conf(5)
Conflicts=openntpd.service ntp.service ntpsec.service
ConditionVirtualization=|!container
ConditionVirtualization=|wsl
ConditionCapability=CAP_SYS_TIME
[Service]
Type=notify
PIDFile=/run/chrony/chronyd.pid
EnvironmentFile=-/etc/default/chrony
User=_chrony
# Daemon is started as root, but still sandboxed
ExecStart=!/usr/sbin/chronyd -n $DAEMON_OPTS
CapabilityBoundingSet=~CAP_AUDIT_CONTROL CAP_AUDIT_READ CAP_AUDIT_WRITE
CapabilityBoundingSet=~CAP_BLOCK_SUSPEND CAP_KILL CAP_LEASE CAP_LINUX_IMMUTABLE
CapabilityBoundingSet=~CAP_MAC_ADMIN CAP_MAC_OVERRIDE CAP_MKNOD CAP_SYS_ADMIN
CapabilityBoundingSet=~CAP_SYS_BOOT CAP_SYS_CHROOT CAP_SYS_MODULE CAP_SYS_PACCT
CapabilityBoundingSet=~CAP_SYS_PTRACE CAP_SYS_RAWIO CAP_SYS_TTY_CONFIG CAP_WAKE_ALARM
DeviceAllow=char-pps rw
DeviceAllow=char-ptp rw
DeviceAllow=char-rtc rw
DevicePolicy=closed
LockPersonality=yes
MemoryDenyWriteExecute=yes
NoNewPrivileges=yes
PrivateTmp=yes
ProcSubset=pid
ProtectControlGroups=yes
ProtectHome=yes
ProtectHostname=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectProc=invisible
ProtectSystem=strict
# Used for gps refclocks
ReadWritePaths=/run
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=yes
RestrictSUIDSGID=yes
SystemCallArchitectures=native
SystemCallFilter=~@cpu-emulation @debug @module @mount @obsolete @raw-io @reboot @swap
ConfigurationDirectory=chrony
RuntimeDirectory=chrony
RuntimeDirectoryMode=0700
# See dumpdir in chrony.conf(5)
RuntimeDirectoryPreserve=restart
StateDirectory=chrony
StateDirectoryMode=0750
LogsDirectory=chrony
LogsDirectoryMode=0750
# Adjust restrictions for /usr/sbin/sendmail (mailonchange directive)
NoNewPrivileges=no
ReadWritePaths=-/var/spool
RestrictAddressFamilies=AF_NETLINK
[Install]
Alias=chronyd.service
WantedBy=multi-user.target
## files
ls: cannot access '/etc/systemd/system/chrony.service.d': No such file or directory
ls: cannot access '/etc/systemd/system/chronyd.service.d': No such file or directory
ls: cannot access '/etc/systemd/system/chrony.service': No such file or directory
lrwxrwxrwx 1 root root 38 May 21 13:32 /etc/systemd/system/chronyd.service -> /usr/lib/systemd/system/chrony.service
-rw-r--r-- 1 root root 1929 Jul 1 22:28 /lib/systemd/system/chrony.service
-rw-r--r-- 1 root root 1929 Jul 1 22:28 /usr/lib/systemd/system/chrony.service
## dpkg verify chrony
rc=0
## conf
/etc/chrony:
total 24
drwxr-xr-x 4 root root 4096 May 21 13:32 .
drwxr-xr-x 102 root root 4096 Nov 8 02:49 ..
-rw-r--r-- 1 root root 1450 Feb 16 2026 chrony.conf
-rw-r----- 1 root _chrony 481 Feb 16 2026 chrony.keys
drwxr-xr-x 2 root root 4096 Oct 10 04:40 conf.d
drwxr-xr-x 2 root root 4096 Oct 10 04:40 sources.d
/etc/chrony/conf.d:
total 12
drwxr-xr-x 2 root root 4096 Oct 10 04:40 .
drwxr-xr-x 4 root root 4096 May 21 13:32 ..
-rw-r--r-- 1 root root 422 Feb 16 2026 README
/etc/chrony/sources.d:
total 12
drwxr-xr-x 2 root root 4096 Oct 10 04:40 .
drwxr-xr-x 4 root root 4096 May 21 13:32 ..
-rw-r--r-- 1 root root 498 Feb 16 2026 README
pool 2.debian.pool.ntp.org iburst
sourcedir /run/chrony-dhcp
sourcedir /etc/chrony/sources.d
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
ntsdumpdir /var/lib/chrony
logdir /var/log/chrony
maxupdateskew 100.0
rtcsync
makestep 1 3
leapseclist /usr/share/zoneinfo/leap-seconds.list
confdir /etc/chrony/conf.d
-- /etc/chrony/conf.d/README
Files found under the /etc/chrony/conf.d directory with the .conf suffix are
parsed in the lexicographical order of the file names when chronyd starts up.
This enables a fragmented configuration of chronyd.
Although those files can contain any directives listed in chrony.conf(5),
it would be wiser to add NTP sources in the /etc/chrony/sources.d
directory. Please read /etc/chrony/sources.d/README for more information.
-- /etc/chrony/sources.d/README
Only NTP sources can be specified in the /etc/chrony/sources.d directory.
Files in this directory must end with the ".sources" suffix. They can only
contain the "peer", "pool" and "server" directives and must have all
lines terminated by a trailing newline.
There is no need to restart chronyd for these time sources to be usable,
running 'chronyc reload sources' is sufficient.
Example:
## default
DAEMON_OPTS="-F 1"
## apparmor
/usr/sbin/chronyd
/usr/sbin/chronyd (780)
/usr/sbin/chronyd (903)
-rw-r--r-- 1 root root 0 May 21 13:32 /etc/apparmor.d/local/usr.sbin.chronyd
-rw-r--r-- 1 root root 2837 Jul 1 22:28 /etc/apparmor.d/usr.sbin.chronyd
/etc/apparmor.d/disable:
total 8
drwxr-xr-x 2 root root 4096 Jul 21 2025 .
drwxr-xr-x 9 root root 4096 Nov 8 02:48 ..
## apparmor profile notify lines
81: # Allow systemd Type=notify using sd_notify's $NOTIFY_SOCKET
82: @{run}/systemd/notify w,
## kernel audit lines for chronyd (last 20)
## journal chrony this boot
Nov 10 15:42:59 demo-hp systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Nov 10 15:42:59 demo-hp chronyd[780]: chronyd version 4.8 starting (+CMDMON +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +NTS +SECHASH +IPV6 -DEBUG)
Nov 10 15:42:59 demo-hp chronyd[780]: Loaded 0 symmetric keys
Nov 10 15:42:59 demo-hp chronyd[780]: Using leap second list /usr/share/zoneinfo/leap-seconds.list
Nov 10 15:42:59 demo-hp chronyd[780]: Frequency -12.230 +/- 1.327 ppm read from /var/lib/chrony/chrony.drift
Nov 10 15:42:59 demo-hp chronyd[780]: Loaded seccomp filter (level 1)
Nov 10 15:42:59 demo-hp systemd[1]: Started chrony.service - chrony, an NTP client/server.
Nov 10 15:43:10 demo-hp chronyd[780]: Selected source 81.0.124.253 (2.debian.pool.ntp.org)
Nov 10 15:43:10 demo-hp chronyd[780]: System clock TAI offset set to 37 seconds
Nov 10 15:43:11 demo-hp chronyd[780]: Selected source 37.221.209.24 (2.debian.pool.ntp.org)
Nov 10 15:44:16 demo-hp chronyd[780]: Selected source 195.111.92.55 (2.debian.pool.ntp.org)
Nov 10 15:46:26 demo-hp chronyd[780]: Selected source 37.221.209.24 (2.debian.pool.ntp.org)
## first failure ever in the journal
count: 0
## oldest journal line
Oct 04 21:42:08 demo-hp systemd[779137]: Queued start job for default target def
## last good start in the journal
Nov 10 15:43:11 demo-hp chronyd[780]: Selected source 37.221.209.24 (2.debian.pool.ntp.org)
Nov 10 15:44:16 demo-hp chronyd[780]: Selected source 195.111.92.55 (2.debian.pool.ntp.org)
Nov 10 15:46:26 demo-hp chronyd[780]: Selected source 37.221.209.24 (2.debian.pool.ntp.org)
## boots
-5 182af91122a941ec97ac7ee247a4d470 Sun 2026-10-04 21:42:08 CEST Sun 2026-10-04 21:43:33 CEST
-4 bb68aee4240244718d8a7f39d999a3a8 Wed 2026-11-04 21:06:10 CET Tue 2026-11-10 02:53:28 CET
-3 b6e0edea2b5647dfa36f5bb3825adc3a Tue 2026-11-10 02:53:57 CET Tue 2026-11-10 13:56:35 CET
-2 d1167d9431bf459abf2a569e8997a569 Tue 2026-11-10 13:57:04 CET Tue 2026-11-10 15:32:31 CET
-1 e2cc9e3e60f14bde8949f01d7ba35ccf Tue 2026-11-10 15:32:59 CET Tue 2026-11-10 15:42:26 CET
0 bed106f79e804eb1ace314a26da7f049 Tue 2026-11-10 15:42:54 CET Wed 2026-11-11 14:07:29 CET
## what starts with it
chrony.service
● ├─-.mount
● ├─system.slice
● ├─systemd-journald.socket
● ├─systemd-remount-fs.service
● ├─systemd-tmpfiles-setup.service
● ├─tmp.mount
● ├─basic.target
● │ ├─-.mount
○ │ ├─systemd-pcrphase-sysinit.service
● │ ├─tmp.mount
● │ ├─paths.target
● │ │ ├─systemd-ask-password-console.path
● │ │ └─systemd-ask-password-wall.path
● │ ├─slices.target
● │ │ ├─-.slice
● │ │ ├─system-xfs_scrub.slice
● │ │ ├─system.slice
● │ │ └─user.slice
● │ ├─sockets.target
● │ │ ├─dbus.socket
● │ │ ├─iscsid.socket
● │ │ ├─netavark-dhcp-proxy.socket
● │ │ ├─rrdcached.socket
● │ │ ├─sshd-unix-local.socket
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.
chrony.service +174ms
└─basic.target @5.760s
└─sockets.target @5.760s
└─systemd-hostnamed.socket @5.760s
└─sysinit.target @5.757s
└─felhom-crash-guard.service @5.583s +173ms
└─local-fs.target @5.576s
└─etc-pve.mount @7.233s
└─local-fs-pre.target @761ms
## other time services
inactive inactive inactive inactive
Local time: Wed 2026-11-11 14:07:32 CET
Universal time: Wed 2026-11-11 13:07:32 UTC
RTC time: Wed 2026-11-11 13:07:33
Time zone: Europe/Budapest (CET, +0100)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
## chronyc
Reference ID : 25DDD118 (37.221.209.24)
Stratum : 3
Ref time (UTC) : Wed Nov 11 12:51:20 2026
System time : 0.000228395 seconds fast of NTP time
Last offset : +0.000003178 seconds
RMS offset : 0.000066409 seconds
Frequency : 12.927 ppm slow
Residual freq : +0.000 ppm
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^- cebece.pomaz.hu 2 10 377 978 -7115us[-7112us] +/- 32ms
^* 37.221.209.24 2 10 377 971 -2744us[-2741us] +/- 15ms
^+ mail.deployis.eu 4 10 377 758 -3122us[-3122us] +/- 24ms
^+ ntp2.niif.hu 2 10 377 377 -4436us[-4436us] +/- 39ms
## dpkg history chrony/apparmor/systemd
2026-08-21 15:41:16 install apparmor:amd64 <none> 4.1.1-pmx1
2026-08-21 15:42:37 configure apparmor:amd64 4.1.1-pmx1 <none>
2026-10-10 04:40:04 upgrade chrony:amd64 4.6.1-3+deb13u1 4.8-4~bpo13+2
2026-10-10 04:40:23 configure chrony:amd64 4.8-4~bpo13+2 <none>
## felhom files that name chrony
## ps
780 _chrony /usr/sbin/chronyd -n -F 1
903 _chrony /usr/sbin/chronyd -n -F 1
## run dir
srwxrwxrwx 1 root root 0 Nov 10 15:42 /run/systemd/notify
/run/chrony:
total 4
drwx------ 2 _chrony _chrony 80 Nov 11 14:07 .
drwxr-xr-x 36 root root 1520 Nov 11 14:07 ..
-rw-r--r-- 1 root root 4 Nov 10 15:42 chronyd.pid
srwxr-xr-x 1 _chrony _chrony 0 Nov 10 15:42 chronyd.sock
## uptime
2026-11-10 15:42:49
@@ -0,0 +1,26 @@
#### demo-hp 2026-11-11T13:16:38Z
## fresh compile of the on-disk profile to stdout (no load, cache not read, not written)
83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1 -
## loaded
83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1
## cache
dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 /var/cache/apparmor/ae1a4050.0/usr.sbin.chronyd
83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1 /var/cache/apparmor/f6da0254.0/usr.sbin.chronyd
drwx------ 2 root root 4096 2026-08-21 17:44:37.686947147 +0200 /var/cache/apparmor/ae1a4050.0/
-rw-r--r-- 1 root root 3177 2026-08-21 17:44:37.159696460 +0200 /var/cache/apparmor/ae1a4050.0/.features
drwx------ 2 root root 4096 2026-11-08 02:48:51.017396841 +0100 /var/cache/apparmor/f6da0254.0/
-rw-r--r-- 1 root root 3219 2026-10-04 09:44:36.115533363 +0200 /var/cache/apparmor/f6da0254.0/.features
## state
active
chrony 4.8-4~bpo13+2
7.0.14-28-pve
System time : 0.000003214 seconds slow of NTP time
Leap status : Normal
1794402998.136055491
## chrony fatal lines
0
## stale check, every cached profile: cache differs from a fresh compile
current dir: /var/cache/apparmor/f6da0254.0/
STALE: lxc-containers
checked=106 stale=1
@@ -0,0 +1,76 @@
#### demo-felhom 2026-11-11T13:15:31Z
## fresh compile of the on-disk profile, to stdout only (no load, no cache write)
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -
apparmor_parser: Too many actions given on the command line.
## loaded raw policy sha256
dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647
## cache dirs
total 16
drwxr-xr-x 4 root root 4096 2026-10-07 12:11:32.040268492 +0200 .
drwxr-xr-x 14 root root 4096 2026-07-19 13:08:02.082082737 +0200 ..
drwx------ 2 root root 4096 2026-10-07 14:16:03.051992948 +0200 ae1a4050.0
drwx------ 2 root root 4096 2026-11-08 05:32:24.065142783 +0100 f6da0254.0
/var/cache/apparmor/ae1a4050.0/ features-sha=0e6b37757d22af1e files=106
-rw-r--r-- 1 root root 3177 2026-07-18 18:17:02.251100923 +0200 /var/cache/apparmor/ae1a4050.0//.features
-rw------- 1 root root 296497 2026-07-18 18:15:53.043202621 +0200 /var/cache/apparmor/ae1a4050.0//usr.sbin.chronyd
/var/cache/apparmor/f6da0254.0/ features-sha=611413ba507a5c19 files=106
-rw-r--r-- 1 root root 3219 2026-10-07 12:11:32.040268492 +0200 /var/cache/apparmor/f6da0254.0//.features
-rw------- 1 root root 295529 2026-07-18 18:15:53.043202621 +0200 /var/cache/apparmor/f6da0254.0//usr.sbin.chronyd
## kernel features now
yes yes yes yes yes
611413ba507a5c19
## newest source file the profile depends on
2025-07-21 12:21:49.000000000 /etc/apparmor.d/abi/kernel-5.4-vanilla
2025-07-21 12:21:49.000000000 /etc/apparmor.d/abstractions/base
2025-07-21 12:21:49.000000000 /etc/apparmor.d/abstractions/nameservice
2025-07-21 12:21:49.000000000 /etc/apparmor.d/tunables/global
2026-05-21 13:32:03.000000000 /etc/apparmor.d/local/usr.sbin.chronyd
2026-07-01 22:28:16.000000000 /etc/apparmor.d/usr.sbin.chronyd
## files in /etc/apparmor.d changed 2026-07-18 or 2026-08-21
/etc/apparmor.d/tunables
/etc/apparmor.d/tunables/home.d
/etc/apparmor.d/tunables/home.d/ubuntu
/etc/apparmor.d/tunables/multiarch.d
/etc/apparmor.d/tunables/xdg-user-dirs.d
/etc/apparmor.d/tunables/xdg-user-dirs.d/site.local
/etc/apparmor.d/lxc
/etc/apparmor.d/local
/etc/apparmor.d/local/lsb_release
/etc/apparmor.d/local/usr.bin.swtpm
## versions
apparmor 4.1.1-pmx1
chrony 4.8-4~bpo13+2
AppArmor parser version 4.1.1
## dpkg log
2026-07-18 16:14:58 install apparmor:amd64 <none> 4.1.1-pmx1
2026-07-18 16:15:24 install proxmox-kernel-7.0.2-6-pve-signed:amd64 <none> 7.0.2-6
2026-10-07 12:09:03 install proxmox-kernel-7.0.14-20-pve-signed:amd64 <none> 7.0.14-20
2026-10-07 14:15:40 upgrade chrony:amd64 4.6.1-3+deb13u1 4.8-4~bpo13+2
2026-10-09 04:44:23 install proxmox-kernel-7.0.14-22-pve-signed:amd64 <none> 7.0.14-22
2026-10-10 04:44:06 install proxmox-kernel-7.0.14-23-pve-signed:amd64 <none> 7.0.14-23
2026-10-16 05:06:43 install proxmox-kernel-7.0.14-24-pve-signed:amd64 <none> 7.0.14-24
2026-10-21 05:21:10 install proxmox-kernel-7.0.14-25-pve-signed:amd64 <none> 7.0.14-25
2026-10-29 04:45:31 install proxmox-kernel-7.0.14-26-pve-signed:amd64 <none> 7.0.14-26
2026-11-08 05:32:59 install proxmox-kernel-7.0.14-28-pve-signed:amd64 <none> 7.0.14-28
## kernels booted
IDX BOOT ID FIRST ENTRY LAST ENTRY
-5 46081f567f384ab88591d4e604555a48 Sun 2026-10-04 21:43:33 CEST Sun 2026-10-04 21:43:46 CEST
-4 e82176ae7cf644518507468fc609eaa9 Wed 2026-10-14 15:43:07 CEST Fri 2026-10-16 05:09:06 CEST
-3 ddd3d7305e244b2a9eafb5cf229476f1 Fri 2026-10-16 05:09:23 CEST Wed 2026-10-21 05:23:34 CEST
-2 56db71938b304a57bdedf30f89860e1f Wed 2026-10-21 05:23:48 CEST Thu 2026-10-29 04:47:58 CET
-1 4f6d8b388e224b8ca8ed45e06a2f96b9 Thu 2026-10-29 04:48:11 CET Sun 2026-11-08 05:35:13 CET
0 72899d2625244e069e8e8099775d1e51 Sun 2026-11-08 05:35:21 CET Wed 2026-11-11 14:15:31 CET
docker0: port 1(vethbb70052) entered disabled state
Linux version 7.0.14-24-pve (build@proxmox) (gcc (Debian 14.
Linux version 7.0.14-25-pve (build@proxmox) (gcc (Debian 14.
Linux version 7.0.14-26-pve (build@proxmox) (gcc (Debian 14.
Linux version 7.0.14-28-pve (build@proxmox) (gcc (Debian 14.
## chronyd confinement now
no chronyd process
## what installed chrony 4.8
Start-Date: 2026-10-07 14:15:37
Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --only-upgrade --no-install-recommends frr=10.6.1-1+pve3 librgw2=19.2.6-pve4 libradosstriper1=19.2.6-pve4 ceph-common=19.2.6-pve4 librbd1=19.2.6-pve4 libra
Requested-By: felhom-agent (999)
Upgrade: librados2:amd64 (19.2.3-pve4, 19.2.6-pve4), pve-docs:amd64 (9.2.1, 9.2.13), ceph-fuse:amd64 (19.2.3-pve4, 19.2.6-pve4), proxmox-widget-toolkit:amd64 (5.2.2, 5.2.10), corosync:amd64 (3.1.10-pve2, 3.1.10-pve3), frr:amd64 (10.6.1-1+pve2, 10.6.1-1+pve3),
Start-Date: 2026-10-08 04:37:54
@@ -0,0 +1,236 @@
#### demo-felhom 2026-11-11T13:08:39Z
net.unix.max_dgram_qlen = 512
kernel.printk_ratelimit = 5
## cmdline
BOOT_IMAGE=/boot/vmlinuz-7.0.14-28-pve root=/dev/mapper/pve-root ro quiet softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10
## apparmor denied this boot: 36
Nov 08 05:35:31 demo-felhom kernel: audit: type=1400 audit(1794112531.870:128): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=1529 comm="vgs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
Nov 08 05:35:31 demo-felhom kernel: audit: type=1400 audit(1794112531.898:129): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=1530 comm="lvs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
Nov 08 05:35:31 demo-felhom kernel: audit: type=1400 audit(1794112531.979:130): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=1536 comm="vgs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
Nov 08 05:35:32 demo-felhom kernel: audit: type=1400 audit(1794112532.008:131): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=1537 comm="lvs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
Nov 08 06:06:23 demo-felhom kernel: audit: type=1400 audit(1794114383.198:133): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=43146 comm="vgs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
## audit lines with chrony, all boots
Oct 16 05:09:25 demo-felhom kernel: audit: type=1400 audit(1792120165.400:121): apparmor="DENIED" operation="sendmsg" class="file" profile="/usr/sbin/chronyd" name="/run/systemd/notify" pid=692 comm="chronyd" requested_mask="w" denied_mask="w" fsuid=100 ouid=0
## the second around the chrony start, this boot
start: 2026-11-08T05:35:23+01:00
[ 3.980022] demo-felhom kernel: intel_pmc_core INT33A1:00: initialized
[ 3.988439] demo-felhom systemd-modules-load[310]: Inserted module 'zfs'
[ 3.989049] demo-felhom kernel: ZFS: Loaded module v2.4.4-pve1, ZFS pool version 5000, ZFS filesystem version 5
[ 3.989820] demo-felhom systemd[1]: Finished systemd-modules-load.service - Load Kernel Modules.
[ 3.991237] demo-felhom systemd[1]: Starting systemd-sysctl.service - Apply Kernel Variables...
[ 4.004023] demo-felhom kernel: usbcore: registered new interface driver btusb
[ 4.011027] demo-felhom kernel: Bluetooth: hci0: RTL: examining hci_ver=0b hci_rev=000b lmp_ver=0b lmp_subver=8852
[ 4.011156] demo-felhom kernel: Bluetooth: hci0: RTL: rom_version status=0 version=1
[ 4.011353] demo-felhom systemd[1]: Starting systemd-rfkill.service - Load/Save RF Kill Switch Status...
[ 4.013369] demo-felhom kernel: Bluetooth: hci0: RTL: btrtl_initialize: key id 0
[ 4.013474] demo-felhom kernel: Bluetooth: hci0: RTL: loading rtl_bt/rtl8852bu_fw.bin
[ 4.018168] demo-felhom systemd[1]: Reached target bluetooth.target - Bluetooth Support.
[ 4.058030] demo-felhom kernel: Bluetooth: hci0: RTL: loading rtl_bt/rtl8852bu_config.bin
[ 4.060122] demo-felhom kernel: Bluetooth: hci0: RTL: cfg_sz 6, total sz 62539
[ 4.069224] demo-felhom systemd[1]: Started systemd-rfkill.service - Load/Save RF Kill Switch Status.
[ 4.072229] demo-felhom systemd[1]: Finished systemd-sysctl.service - Apply Kernel Variables.
[ 4.094018] demo-felhom kernel: usb 1-6: Warning! Unlikely big volume range (=511), cval->res is probably wrong.
[ 4.094444] demo-felhom kernel: usb 1-6: [5] FU [Mic Capture Volume] ch = 1, val = -8192/-1/16
[ 4.100058] demo-felhom kernel: usb 1-6: Warning! Unlikely big volume range (=896), cval->res is probably wrong.
[ 4.100362] demo-felhom kernel: usb 1-6: [6] FU [PCM Playback Volume] ch = 2, val = -16384/-2048/16
[ 4.100548] demo-felhom kernel: usbcore: registered new interface driver snd-usb-audio
[ 4.106727] demo-felhom systemd[1]: Reached target sound.target - Sound Card.
[ 4.116015] demo-felhom kernel: rtw89_8852be 0000:02:00.0: loaded firmware rtw89/rtw8852b_fw-1.bin
[ 4.116308] demo-felhom kernel: rtw89_8852be 0000:02:00.0: enabling device (0000 -> 0003)
[ 4.124022] demo-felhom kernel: rtw89_8852be 0000:02:00.0: Firmware version 0.29.29.15 (6fb3ec41), cmd version 0, type 5
[ 4.124318] demo-felhom kernel: rtw89_8852be 0000:02:00.0: Firmware version 0.29.29.15 (6fb3ec41), cmd version 0, type 3
[ 4.137907] demo-felhom systemd[1]: Mounting boot-efi.mount - /boot/efi...
[ 4.139530] demo-felhom systemd[1]: mnt-hdd_1.mount: Directory /mnt/hdd_1 to mount over is not empty, mounting anyway.
[ 4.141158] demo-felhom systemd[1]: Mounting mnt-hdd_1.mount - Felhom storage mount 47a3361a-91e0-4831-a69d-27f540ed3f48...
[ 4.142916] demo-felhom systemd[1]: Mounting tmp.mount - Temporary Directory /tmp...
[ 4.267728] demo-felhom systemd[1]: Mounted tmp.mount - Temporary Directory /tmp.
[ 4.268510] demo-felhom systemd[1]: Mounted boot-efi.mount - /boot/efi.
[ 4.421015] demo-felhom kernel: Bluetooth: hci0: RTL: fw version 0x127cfd78
[ 4.475042] demo-felhom kernel: rtw89_8852be 0000:02:00.0: chip info CID: 0, CV: 1, AID: 0, ACV: 1, RFE: 5
[ 4.476021] demo-felhom kernel: rtw89_8852be 0000:02:00.0: rfkill hardware state changed to enable
[ 4.494046] demo-felhom kernel: intel_tcc_cooling: TCC Offset locked
[ 4.499095] demo-felhom kernel: rtw89_8852be 0000:02:00.0 wlp2s0: renamed from wlan0
[ 4.499354] demo-felhom kernel: i915 0000:00:02.0: [drm] Found alderlake_p/alderlake_n (device ID 46d1) integrated display version 13.00 stepping D0
[ 4.499561] demo-felhom kernel: i915 0000:00:02.0: [drm] VT-d active for gfx access
[ 4.499846] demo-felhom kernel: i915 0000:00:02.0: vgaarb: deactivate vga console
[ 4.500020] demo-felhom kernel: i915 0000:00:02.0: [drm] Using Transparent Hugepages
[ 4.501210] demo-felhom kernel: i915 0000:00:02.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[ 4.505030] demo-felhom kernel: i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adlp_dmc.bin (v2.20)
[ 4.505306] demo-felhom kernel: Bluetooth: hci0: AOSP extensions version v1.00
[ 4.515016] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: GuC firmware i915/tgl_guc_70.bin version 70.49.4
[ 4.515287] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: HuC firmware i915/tgl_huc.bin version 7.9.3
[ 4.517022] demo-felhom kernel: intel_rapl_msr: PL4 support detected.
[ 4.517113] demo-felhom kernel: intel_rapl_common: Found RAPL domain package
[ 4.517134] demo-felhom kernel: intel_rapl_common: Found RAPL domain core
[ 4.517152] demo-felhom kernel: intel_rapl_common: Found RAPL domain uncore
[ 4.520023] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: HuC: authenticated for all workloads
[ 4.520252] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: GUC: submission enabled
[ 4.520417] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: GUC: SLPC enabled
[ 4.520574] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: GUC: RC enabled
[ 4.522017] demo-felhom kernel: i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
[ 4.522233] demo-felhom kernel: i915 0000:00:02.0: [drm] Registered 3 planes with drm panic
[ 4.530018] demo-felhom kernel: [drm] Initialized i915 1.6.0 for 0000:00:02.0 on minor 0
[ 4.531012] demo-felhom kernel: ACPI: video: Video Device [GFX0] (multi-head: yes rom: no post: no)
[ 4.531087] demo-felhom kernel: input: Video Bus as /devices/pci0000:00/acpi.video_bus.0/input/input4
[ 4.536037] demo-felhom kernel: EXT4-fs (sdb): mounted filesystem 47a3361a-91e0-4831-a69d-27f540ed3f48 r/w with ordered data mode. Quota mode: none.
[ 4.537227] demo-felhom systemd[1]: Mounted mnt-hdd_1.mount - Felhom storage mount 47a3361a-91e0-4831-a69d-27f540ed3f48.
[ 4.545015] demo-felhom kernel: i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[ 4.569019] demo-felhom kernel: i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[ 4.581021] demo-felhom kernel: i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[ 4.610025] demo-felhom kernel: snd_hda_intel 0000:00:1f.3: enabling device (0000 -> 0002)
[ 4.610349] demo-felhom kernel: snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops intel_audio_component_bind_ops [i915])
[ 4.917022] demo-felhom kernel: input: HDA Intel PCH HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1f.3/sound/card1/input5
[ 4.917152] demo-felhom kernel: input: HDA Intel PCH HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:1f.3/sound/card1/input6
[ 4.917202] demo-felhom kernel: input: HDA Intel PCH HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:1f.3/sound/card1/input7
[ 4.917241] demo-felhom kernel: input: HDA Intel PCH HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:1f.3/sound/card1/input8
[ 4.940127] demo-felhom systemd[1]: Finished ifupdown2-pre.service - Helper to synchronize boot up for ifupdown.
[ 4.943663] demo-felhom systemd[1]: Finished systemd-udev-settle.service - Wait for udev To Complete Device Initialization.
[ 4.943871] demo-felhom systemd[1]: zfs-import-cache.service - Import ZFS pools by cache file skipped, unmet condition check ConditionFileNotEmpty=/etc/zfs/zpool.cache
[ 4.943900] demo-felhom systemd[1]: Reached target zfs-import.target - ZFS pool import target.
[ 4.945492] demo-felhom systemd[1]: Starting zfs-mount.service - Mount ZFS filesystems...
[ 4.946429] demo-felhom systemd[1]: Starting zfs-volume-wait.service - Wait for ZFS Volume (zvol) links in /dev...
[ 4.966316] demo-felhom systemd[1]: Finished zfs-mount.service - Mount ZFS filesystems.
[ 4.967027] demo-felhom systemd[1]: Finished zfs-volume-wait.service - Wait for ZFS Volume (zvol) links in /dev.
[ 4.968549] demo-felhom zvol_wait[531]: No zvols found, nothing to do.
[ 4.967458] demo-felhom systemd[1]: Reached target local-fs.target - Local File Systems.
[ 4.967518] demo-felhom systemd[1]: Reached target zfs-volumes.target - ZFS volumes are ready.
[ 4.969218] demo-felhom systemd[1]: Listening on systemd-sysext.socket - System Extension Image Management.
[ 4.970224] demo-felhom systemd[1]: Starting apparmor.service - Load AppArmor profiles...
[ 4.971086] demo-felhom systemd[1]: Starting console-setup.service - Set console font and keymap...
[ 4.972029] demo-felhom systemd[1]: Starting felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit)...
[ 4.973401] demo-felhom systemd[1]: Starting networking.service - Network initialization...
[ 4.977223] demo-felhom systemd[1]: Starting pvebanner.service - Proxmox VE Login Banner...
[ 4.978444] demo-felhom systemd[1]: Starting pvefw-logger.service - Proxmox VE firewall logger...
[ 4.979795] demo-felhom systemd[1]: Starting pvenetcommit.service - Commit Proxmox VE network changes...
[ 4.983183] demo-felhom systemd[1]: Starting systemd-binfmt.service - Set Up Additional Binary Formats...
[ 4.985507] demo-felhom systemd[1]: Starting systemd-tmpfiles-setup.service - Create System Files and Directories...
[ 4.986954] demo-felhom systemd[1]: Finished console-setup.service - Set console font and keymap.
[ 4.998578] demo-felhom systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 547 (systemd-binfmt)
[ 4.999914] demo-felhom systemd[1]: Mounting proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System...
[ 5.000113] demo-felhom systemd-tmpfiles[550]: /etc/tmpfiles.d/felhom-privsep.conf:15: Duplicate line for path "/run/sshd", ignoring.
[ 5.000258] demo-felhom systemd-tmpfiles[550]: /usr/lib/tmpfiles.d/legacy.conf:14: Duplicate line for path "/run/lock", ignoring.
[ 5.001716] demo-felhom systemd[1]: Finished pvenetcommit.service - Commit Proxmox VE network changes.
[ 5.003282] demo-felhom apparmor.systemd[538]: Restarting AppArmor
[ 5.003282] demo-felhom apparmor.systemd[538]: Reloading AppArmor profiles
[ 5.021348] demo-felhom networking[541]: networking: Configuring network interfaces
[ 5.022383] demo-felhom systemd[1]: Mounted proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System.
[ 5.025891] demo-felhom systemd[1]: Finished systemd-binfmt.service - Set Up Additional Binary Formats.
[ 5.048314] demo-felhom pvefw-logger[573]: starting pvefw logger
[ 5.048548] demo-felhom systemd[1]: Started pvefw-logger.service - Proxmox VE firewall logger.
[ 5.049951] demo-felhom systemd[1]: Finished systemd-tmpfiles-setup.service - Create System Files and Directories.
[ 5.050310] demo-felhom systemd[1]: ldconfig.service - Rebuild Dynamic Linker Cache skipped, no trigger condition checks were met.
[ 5.055441] demo-felhom systemd[1]: Starting rpcbind.service - RPC bind portmap service...
[ 5.055478] demo-felhom systemd[1]: systemd-firstboot.service - First Boot Wizard skipped, unmet condition check ConditionFirstBoot=yes
[ 5.055507] demo-felhom systemd[1]: first-boot-complete.target - First Boot Complete skipped, unmet condition check ConditionFirstBoot=yes
[ 5.055569] demo-felhom systemd[1]: systemd-journal-catalog-update.service - Rebuild Journal Catalog skipped, unmet condition check ConditionNeedsUpdate=/var
[ 5.055622] demo-felhom systemd[1]: systemd-machine-id-commit.service - Save Transient machine-id to Disk skipped, unmet condition check ConditionPathIsMountPoint=/etc/machine-id
[ 5.055680] demo-felhom systemd[1]: systemd-update-done.service - Update is Completed skipped, no trigger condition checks were met.
[ 5.069037] demo-felhom kernel: audit: type=1400 audit(1794112523.118:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name=4D6F6E676F444220436F6D70617373 pid=572 comm="apparmor_
[ 5.069130] demo-felhom kernel: audit: type=1400 audit(1794112523.118:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="QtWebEngineProcess" pid=574 comm="apparmor_parser"
[ 5.069157] demo-felhom kernel: audit: type=1400 audit(1794112523.118:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="1password" pid=570 comm="apparmor_parser"
[ 5.069183] demo-felhom kernel: audit: type=1400 audit(1794112523.119:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="Discord" pid=571 comm="apparmor_parser"
[ 5.072026] demo-felhom kernel: audit: type=1400 audit(1794112523.123:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="brave" pid=581 comm="apparmor_parser"
[ 5.073026] demo-felhom kernel: audit: type=1400 audit(1794112523.124:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="busybox" pid=583 comm="apparmor_parser"
[ 5.074325] demo-felhom kernel: audit: type=1400 audit(1794112523.125:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="balena-etcher" pid=580 comm="apparmor_parser"
[ 5.074387] demo-felhom kernel: audit: type=1400 audit(1794112523.125:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="buildah" pid=582 comm="apparmor_parser"
[ 5.074411] demo-felhom kernel: audit: type=1400 audit(1794112523.125:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="cam" pid=584 comm="apparmor_parser"
[ 5.075909] demo-felhom systemd[1]: Started rpcbind.service - RPC bind portmap service.
[ 5.076068] demo-felhom systemd[1]: Reached target rpcbind.target - RPC Port Mapper.
[ 5.119035] demo-felhom kernel: kauditd_printk_skb: 40 callbacks suppressed
[ 5.119141] demo-felhom kernel: audit: type=1400 audit(1794112523.170:51): apparmor="STATUS" operation="profile_load" profile="unconfined" name="opera" pid=624 comm="apparmor_parser"
[ 5.120172] demo-felhom kernel: audit: type=1400 audit(1794112523.171:52): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default" pid=610 comm="apparmor_parser
[ 5.120257] demo-felhom kernel: audit: type=1400 audit(1794112523.171:53): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-cgns" pid=610 comm="apparmor_p
[ 5.120282] demo-felhom kernel: audit: type=1400 audit(1794112523.171:54): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-mounting" pid=610 comm="a
[ 5.120308] demo-felhom kernel: audit: type=1400 audit(1794112523.171:55): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-nesting" pid=610 comm="ap
[ 5.120329] demo-felhom kernel: audit: type=1400 audit(1794112523.172:56): apparmor="STATUS" operation="profile_load" profile="unconfined" name="polypane" pid=627 comm="apparmor_parser"
[ 5.121021] demo-felhom kernel: audit: type=1400 audit(1794112523.172:57): apparmor="STATUS" operation="profile_load" profile="unconfined" name="pageedit" pid=625 comm="apparmor_parser"
[ 5.124025] demo-felhom kernel: audit: type=1400 audit(1794112523.175:58): apparmor="STATUS" operation="profile_load" profile="unconfined" name="privacybrowser" pid=628 comm="apparmor_parser"
[ 5.124103] demo-felhom kernel: audit: type=1400 audit(1794112523.176:59): apparmor="STATUS" operation="profile_load" profile="unconfined" name="qcam" pid=630 comm="apparmor_parser"
[ 5.126023] demo-felhom kernel: audit: type=1400 audit(1794112523.177:60): apparmor="STATUS" operation="profile_load" profile="unconfined" name="pve-container-mounthotplug" pid=629 comm="apparmor_p
[ 5.133470] demo-felhom felhom-crash-guard[540]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10
[ 5.138735] demo-felhom systemd[1]: Mounting run-rpc_pipefs.mount - RPC Pipe File System...
[ 5.153807] demo-felhom felhom-crash-guard[665]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10
[ 5.162616] demo-felhom systemd[1]: Finished felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit).
[ 5.199068] demo-felhom kernel: RPC: Registered named UNIX socket transport module.
[ 5.199201] demo-felhom kernel: RPC: Registered udp transport module.
[ 5.199231] demo-felhom kernel: RPC: Registered tcp transport module.
[ 5.199252] demo-felhom kernel: RPC: Registered tcp-with-tls transport module.
[ 5.199278] demo-felhom kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 5.201359] demo-felhom systemd[1]: Mounted run-rpc_pipefs.mount - RPC Pipe File System.
[ 5.201831] demo-felhom systemd[1]: Reached target rpc_pipefs.target.
[ 5.204008] demo-felhom systemd[1]: Starting nfs-blkmap.service - pNFS block layout mapping daemon...
[ 5.204272] demo-felhom systemd[1]: rpc-gssd.service - RPC security service for NFS client and server skipped, unmet condition check ConditionPathExists=/etc/krb5.keytab
[ 5.204309] demo-felhom systemd[1]: Reached target nfs-client.target - NFS client services.
[ 5.207340] demo-felhom systemd[1]: Finished apparmor.service - Load AppArmor profiles.
[ 5.207747] demo-felhom systemd[1]: Reached target sysinit.target - System Initialization.
## chrony lines per boot (all boots)
-- boot -5
-- boot -4
-- boot -3
Oct 16 05:09:25 demo-felhom systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Oct 16 05:09:25 demo-felhom chronyd[692]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Oct 16 05:09:25 demo-felhom chronyd[692]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Oct 16 05:09:25 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE
Oct 16 05:09:25 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'.
Oct 16 05:09:25 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.
-- boot -2
Oct 21 05:23:50 demo-felhom systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Oct 21 05:23:50 demo-felhom chronyd[676]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Oct 21 05:23:50 demo-felhom chronyd[676]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Oct 21 05:23:50 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE
Oct 21 05:23:50 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'.
Oct 21 05:23:50 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.
-- boot -1
Oct 29 04:48:13 demo-felhom systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Oct 29 04:48:14 demo-felhom chronyd[682]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Oct 29 04:48:14 demo-felhom chronyd[682]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Oct 29 04:48:14 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE
Oct 29 04:48:14 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'.
Oct 29 04:48:14 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.
-- boot 0
Nov 08 05:35:23 demo-felhom systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE
Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'.
Nov 08 05:35:23 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.
## sandbox env of pid1
total 0
drwxr-xr-x 21 root root 560 Nov 11 14:06 .
drwxr-xr-x 36 root root 1520 Nov 11 14:08 ..
drwxr-xr-x 2 root root 40 Nov 8 05:35 ask-password
drwx------ 2 root root 60 Nov 8 21:45 ask-password-block
drwxr-xr-x 6 root root 260 Nov 10 15:50 generator
dr-xr-xr-x 3 root root 160 Nov 8 05:35 inaccessible
drwxr-xr-x 2 root root 40 Nov 8 05:35 incoming
drwxr-xr-x 2 root root 40 Nov 9 05:33 inhibit
srw-rw-rw- 1 root root 0 Nov 8 05:35 io.systemd.Credentials
srw-rw-rw- 1 root root 0 Nov 8 05:35 io.systemd.Hostname
srw-rw-rw- 1 root root 0 Nov 8 05:35 io.systemd.ManagedOOM
srw------- 1 root root 0 Nov 8 05:35 io.systemd.sysext
drwxr-xr-x 3 root root 200 Nov 8 05:35 journal
drwxr-xr-x 2 root root 40 Nov 8 05:35 machines
dr-xr-xr-x 2 root root 40 Nov 8 05:35 mount-rootfs
drwxr-xr-x 4 systemd-network systemd-network 80 Nov 8 05:35 netif
srwxrwxrwx 1 root root 0 Nov 8 05:35 notify
srwx------ 1 root root 0 Nov 8 05:35 private
drwxr-xr-x 6 root root 120 Nov 11 14:07 propagate
drwxr-xr-x 2 root root 60 Nov 8 05:35 seats
drwxr-xr-x 2 root root 120 Nov 11 14:08 sessions
drwxr-xr-x 2 root root 40 Nov 8 05:35 shutdown
drwxr-xr-x 2 root root 40 Nov 8 05:35 system
-r--r--r-- 1 root root 0 Nov 10 15:50 systemd-units-load
drwxr-xr-x 2 root root 60 Nov 11 14:08 transient
drwxr-xr-x 2 root root 1880 Nov 11 14:09 units
drwxr-xr-x 2 root root 60 Nov 8 05:35 userdb
drwxr-xr-x 2 root root 60 Nov 11 14:08 users
## tailscale
active
## rtc
ls: cannot access '/dev/ptp*': No such file or directory
ls: cannot access '/dev/pps*': No such file or directory
/dev/rtc
/dev/rtc0
## cpu
4
model name : Intel(R) N100
@@ -0,0 +1,250 @@
#### host demo-felhom 2026-11-11T13:06:53Z local 14:06:53+0100
## versions
pve-manager/9.2.26/41a0e2e09b9e6f9d (running kernel: 7.0.14-28-pve)
ii chrony 4.8-4~bpo13+2
ii systemd 257.13-1~deb13u1
un systemd-timesyncd <none>
7.0.14-28-pve
## unit state
Type=notify
PIDFile=/run/chrony/chronyd.pid
NotifyAccess=main
Result=exit-code
NRestarts=0
ExecMainCode=1
ExecMainStatus=1
ActiveState=failed
SubState=failed
FragmentPath=/usr/lib/systemd/system/chrony.service
DropInPaths=
UnitFileState=enabled
ActiveEnterTimestamp=
InactiveEnterTimestamp=Sun 2026-11-08 05:35:23 CET
## unit cat
# /usr/lib/systemd/system/chrony.service
[Unit]
Description=chrony, an NTP client/server
Documentation=man:chronyd(8) man:chronyc(1) man:chrony.conf(5)
Conflicts=openntpd.service ntp.service ntpsec.service
ConditionVirtualization=|!container
ConditionVirtualization=|wsl
ConditionCapability=CAP_SYS_TIME
[Service]
Type=notify
PIDFile=/run/chrony/chronyd.pid
EnvironmentFile=-/etc/default/chrony
User=_chrony
# Daemon is started as root, but still sandboxed
ExecStart=!/usr/sbin/chronyd -n $DAEMON_OPTS
CapabilityBoundingSet=~CAP_AUDIT_CONTROL CAP_AUDIT_READ CAP_AUDIT_WRITE
CapabilityBoundingSet=~CAP_BLOCK_SUSPEND CAP_KILL CAP_LEASE CAP_LINUX_IMMUTABLE
CapabilityBoundingSet=~CAP_MAC_ADMIN CAP_MAC_OVERRIDE CAP_MKNOD CAP_SYS_ADMIN
CapabilityBoundingSet=~CAP_SYS_BOOT CAP_SYS_CHROOT CAP_SYS_MODULE CAP_SYS_PACCT
CapabilityBoundingSet=~CAP_SYS_PTRACE CAP_SYS_RAWIO CAP_SYS_TTY_CONFIG CAP_WAKE_ALARM
DeviceAllow=char-pps rw
DeviceAllow=char-ptp rw
DeviceAllow=char-rtc rw
DevicePolicy=closed
LockPersonality=yes
MemoryDenyWriteExecute=yes
NoNewPrivileges=yes
PrivateTmp=yes
ProcSubset=pid
ProtectControlGroups=yes
ProtectHome=yes
ProtectHostname=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectProc=invisible
ProtectSystem=strict
# Used for gps refclocks
ReadWritePaths=/run
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=yes
RestrictSUIDSGID=yes
SystemCallArchitectures=native
SystemCallFilter=~@cpu-emulation @debug @module @mount @obsolete @raw-io @reboot @swap
ConfigurationDirectory=chrony
RuntimeDirectory=chrony
RuntimeDirectoryMode=0700
# See dumpdir in chrony.conf(5)
RuntimeDirectoryPreserve=restart
StateDirectory=chrony
StateDirectoryMode=0750
LogsDirectory=chrony
LogsDirectoryMode=0750
# Adjust restrictions for /usr/sbin/sendmail (mailonchange directive)
NoNewPrivileges=no
ReadWritePaths=-/var/spool
RestrictAddressFamilies=AF_NETLINK
[Install]
Alias=chronyd.service
WantedBy=multi-user.target
## files
ls: cannot access '/etc/systemd/system/chrony.service.d': No such file or directory
ls: cannot access '/etc/systemd/system/chronyd.service.d': No such file or directory
ls: cannot access '/etc/systemd/system/chrony.service': No such file or directory
lrwxrwxrwx 1 root root 38 May 21 13:32 /etc/systemd/system/chronyd.service -> /usr/lib/systemd/system/chrony.service
-rw-r--r-- 1 root root 1929 Jul 1 22:28 /lib/systemd/system/chrony.service
-rw-r--r-- 1 root root 1929 Jul 1 22:28 /usr/lib/systemd/system/chrony.service
## dpkg verify chrony
rc=0
## conf
/etc/chrony:
total 24
drwxr-xr-x 4 root root 4096 May 21 13:32 .
drwxr-xr-x 102 root root 4096 Nov 8 05:32 ..
-rw-r--r-- 1 root root 1450 Feb 16 2026 chrony.conf
-rw-r----- 1 root _chrony 481 Feb 16 2026 chrony.keys
drwxr-xr-x 2 root root 4096 Oct 7 14:15 conf.d
drwxr-xr-x 2 root root 4096 Oct 7 14:15 sources.d
/etc/chrony/conf.d:
total 12
drwxr-xr-x 2 root root 4096 Oct 7 14:15 .
drwxr-xr-x 4 root root 4096 May 21 13:32 ..
-rw-r--r-- 1 root root 422 Feb 16 2026 README
/etc/chrony/sources.d:
total 12
drwxr-xr-x 2 root root 4096 Oct 7 14:15 .
drwxr-xr-x 4 root root 4096 May 21 13:32 ..
-rw-r--r-- 1 root root 498 Feb 16 2026 README
pool 2.debian.pool.ntp.org iburst
sourcedir /run/chrony-dhcp
sourcedir /etc/chrony/sources.d
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
ntsdumpdir /var/lib/chrony
logdir /var/log/chrony
maxupdateskew 100.0
rtcsync
makestep 1 3
leapseclist /usr/share/zoneinfo/leap-seconds.list
confdir /etc/chrony/conf.d
-- /etc/chrony/conf.d/README
Files found under the /etc/chrony/conf.d directory with the .conf suffix are
parsed in the lexicographical order of the file names when chronyd starts up.
This enables a fragmented configuration of chronyd.
Although those files can contain any directives listed in chrony.conf(5),
it would be wiser to add NTP sources in the /etc/chrony/sources.d
directory. Please read /etc/chrony/sources.d/README for more information.
-- /etc/chrony/sources.d/README
Only NTP sources can be specified in the /etc/chrony/sources.d directory.
Files in this directory must end with the ".sources" suffix. They can only
contain the "peer", "pool" and "server" directives and must have all
lines terminated by a trailing newline.
There is no need to restart chronyd for these time sources to be usable,
running 'chronyc reload sources' is sufficient.
Example:
## default
DAEMON_OPTS="-F 1"
## apparmor
/usr/sbin/chronyd
-rw-r--r-- 1 root root 0 May 21 13:32 /etc/apparmor.d/local/usr.sbin.chronyd
-rw-r--r-- 1 root root 2837 Jul 1 22:28 /etc/apparmor.d/usr.sbin.chronyd
/etc/apparmor.d/disable:
total 8
drwxr-xr-x 2 root root 4096 Jul 21 2025 .
drwxr-xr-x 9 root root 4096 Nov 8 05:32 ..
## apparmor profile notify lines
81: # Allow systemd Type=notify using sd_notify's $NOTIFY_SOCKET
82: @{run}/systemd/notify w,
## kernel audit lines for chronyd (last 20)
## journal chrony this boot
Nov 08 05:35:23 demo-felhom systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Nov 08 05:35:23 demo-felhom chronyd[690]: chronyd version 4.8 starting (+CMDMON +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +NTS +SECHASH +IPV6 -DEBUG)
Nov 08 05:35:23 demo-felhom chronyd[690]: Loaded 0 symmetric keys
Nov 08 05:35:23 demo-felhom chronyd[690]: Using leap second list /usr/share/zoneinfo/leap-seconds.list
Nov 08 05:35:23 demo-felhom chronyd[690]: Frequency -2.666 +/- 0.016 ppm read from /var/lib/chrony/chrony.drift
Nov 08 05:35:23 demo-felhom chronyd[690]: Loaded seccomp filter (level 1)
Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE
Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'.
Nov 08 05:35:23 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.
## first failure ever in the journal
Oct 16 05:09:25 demo-felhom chronyd[692]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Oct 16 05:09:25 demo-felhom chronyd[692]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Oct 21 05:23:50 demo-felhom chronyd[676]: Fatal error : Could not send notification to $NOTIFY_SOCKET
count: 8
## oldest journal line
Oct 04 21:43:33 demo-felhom systemd[3282788]: Queued start job for default targe
## last good start in the journal
## boots
-5 46081f567f384ab88591d4e604555a48 Sun 2026-10-04 21:43:33 CEST Sun 2026-10-04 21:43:46 CEST
-4 e82176ae7cf644518507468fc609eaa9 Wed 2026-10-14 15:43:07 CEST Fri 2026-10-16 05:09:06 CEST
-3 ddd3d7305e244b2a9eafb5cf229476f1 Fri 2026-10-16 05:09:23 CEST Wed 2026-10-21 05:23:34 CEST
-2 56db71938b304a57bdedf30f89860e1f Wed 2026-10-21 05:23:48 CEST Thu 2026-10-29 04:47:58 CET
-1 4f6d8b388e224b8ca8ed45e06a2f96b9 Thu 2026-10-29 04:48:11 CET Sun 2026-11-08 05:35:13 CET
0 72899d2625244e069e8e8099775d1e51 Sun 2026-11-08 05:35:21 CET Wed 2026-11-11 14:06:53 CET
## what starts with it
chrony.service
● ├─-.mount
● ├─system.slice
● ├─systemd-journald.socket
● ├─systemd-remount-fs.service
● ├─systemd-tmpfiles-setup.service
● ├─tmp.mount
● ├─basic.target
● │ ├─-.mount
○ │ ├─systemd-pcrphase-sysinit.service
● │ ├─tmp.mount
● │ ├─paths.target
● │ │ ├─systemd-ask-password-console.path
● │ │ └─systemd-ask-password-wall.path
● │ ├─slices.target
● │ │ ├─-.slice
● │ │ ├─system-xfs_scrub.slice
● │ │ ├─system.slice
● │ │ └─user.slice
● │ ├─sockets.target
● │ │ ├─dbus.socket
● │ │ ├─iscsid.socket
● │ │ ├─netavark-dhcp-proxy.socket
● │ │ ├─rrdcached.socket
● │ │ ├─sshd-unix-local.socket
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.
chrony.service +256ms
└─basic.target @2.481s
└─sockets.target @2.481s
└─systemd-hostnamed.socket @2.481s
└─sysinit.target @2.478s
└─apparmor.service @2.241s +237ms
└─local-fs.target @2.238s
└─mnt-felhom\x2ddrives-hdd_1.mount @11.537s
└─mnt-felhom\x2ddrives.mount @2.549s
## other time services
inactive inactive inactive inactive
Local time: Wed 2026-11-11 14:06:55 CET
Universal time: Wed 2026-11-11 13:06:55 UTC
RTC time: Wed 2026-11-11 13:06:52
Time zone: Europe/Budapest (CET, +0100)
System clock synchronized: no
NTP service: inactive
RTC in local TZ: no
## chronyc
506 Cannot talk to daemon
506 Cannot talk to daemon
## dpkg history chrony/apparmor/systemd
2026-07-18 16:14:58 install apparmor:amd64 <none> 4.1.1-pmx1
2026-07-18 16:15:52 configure apparmor:amd64 4.1.1-pmx1 <none>
2026-10-07 14:15:40 upgrade chrony:amd64 4.6.1-3+deb13u1 4.8-4~bpo13+2
2026-10-07 14:15:46 configure chrony:amd64 4.8-4~bpo13+2 <none>
## felhom files that name chrony
## ps
## run dir
ls: cannot access '/run/chrony': No such file or directory
srwxrwxrwx 1 root root 0 Nov 8 05:35 /run/systemd/notify
## uptime
2026-11-08 05:35:18
@@ -0,0 +1,25 @@
#### demo-felhom 2026-11-11T13:16:03Z
## fresh compile of the on-disk profile to stdout (no load, cache not read, not written)
83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1 -
## loaded
dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647
## cache
83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1 /var/cache/apparmor/ae1a4050.0/usr.sbin.chronyd
dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 /var/cache/apparmor/f6da0254.0/usr.sbin.chronyd
drwx------ 2 root root 4096 2026-10-07 14:16:03.051992948 +0200 /var/cache/apparmor/ae1a4050.0/
-rw-r--r-- 1 root root 3177 2026-07-18 18:17:02.251100923 +0200 /var/cache/apparmor/ae1a4050.0/.features
drwx------ 2 root root 4096 2026-11-08 05:32:24.065142783 +0100 /var/cache/apparmor/f6da0254.0/
-rw-r--r-- 1 root root 3219 2026-10-07 12:11:32.040268492 +0200 /var/cache/apparmor/f6da0254.0/.features
## state
failed
chrony 4.8-4~bpo13+2
7.0.14-28-pve
1794402963.132843793
## chrony fatal lines
8
## stale check, every cached profile: cache differs from a fresh compile
current dir: /var/cache/apparmor/f6da0254.0/
STALE: lxc-containers
STALE: usr.sbin.chronyd
checked=106 stale=2
@@ -0,0 +1,33 @@
#### felhom 2026-11-11T13:17:03Z
## fresh compile of the on-disk profile to stdout (no load, cache not read, not written)
dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 -
## loaded
dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647
## cache
dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 /var/cache/apparmor/ae1a4050.0/usr.sbin.chronyd
dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 /var/cache/apparmor/f6da0254.0/usr.sbin.chronyd
drwx------ 2 root root 4096 2026-10-04 21:37:58.088000000 +0200 /var/cache/apparmor/ae1a4050.0/
-rw-r--r-- 1 root root 3177 2026-10-04 21:37:57.515000000 +0200 /var/cache/apparmor/ae1a4050.0/.features
drwx------ 2 root root 4096 2026-10-07 10:41:12.158575707 +0200 /var/cache/apparmor/f6da0254.0/
-rw-r--r-- 1 root root 3219 2026-10-07 10:41:11.594575707 +0200 /var/cache/apparmor/f6da0254.0/.features
## state
active
chrony 4.6.1-3+deb13u1
7.0.14-22-pve
System time : 0.000177019 seconds slow of NTP time
Leap status : Normal
1794403023.309780841
## chrony fatal lines
0
## stale check, every cached profile: cache differs from a fresh compile
current dir: /var/cache/apparmor/f6da0254.0/
STALE: lxc-containers
checked=106 stale=1
2026-10-04 19:31:38 install apparmor:amd64 <none> 4.1.1-pmx1
2026-10-04 19:32:11 install proxmox-kernel-7.0.2-6-pve-signed:amd64 <none> 7.0.2-6
2026-10-07 10:02:07 install proxmox-kernel-7.0.14-20-pve-signed:amd64 <none> 7.0.14-20
2026-10-07 17:14:55 install proxmox-kernel-7.0.14-22-pve-signed:amd64 <none> 7.0.14-22
NRestarts=0
ExecMainStartTimestamp=Tue 2026-11-10 15:43:32 CET
ActiveState=active
@@ -0,0 +1,26 @@
# READ ONLY. Part 3: the time service on a Proxmox host.
echo "#### host $(hostname) $(date -u +%FT%TZ) local $(date +%T%z)"
echo "## versions"; pveversion; dpkg -l chrony systemd systemd-timesyncd 2>/dev/null | grep '^[a-z][a-z] ' | awk '{print $1,$2,$3}'
uname -r
echo "## unit state"; systemctl show chrony -p ActiveState -p SubState -p Result -p Type -p NotifyAccess -p ExecMainStatus -p ExecMainCode -p NRestarts -p ActiveEnterTimestamp -p InactiveEnterTimestamp -p FragmentPath -p DropInPaths -p UnitFileState -p PIDFile
echo "## unit cat"; systemctl cat chrony 2>&1
echo "## files"; ls -la /etc/systemd/system/chrony.service.d /etc/systemd/system/chronyd.service.d /usr/lib/systemd/system/chrony.service /lib/systemd/system/chrony.service /etc/systemd/system/chrony.service /etc/systemd/system/chronyd.service 2>&1
echo "## dpkg verify chrony"; dpkg -V chrony 2>&1; echo "rc=$?"
echo "## conf"; ls -la /etc/chrony /etc/chrony/conf.d /etc/chrony/sources.d 2>&1; grep -v '^\s*#' /etc/chrony/chrony.conf 2>&1 | grep -v '^$'; for f in /etc/chrony/conf.d/* /etc/chrony/sources.d/*; do [ -f "$f" ] && { echo "-- $f"; grep -v '^\s*#' "$f" | grep -v '^$'; }; done
echo "## default"; cat /etc/default/chrony 2>&1 | grep -v '^#'
echo "## apparmor"; aa-status 2>/dev/null | grep -i chrony; ls -la /etc/apparmor.d/usr.sbin.chronyd /etc/apparmor.d/local/usr.sbin.chronyd /etc/apparmor.d/disable 2>&1; cat /etc/apparmor.d/local/usr.sbin.chronyd 2>/dev/null | grep -v '^#'
echo "## apparmor profile notify lines"; grep -n 'notify\|/run/systemd' /etc/apparmor.d/usr.sbin.chronyd 2>&1
echo "## kernel audit lines for chronyd (last 20)"; journalctl -k --no-pager 2>/dev/null | grep -i 'chrony' | tail -20 | cut -c1-330
echo "## journal chrony this boot"; journalctl -u chrony -b --no-pager 2>&1 | tail -40 | cut -c1-300
echo "## first failure ever in the journal"; journalctl -u chrony --no-pager 2>/dev/null | grep -m3 'NOTIFY_SOCKET' | cut -c1-200; echo "count: $(journalctl -u chrony --no-pager 2>/dev/null | grep -c 'NOTIFY_SOCKET')"
echo "## oldest journal line"; journalctl --no-pager 2>/dev/null | head -1 | cut -c1-80
echo "## last good start in the journal"; journalctl -u chrony --no-pager 2>/dev/null | grep 'Started chrony\|Selected source' | tail -3 | cut -c1-200
echo "## boots"; journalctl --list-boots --no-pager 2>/dev/null | tail -6
echo "## what starts with it"; systemctl list-dependencies --after chrony 2>/dev/null | head -25; systemd-analyze critical-chain chrony.service 2>&1 | head -12
echo "## other time services"; systemctl is-active systemd-timesyncd ntp ntpsec openntpd 2>&1 | tr '\n' ' '; echo; timedatectl 2>&1
echo "## chronyc"; chronyc tracking 2>&1 | head -8; chronyc sources 2>&1 | tail -6
echo "## dpkg history chrony/apparmor/systemd"; zgrep -h ' chrony\| apparmor:\| systemd:' /var/log/dpkg.log* 2>/dev/null | grep ' upgrade \| install \|configure ' | sort | tail -25
echo "## felhom files that name chrony"; grep -rl chrony /etc/felhom* /usr/local/sbin /usr/local/bin /etc/systemd/system 2>/dev/null | head;
echo "## ps"; ps -eo pid,user,args | grep '[c]hronyd'
echo "## run dir"; ls -la /run/chrony /run/systemd/notify 2>&1
echo "## uptime"; uptime -s
@@ -0,0 +1,14 @@
# READ ONLY
echo "#### $(hostname) $(date -u +%FT%TZ)"
sysctl net.unix.max_dgram_qlen kernel.printk_ratelimit 2>&1
echo "## cmdline"; cat /proc/cmdline
echo "## apparmor denied this boot: $(journalctl -b --no-pager 2>/dev/null | grep -c 'apparmor=\"DENIED\"')"; journalctl -b --no-pager | grep 'apparmor="DENIED"' | head -5 | cut -c1-300
echo "## audit lines with chrony, all boots"; journalctl --no-pager 2>/dev/null | grep -i 'audit.*chrony\|chrony.*audit' | head -5 | cut -c1-300
echo "## the second around the chrony start, this boot"
T=$(journalctl -u chrony -b --no-pager -o short-iso | grep -m1 'Starting chrony' | awk '{print $1}'); echo "start: $T"
journalctl -b --no-pager -o short-monotonic --since "$(date -d "$T - 1 sec" '+%F %T')" --until "$(date -d "$T + 2 sec" '+%F %T')" | cut -c1-200 | head -150
echo "## chrony lines per boot (all boots)"; for b in -5 -4 -3 -2 -1 0; do echo "-- boot $b"; journalctl -u chrony -b $b --no-pager 2>/dev/null | grep -v '^-- ' | grep 'Starting\|Started\|Fatal\|Failed\|Stopp\|exit' | head -8 | cut -c1-160; done
echo "## sandbox env of pid1"; systemctl show -p NotifySocket -p DefaultEnvironment 2>&1 | head; ls -la /run/systemd/ | head -30
echo "## tailscale"; systemctl is-active tailscaled 2>&1
echo "## rtc"; ls /dev/rtc* /dev/ptp* /dev/pps* 2>&1
echo "## cpu"; nproc; grep -m1 'model name' /proc/cpuinfo
@@ -0,0 +1,18 @@
main() {
echo "#### $(hostname) $(date -u +%FT%TZ)"
echo "## fresh compile of the on-disk profile, to stdout only (no load, no cache write)"
apparmor_parser -S -K --skip-bad-cache-rebuild -o /dev/stdout /etc/apparmor.d/usr.sbin.chronyd 2>/dev/null | sha256sum
apparmor_parser -S -K -o /dev/stdout /etc/apparmor.d/usr.sbin.chronyd 2>&1 >/dev/null | head -3
echo "## loaded raw policy sha256"; cat /sys/kernel/security/apparmor/policy/profiles/usr.sbin.chronyd*/raw_sha256; echo
echo "## cache dirs"; ls -la --time-style=full-iso /var/cache/apparmor/
for d in /var/cache/apparmor/*/; do echo "$d features-sha=$(sha256sum < $d/.features | cut -c1-16) files=$(ls $d | wc -l)"; ls -l --time-style=full-iso $d/usr.sbin.chronyd $d/.features; done
echo "## kernel features now"; cat /sys/kernel/security/apparmor/features/policy/versions/* 2>/dev/null | tr '\n' ' '; echo; aa-features-abi -x 2>/dev/null | sha256sum | cut -c1-16
echo "## newest source file the profile depends on"; for f in /etc/apparmor.d/usr.sbin.chronyd /etc/apparmor.d/local/usr.sbin.chronyd /etc/apparmor.d/tunables/global /etc/apparmor.d/abstractions/base /etc/apparmor.d/abstractions/nameservice /etc/apparmor.d/abi/*; do ls -l --time-style=full-iso $f 2>/dev/null; done | awk '{print $6,$7,$9}' | sort | tail -6
echo "## files in /etc/apparmor.d changed 2026-07-18 or 2026-08-21"; find /etc/apparmor.d -newermt 2026-07-18 ! -newermt 2026-07-19 | head; find /etc/apparmor.d -newermt 2026-08-21 ! -newermt 2026-08-22 | head
echo "## versions"; dpkg-query -W apparmor chrony; apparmor_parser --version | head -1
echo "## dpkg log"; zgrep -h -E ' (upgrade|install) (apparmor|chrony|proxmox-kernel-[0-9.]+-[0-9]+-pve-signed):' /var/log/dpkg.log* | sort | tail -14
echo "## kernels booted"; journalctl --list-boots --no-pager | tail -8; for b in -4 -3 -2 -1 0; do journalctl -b $b -k --no-pager -o cat 2>/dev/null | head -1 | cut -c1-60; done
echo "## chronyd confinement now"; ps -eZ 2>/dev/null | grep -i chrony || echo "no chronyd process"
echo "## what installed chrony 4.8"; zgrep -h -B3 -A3 'chrony' /var/log/apt/history.log* 2>/dev/null | grep -E 'Start-Date|Commandline|Requested-By|chrony' | cut -c1-260 | tail -12
}
main </dev/null
@@ -0,0 +1,12 @@
main() {
echo "#### $(hostname) $(date -u +%FT%TZ)"
echo "## fresh compile of the on-disk profile to stdout (no load, cache not read, not written)"
apparmor_parser -S -K /etc/apparmor.d/usr.sbin.chronyd 2>/dev/null | sha256sum
echo "## loaded"; cat /sys/kernel/security/apparmor/policy/profiles/usr.sbin.chronyd*/raw_sha256; echo
echo "## cache"; sha256sum /var/cache/apparmor/*/usr.sbin.chronyd; ls -ld --time-style=full-iso /var/cache/apparmor/*/ /var/cache/apparmor/*/.features
echo "## state"; systemctl is-active chrony; dpkg-query -W chrony; uname -r; chronyc -n tracking 2>&1 | grep -E 'System time|Leap' ; date -u +%s.%N
echo "## chrony fatal lines"; journalctl -u chrony --no-pager -o short-iso 2>/dev/null | grep -c 'Could not send notification'
echo "## stale check, every cached profile: cache differs from a fresh compile"; n=0; bad=0; cur=$(ls -d /var/cache/apparmor/*/ | while read d; do [ "$(sha256sum < $d/.features)" = "$(aa-features-abi -x 2>/dev/null | sha256sum)" ] && echo $d; done | head -1); echo "current dir: $cur"
for f in /etc/apparmor.d/*; do [ -f "$f" ] || continue; b=$(basename $f); [ -f "$cur$b" ] || continue; n=$((n+1)); a=$(apparmor_parser -S -K "$f" 2>/dev/null | sha256sum | cut -d' ' -f1); c=$(sha256sum < "$cur$b" | cut -d' ' -f1); [ "$a" = "$c" ] || { bad=$((bad+1)); echo "STALE: $b"; }; done; echo "checked=$n stale=$bad"
}
main </dev/null
+10 -7
View File
@@ -262,7 +262,7 @@ stopping line that lies.
| **R-775** | Security & access | P3 | **[P2-MEDIUM] Grimmory: a stranger's 5 wrong sign-ins lock EVERY visitor out of the web login for 15 minutes — so Grimmory was not published.** MEASURED 2026-10-01 on 9202 (drill catalog, v3.4.1, through traefik): after 5 wrong tries for `admin` every further sign-in answered 429 — the household's right password AND a different name — and stayed 429 for 10+ minutes of retries. Read in the jar: `AuthRateLimitService` — Caffeine `expireAfterWrite(ofMinutes(15))`, `MAX_ATTEMPTS 5`, keys `login:ip:` and `login:user:`; Spring `forward-headers-strategy: native` takes the address from X-Forwarded-For, and behind the tunnel every visitor is the tunnel container's address (R-753) — the wger shape (R-752), with no setting to change it. Everything else in the checklist passed (bench + box step v3.4.1 → v3.5.0, gate by its own probe, OPDS through traefik); two smaller findings for the publishing session: on a reinstall over the first install's kept books, a new upload was saved to the drive but not added to the library (`box/grimmory/reinstall-c1.txt`, not investigated); and the remove + restore round trip (2.5) cannot be shown on 9202 for a drive app — its backup lives on the scratch drive, which is not a registered drive (R-756). The template waits in `audits/new-apps-2026-10-01/wip/grimmory/`. **Needs (operator):** (A) publish with a sentence on the page that wrong guesses by others can lock the login for 15 minutes (MEASURED: during the lock an e-reader's OPDS feed still answered 200 with its own login, wrong 401 — `box/grimmory/opds-under-lock.txt`), or (B) wait until the box passes each visitor's real address (R-753). `audits/new-apps-2026-10-01/box/grimmory/throttle.txt` **-- 2026-10-01 (evening):** option B's precondition SHIPPED (controller v0.286.1, R-753): Grimmory's Tomcat RemoteIpValve walks from the right and counts `172.16.0.0/12` as a proxy (READ in source, Spring Boot 4.1.1 — not yet measured with Grimmory's own lock), so `login:ip:` becomes per visitor; `login:user:` still lets a stranger lock the public name `admin` 15 min. A third route was spiked and passed: Grimmory behind the permanent family gate with its e-reader paths excepted (R-780). Recommendation: publish behind the family gate if R-780 is built; otherwise B with a measured 3.6. **UPDATE 2026-10-02 — NARROWED, Grimmory PUBLISHED behind the family gate:** a stranger cannot reach Grimmory's web sign-in at all (6 tries through the simulated tunnel: the gate's 401, then the household signs in 200 — `audits/family-gate-2026-10-02/A/items.txt`), and the e-reader exceptions keep Grimmory's own login. The 2.5 round trip now WORKS on 9202 (`audits/family-gate-2026-10-02/B/box/life.txt`). **What is left:** a family member past the gate can still lock a NAME (the admin's) for 15 minutes with 5 wrong tries — hard-coded in Grimmory; and the reinstall-over-kept-books finding (`new-apps-2026-10-01/box/grimmory/reinstall-c1.txt`) is still not investigated. | **WATCHING — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P2→P3: Grimmory is now behind the family gate; only a family member can still lock a name.** | — | — | CC |
| **R-908** | Security & access | P3 | **An old Resend API key is still in the history of the `homelab-manifests` repository; it was replaced on 2026-06-29, but whether it is also revoked at Resend is unknown.** FOUND 2026-10-08 by the contact-mailer source search (`audits/mailer-source-2026-10-08/SEARCH.md`, „A side finding"): commit c9648cd (2026-02-05, „added mailer pod") put a key literal in the old `felhom-system/contact-mailer.yaml` comment; the file was deleted in ee93b50 but the history keeps it. Compared by hash only, never printed: it is NOT today's key (`Secret/resend-api`, rotated 2026-06-29, felhom.eu feea0606). If the old key still works at Resend, anyone with read access to that repository can send mail as felhom.eu. | **OPEN — owner: operator** **Note 2026-10-12 (security review):** gitleaks 8.24.3 with its default rules did NOT flag this key in the full history scan, so a scanner gate alone would not have caught it (R-986). | — | In the Resend console, check that the old key is revoked (revoke it if not); rewriting the repository's history is NOT needed once it is revoked | operator |
| **R-525** | Security & access | P4 | **[P3-LOW] FileBrowser has its own login; putting it behind the dashboard session (traefik forwardAuth or Quantum proxy auth) is a new mechanism nobody has measured.** Filed 2026-09-15 by the P1-fixes task (B.5). R-513 closed the default-password hole with a generated password; a household still has two logins. **What it needs:** a spike on a scratch guest — forwardAuth to the controller session, and what FileBrowser Quantum does with a trusted header. | **READY — rank P3-LOW; owner: CC (spike)** **Re-ranked 2026-10-03: P3->P4: comfort feature needing a new unmeasured mechanism; the default password hole is closed.** | — | — | CC |
| **R-925** | Security & access | P1 | **`manifests/felhom.secret.yaml` carries REAL secret values, and the Gitea repo that holds it is readable by anyone on the internet with no login.** FOUND 2026-10-09 by a background security review of the legal-pages commit, which flagged the served `<!-- source -->` comments; chasing what those comments pointed at found this instead. **MEASURED, in this order:** (1) `scripts/manifest_bearer_gate.py` itself prints `manifests/felhom.secret.yaml:39 KNOWN-BACKLOG committed secret 65cee3c4…7a86`; (2) the file holds **live-shaped values, not placeholders** — `SECRET_KEY` (69 chars), `SUPERUSER_EMAIL`/`SUPERUSER_PASSWORD` (18), Umami `APP_SECRET` (66) and `POSTGRES_PASSWORD` (34), and a `username`/`password` pair (18); values were never printed, only measured by length; (3) `gitea.dooplex.hu` resolves to **37.191.56.193**, the same public address as the website; (4) an **anonymous** `curl` returns HTTP 200 and 1686 bytes for that exact file, and 200 for `hub/internal/store/store.go` and `manifests/hub.yaml`; (5) **the off-network control**, which is what makes (4) mean anything — fetched from outside the operator's network entirely: a real path returns the file's first line, a nonsense path returns 404, so the 200 is genuine anonymous read from the internet and not a LAN-only ACL. **This contradicts the project's own written model**: `documentation/runbooks/secrets.md:3-4` says *“Secret values are never committed to git. The manifests in `manifests/` carry only placeholders + comments that point here.”* That promise is false today, which is the P1 wording in this register's own scale — see the ranking note below. The same runbook already states the rule that governs the fix: *“The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.”* **Known neighbours, none of which cover this:** R-887 records that an outside crawler walks the public Gitea pages and leaves *“Gitea's exposure to the crawler”* as the operator's undone item; R-580 still owes a Gitea admin token rotation. Neither says a secret file is world-readable. **Why P2 and not P1, stated so the operator can overrule it:** the exposed credentials guard **analytics and an undeployed healthchecks instance, not household data** — measured: `umami-db` is a **ClusterIP** service on 5432 with no external IP, so the Postgres password is not reachable from the internet; the realistic harm is session forgery against the public `stats.felhom.eu` via `APP_SECRET`, and reuse of those passwords anywhere else. No customer box, hub token or escrow key is in this file. **CC did NOT change anything**: making the repo private could break the public day-0 path (the installer is fetched from a public tag, R-110), and rotation plus repository visibility are operator decisions on production infrastructure. **PARTLY FIXED 2026-10-09, and RE-RANKED P2 -> P1 on what it turned out to be.** The exposed value was not an analytics password: it was the **Gitea `admin` account password** (`is_admin: true`; `/api/v1/admin/users` answered 200), published in a repo `gitea.dooplex.hu` serves anonymously to the internet. That is push access to every repo — including the one whose `website/` is git-synced live and whose `scripts/` is published by tag to **every new box installer** (R-110), i.e. a supply-chain path onto customer hardware. My first ranking said P2 because I had only measured the analytics blast radius; the credential test is what corrected it. **DONE (CC, operator-instructed):** (1) `umami-config` `APP_SECRET` + `POSTGRES_PASSWORD` rotated — the password changed **inside Postgres** too (`ALTER USER`), because the env var is only read at first init; verified by a real beacon returning 200 with a bogus-site-id control still returning 400. (2) `healthchecks-config` `SECRET_KEY` + `SUPERUSER_PASSWORD` rotated (nothing consumes them — there is no healthchecks Deployment). (3) `gitea-creds` no longer holds the admin password at all: it holds a **scoped token** (`read:package` + `read:repository`). **Both scopes are load-bearing and the second was found by breaking it** — a package-only token made the hub log `Template fetch: unexpected status 403`, because the template fetcher reads a raw file out of the `felhom-controller` repo. (4) The Gitea **admin password** was changed and `gitea-system/gitea-admin` updated; **verified: new password 200, old published password 401 — the leak is dead.** (5) The file is removed from git and `.gitignore`'s `*secret*` now applies to it; `manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out is gone, red-proofed with a decoy (exit 1 with, 0 without). **AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part:** the `rollout restart` needed to pick up the new umami secret put umami into **CrashLoopBackOff and took `stats.felhom.eu` down (503)**. Cause was not the rotation — at `memory: 512Mi` the pod runs for months but **cannot restarLine truncated
| **R-925** | Security & access | P1 | **`manifests/felhom.secret.yaml` carries REAL secret values, and the Gitea repo that holds it is readable by anyone on the internet with no login.** FOUND 2026-10-09 by a background security review of the legal-pages commit, which flagged the served `<!-- source -->` comments; chasing what those comments pointed at found this instead. **MEASURED, in this order:** (1) `scripts/manifest_bearer_gate.py` itself prints `manifests/felhom.secret.yaml:39 KNOWN-BACKLOG committed secret 65cee3c4…7a86`; (2) the file holds **live-shaped values, not placeholders** — `SECRET_KEY` (69 chars), `SUPERUSER_EMAIL`/`SUPERUSER_PASSWORD` (18), Umami `APP_SECRET` (66) and `POSTGRES_PASSWORD` (34), and a `username`/`password` pair (18); values were never printed, only measured by length; (3) `gitea.dooplex.hu` resolves to **37.191.56.193**, the same public address as the website; (4) an **anonymous** `curl` returns HTTP 200 and 1686 bytes for that exact file, and 200 for `hub/internal/store/store.go` and `manifests/hub.yaml`; (5) **the off-network control**, which is what makes (4) mean anything — fetched from outside the operator's network entirely: a real path returns the file's first line, a nonsense path returns 404, so the 200 is genuine anonymous read from the internet and not a LAN-only ACL. **This contradicts the project's own written model**: `documentation/runbooks/secrets.md:3-4` says *“Secret values are never committed to git. The manifests in `manifests/` carry only placeholders + comments that point here.”* That promise is false today, which is the P1 wording in this register's own scale — see the ranking note below. The same runbook already states the rule that governs the fix: *“The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.”* **Known neighbours, none of which cover this:** R-887 records that an outside crawler walks the public Gitea pages and leaves *“Gitea's exposure to the crawler”* as the operator's undone item; R-580 still owes a Gitea admin token rotation. Neither says a secret file is world-readable. **Why P2 and not P1, stated so the operator can overrule it:** the exposed credentials guard **analytics and an undeployed healthchecks instance, not household data** — measured: `umami-db` is a **ClusterIP** service on 5432 with no external IP, so the Postgres password is not reachable from the internet; the realistic harm is session forgery against the public `stats.felhom.eu` via `APP_SECRET`, and reuse of those passwords anywhere else. No customer box, hub token or escrow key is in this file. **CC did NOT change anything**: making the repo private could break the public day-0 path (the installer is fetched from a public tag, R-110), and rotation plus repository visibility are operator decisions on production infrastructure. **PARTLY FIXED 2026-10-09, and RE-RANKED P2 -> P1 on what it turned out to be.** The exposed value was not an analytics password: it was the **Gitea `admin` account password** (`is_admin: true`; `/api/v1/admin/users` answered 200), published in a repo `gitea.dooplex.hu` serves anonymously to the internet. That is push access to every repo — including the one whose `website/` is git-synced live and whose `scripts/` is published by tag to **every new box installer** (R-110), i.e. a supply-chain path onto customer hardware. My first ranking said P2 because I had only measured the analytics blast radius; the credential test is what corrected it. **DONE (CC, operator-instructed):** (1) `umami-config` `APP_SECRET` + `POSTGRES_PASSWORD` rotated — the password changed **inside Postgres** too (`ALTER USER`), because the env var is only read at first init; verified by a real beacon returning 200 with a bogus-site-id control still returning 400. (2) `healthchecks-config` `SECRET_KEY` + `SUPERUSER_PASSWORD` rotated (nothing consumes them — there is no healthchecks Deployment). (3) `gitea-creds` no longer holds the admin password at all: it holds a **scoped token** (`read:package` + `read:repository`). **Both scopes are load-bearing and the second was found by breaking it** — a package-only token made the hub log `Template fetch: unexpected status 403`, because the template fetcher reads a raw file out of the `felhom-controller` repo. (4) The Gitea **admin password** was changed and `gitea-system/gitea-admin` updated; **verified: new password 200, old published password 401 — the leak is dead.** (5) The file is removed from git and `.gitignore`'s `*secret*` now applies to it; `manifest_bearer_gate.py`'s `KNOWN_BACKLOG` carve-out is gone, red-proofed with a decoy (exit 1 with, 0 without). **AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part:** the `rollout restart` needed to pick up the new umami secret put umami into **CrashLoopBackOff and took `stats.felhom.eu` down (503)**. Cause was not the rotation — at `memory: 512Mi` the pod runs for months but **cannot restarLine truncated
| **R-1243** | Security & access | P2 | **The hub's Longhorn backups sit unencrypted in the NFS folder any DooPlex pod can read (R-1241) — the two there now (2026-10-18, 2026-10-23) hold every household's Cloudflare and tunnel tokens in plaintext, because they predate hub 0.168.0.** Measured 2026-10-23 night (`audits/hub-token-seal-2026-10-23/` §3): the hub volume is `encrypted=false` and in Longhorn's `default` backup group (daily 04:00 + weekly Sunday 05:00, retain 1 each); the hub's own copy to ep0 is encrypted at rest (16/16 snapshots `crypt-mode: encrypt`, key on DooPlex's root disk and on paper) and in transit (SSH tunnel + pinned PBS TLS), every 6 h, restore-tested weekly. DECIDE: (a) an encrypted Longhorn volume for the hub (~5–10 min downtime, a homelab StorageClass + key in the cluster) or (b) take the hub volume out of the Longhorn backup — ep0's copy is the backup; the access log then has no backup until it rides the ep0 push. **CC's pick: (b).** And: delete the two plain copies (the operator's call; with (b) they never rotate out). | **VERIFY** — **2026-10-24: D40 put into effect** (operator ruling 07:22): the hub PVC `hub-data` carries `recurring-job-group.longhorn.io/default: disabled` (felhom.eu `3363ffa8`, ArgoCD sync of the PVC only) and the Longhorn volume `pvc-486c9809…` reads `disabled` too (labelled live to match, unchanged after 2 min); the NFS folder holds no copy of the volume (no `pvc-486c9809*` directory, no backup config naming it; Longhorn lists no backup volume for it; 53 other volumes backed up this morning). Closes after the dated check: tonight's 04:00 job makes no new hub backup while the others run. WAITING-ON-OPERATOR **2026-10-25 dated check DONE (read-only, the restore-truth session):** `kubectl -n longhorn-system get backups.longhorn.io` lists 106 backups made today (the newest 04:08:28Z — the job ran) and none of `pvc-486c9809` — the hub volume is out of the Longhorn backup as D40 meant. | the operator's pick; a homelab change for either option | Operator: pick (a) or (b), and say whether to delete the two plain copies after hub 0.168.0 is live | CC (the dated check) |
| **R-1355** | Security & access | P2 | **DooPlex's own Gitea 1.26.2 is inside 19 new notices (2026-11-02, fixed 28.3.0: `GHSA-8p3x` OAuth2 account takeover, `GHSA-37xv` and `GHSA-8g99` public-only tokens reach private data, `GHSA-32h8` Actions secrets) — on the host that holds our code and registry.** Also new on DooPlex: Vaultwarden 1.37.4 (fix 1.37.5), Uptime Kuma 2.5.6 (fix 2.5.9), Paperless 3.3.4 (fix 3.4.0). Felhom changes nothing on DooPlex. Read in `documentation/security-watch/state.json`, group DooPlex. | OPEN — filed 2026-11-03 | the homelab session | add them to `audits/http2shell-2026-10-27/HOMELAB-LIST.md`'s work; which notices reach a self-hosted Gitea with our settings is the homelab session's first read | operator (homelab session) |
| **R-779** | Security & access | P4 | **[P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box.** Measured 2026-10-01 19:51 UTC (`audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt`): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. **Needs:** one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs** **Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up.** | — | — | CC + operator |
@@ -279,7 +279,7 @@ stopping line that lies.
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` **2026-11-11:** the Clock list measured it again (32.6 s behind, time service not running) and leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CA03-6; class R-1384). | OPEN — filed 2026-10-26 **2026-11-10:** still so. The host clock is 32 s behind DooPlex (it was 14 s on 2026-10-26); `chrony` failed again at the 2026-11-08 05:35 boot with the same line; `NTPSynchronized=no`. Seen in a release run, where the box's log times read half a minute early (`documentation/audits/release-space-fixes-2026-11-10/agent/13-n100-clock.txt`). Not fixed by hand: the N100 takes no hand step in a release run. | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC |
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` **2026-11-11:** the Clock list measured it again (32.6 s behind, time service not running) and leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CA03-6; class R-1384). **2026-11-11 (afternoon, read only): the cause is found.** The kernel loaded an OLD cached AppArmor profile for chrony (the one of chrony 4.6.1); chrony 4.8 writes to `/run/systemd/notify`, the old profile refuses that write, chrony treats it as fatal. Proof: loaded profile sha256 `dbf67f16…` against a fresh compile `83417e83…` on the N100, equal on demo-hp; the kernel's `DENIED` line of 2026-10-16. The order that made it: a new kernel started once (2026-10-07 12:11), the box went back to the old kernel, chrony was updated there (14:15), the new kernel started again and used its stale cache folder (`documentation/audits/n100-clock-2026-11-11/README.md`). The product side is R-1388. The hand step is in `STATUS.md` „do”; it was not run and not tested. | OPEN — filed 2026-10-26 **2026-11-10:** still so. The host clock is 32 s behind DooPlex (it was 14 s on 2026-10-26); `chrony` failed again at the 2026-11-08 05:35 boot with the same line; `NTPSynchronized=no`. Seen in a release run, where the box's log times read half a minute early (`documentation/audits/release-space-fixes-2026-11-10/agent/13-n100-clock.txt`). Not fixed by hand: the N100 takes no hand step in a release run. | — | the operator runs the one hand step on the N100 (`STATUS.md`); then read `chronyc tracking` and close. The product fix is R-1388 | CC |
| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). **2026-11-11:** the Clock list leans on this row (`documentation/audits/clock-list-2026-11-11/TABLE.md` CC02, CC06-6, CC07-6, CC22-6). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC |
| **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC |
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. **2026-11-10 (release run) — part (2) RELEASED, VOUCHED and PROVEN on three test boxes** (agent 0.173.0, vouched by the operator 15:29 UTC; `documentation/audits/release-space-fixes-2026-11-10/agent/`): the folder went 49 → 1, 48 → 1 and 33 → 1 files (about 700 / 690 / 485 MB back) at the commit of THIS update, not at the next one (the earlier sentence here and in the agent's CHANGELOG was wrong, and is corrected there). A rollback after the clean-up worked on Tester 1: a signed update to the older version was crashed once, 9 seconds into the 60-second dwell, and the wrapper put the released version back in the same second; the failed version's file stays until the next good update (by design). The kept older kernel on demo-hp booted to healthy in 313 s and the box went back (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **Still open:** (1) the kernel clean-up as a program (`11-os-updates.md` §5.11.1, not built) and a second box; (3) template archives; (4) the `.bak-*` copies — the release does not reach them; (5) the failed-run logs. A household box gets part (2) at its first good agent update after the vouch. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
@@ -291,11 +291,12 @@ stopping line that lies.
| **R-1214** | Box system & updates | P3 | **On the 8 GB box with the swap fix, the night's second-drive backup still pushes memory „full" pressure above 20 % for about 3 minutes; the box keeps answering.** Night soak 2026-10-21→22 (`audits/night-2026-10-21/P1-soak/03-rec.jsonl`, 10-s samples; fresh box from the public ISO, agent with swap = RAM, 8 apps, the rehearsal uploads for 2 h): at 01:30–01:34 UTC (the 03:30 tier2 job) host 39 % / guest 45 % over 10 s, over 60 s at most 20 % / 25 %; IO „full" 66 %; guest swap 1.35 → 2.1 GB; `/api/health` answered every probe (slowest 4.1 s, largest interval 16 s); 0 OOM kills. Once more in the upload window (Immich loading its models, 31 % / 35 % for one sample). The brief's soak pass line was „full" < 20 %, so the soak missed that one line; the freeze itself (96 % for an hour, R-1192) did not come back. Not diagnosed: which part of the tier2 job (the copy's page cache, the volume dumps) causes it. | WATCHING — **2026-10-24 02:4xZ dated check read (weekend-night session, read only):** Tester 1 (8 GB host, guest 4 GB, controller 0.341.0) ran its 03:30 copy at 01:30:00–01:30:13Z (3 apps, 157 MB + 86 MB + 16 KB; no „capped copy container did not start" WARN, no leftover `felhom-tier2-copy-*`); the guest cgroup's memory „full" stall counter on the host (`/sys/fs/cgroup/lxc/9201/memory.pressure`) read **total 444 µs** over the whole 11.1 h since the host started (15:32Z), so „full" stayed far under 20 % (20 % of 10 s = 2 s); 0 OOM kills. **Not shown:** that the copy ran in the capped container (the INFO log does not name it; Docker keeps only ~200 events, from 02:36Z). **Weak load:** 0.24 GB, not the soak's large copy — so the row stays WATCHING; close after a night whose copy moves ≥ 1 GB, read from the same counter. WATCHING — **RELEASED 2026-10-23 in controller 0.340.0** (floor 06:41 UTC; on all three test boxes, start-up „no leftover”, 0 copy containers; Tester 1's manual second-drive run: 3 apps, 0 containers left). **Open only for the consequence:** dated check 2026-10-24 — Tester 1's first night copy ran in the capped container and memory „full” pressure stayed under 20 % (the box's own night log). READY — rides controller 0.340.0. READY — waits for the controller 0.340.0 release. **2026-10-22 night — cause measured, fix built and proven on 9202, waits for the controller release.** Cause: the copy's page cache is charged to the guest and evicts the apps' working set. On 9202 (4 GB RAM, 1.8 GB held by a hog, a 6 GB copy): plain rsync full stall 5 702 / 2 464 ms, peak „full” 16.8 / 6.9 %; the same copy in a 512 MB memory-capped container 374 / 320 ms, 0.9 / 1.0 %. Controller main `03c0a69` + `ae39467` (unreleased, rides 0.340.0): the copy runs capped only for clean `/mnt/` paths with no symlink (security review); a docker error (exit 125) falls back to plain rsync. A real `POST /api/backup/tier2` on rc.4: 6 capped containers, each exit 0; Immich's copy equals its source (dry-run 0 differences, 44 = 44 files). Evidence `audits/night-2026-10-22/P4-R1214/`. **2026-10-23 night (stray-copy fix, night brief Part 1).** The capped copy's container outlived the copy (a timeout or a controller restart killed only the `docker` CLI; `--delete` went on outside the R-1180 guard). Controller main `66a69d4` + `912e592` (unreleased, rides 0.340.0): a fixed name + label per copy, removed by name on any error, every leftover removed at start before the scheduler and the web server, a restore refused while a copy runs or may run; `--mount` (a path the daemon does not see is refused, not created empty) and the image ID with `--pull never` (review C4, C7). Proven on 9202 with rc.5 (`audits/night-2026-10-23/P1-straycopy/`): a restart in the middle of a 16 GB copy → the new controller removed `felhom-tier2-copy-immich-948f2280` at start, the restore probe was refused during the copy, the redone copy equals the source; a 15 s test build → the timed-out copy's container removed, the partial copy stopped growing. A second review of that code (N2–N5) and the commit's own security review (two races) → `9d9e404`, `1babfe9`: a stray clears when a later removal works, the start-up check never removes this process's own copy and holds the destination's lock, one copy per destination at a time, `RunTier2` refuses during a restore for every caller and records itself in one step with that check, every leg asks again. Re-proven on 9202 with rc.8 (`audits/night-2026-10-23/P1-straycopy/1*`). 9202 back on 0.339.0. | — | Release 0.340.0; then re-measure the 03:30 job on the 8 GB soak box and close if „full” stays low | CC |
| **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). **2026-10-22: closes when Tester 2 is reinstalled (operator ruling 2026-10-22 14:11 — a new install from the public download replaces the bootstrap and the off-site check; `pilot/TESTER2-handover.md`).** | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator |
| **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC |
| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB21); the newest golden passed its 7-day limit on 2026-11-10 and no waiver is in force, so the gate refuses code pushes to `felhom.eu`. | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement **2026-11-03: the Sunday 2026-11-01 bake was missed (it was in no brief): golden 0.344.0 is 8 days old, the floor 0.349.0; a CODE push to felhom.eu is refused until a bake and the vouch or a dated waiver. The night's close-out was documents only, which the gate exempts; no waiver written. First on the morning list.** | — | — | CC |
| **R-468** | Box system & updates | P3 | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** **2026-10-15 (`09` §3 decision 250, operator):** the cadence is now the RULE, not a waiver — a controller release needs no golden (the floor carries it), the golden is baked every Sunday night when the floor is above it (`runbooks/nightly-rotation.md`), and `golden_currency_gate.py` convicts only a golden MORE THAN 7 DAYS old while a release is ahead (CASE 18–21, red-proved). The waiver file stays as a dated exception for a missed Sunday. **2026-11-11:** the End-dates list leans on this row (`documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB21); the newest golden passed its 7-day limit on 2026-11-10 and no waiver is in force, so the gate refuses code pushes to `felhom.eu`. **2026-11-11: golden 0.352.0 baked and published; one fresh box installed from it (`documentation/tests/golden-0.352.0-2026-11-11/README.md`); the gate is green (`python3 scripts/golden_currency_gate.py`, rc 0). The operator vouched it on 2026-11-11 at 14:48:11 CET.** | **WATCHING** — the weekly bake is a night-routine step; the first external install still retires the waiver arrangement **2026-11-03: the Sunday 2026-11-01 bake was missed (it was in no brief): golden 0.344.0 is 8 days old, the floor 0.349.0; a CODE push to felhom.eu is refused until a bake and the vouch or a dated waiver. The night's close-out was documents only, which the gate exempts; no waiver written. First on the morning list.** | — | — | CC |
| **R-1358** | Box system & updates | P3 | **cloudflared 2026.10.4 misses two dependency fixes (VULN-183456 chi, 2026.10.5; VULN-184686 go-jose, 2026.10.6; the newest is 2026.10.7).** No public text on reach. Not pinned tonight: the tunnel is a box's only door, and 9202 has none to prove it on. | OPEN — filed 2026-11-03 | — | raise `CloudflaredImage` on the next controller release with a tunnel proof on Tester 1 | CC |
| **R-1380** | Box system & updates | P3 | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** Found 2026-11-11 (the End-dates list, class T2, `documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB02-3 to DB02-7). MEASURED: the agent trusts one exact Proxmox certificate (a pin), and the certificate on the three test boxes ends about two years after install — the N100's about 2028-07-16, then demo-hp and Tester 1 within three months. Proxmox makes a new certificate by itself about 14 days before the old one ends (so about 2028-07-02 on the N100). No code path gives the agent the new fingerprint, no alarm names the cause, no runbook holds the hand step, and nobody has tried it. Every box installed in the same season fails in the same fortnight. P3 only because the first date is in 2028; it is certain, not a risk. Decision D76 in `STATUS.md`. | OPEN — filed 2026-11-11 (end-dates list) | — | After D76: the agent reads the certificate file on its own host and takes the new fingerprint (design in `03-host-agent.md` first); prove on a scratch host with a forced renewal. | CC |
| **R-1380** | Box system & updates | P3 | **Proxmox renews its own certificate and the agent's pin does not follow: from that day the agent cannot control or back up a guest (5 answers).** Found 2026-11-11 (the End-dates list, class T2, `documentation/audits/end-dates-list-2026-11-11/TABLE.md` DB02-3 to DB02-7). MEASURED: the agent trusts one exact Proxmox certificate (a pin), and the certificate on the three test boxes ends about two years after install — the N100's about 2028-07-16, then demo-hp and Tester 1 within three months. Proxmox makes a new certificate by itself about 14 days before the old one ends (so about 2028-07-02 on the N100). No code path gives the agent the new fingerprint, no alarm names the cause, no runbook holds the hand step, and nobody has tried it. Every box installed in the same season fails in the same fortnight. P3 only because the first date is in 2028; it is certain, not a risk. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 297). Not built.** **2026-11-11 (afternoon): tried on demo-hp (`documentation/audits/certpin-spike-2026-11-11/README.md`).** A forced new certificate (`pvecm updatecerts --force`, then a reload of the web service; Proxmox's own renewal does that reload by itself) blinds the agent at once: `TLS pin mismatch`, no reconcile, no host report to the hub, 502 to the controller; guests and apps stay up. One operator mail names a symptom (`agent_channel_unknown`), none names the cause; the hub's Hosts page still read ONLINE eleven minutes after the last report (when it turns offline was not measured). The hand repair works and is written down (`documentation/runbooks/RUNBOOK-agent-proxmox-pin-repair.md`, 2.1 s from restart to a working call). **A CA file in place of the fingerprint works with no agent code:** with `proxmox.tls.ca_file` on a readable copy of the box's root CA and the fingerprint empty, the agent went through a second new certificate with no hand step; a wrong CA file was refused. Two facts for the fix: the agent's user cannot read Proxmox's certificate files, and a set fingerprint wins over the CA file. The design with both ways is `03-host-agent.md` §12a; CC's pick differs from the ruled shape, so it is decision D80. | OPEN — filed 2026-11-11 (end-dates list) | — | the operator rules D80 (CA file, or follow the fingerprint); then the design's open parts in `03-host-agent.md` §12a (the root step that refreshes the copy; an alarm that names the cause), then build, then prove on a scratch host with a forced renewal | CC |
| **R-1382** | Box system & updates | P3 | **A box that was off past an end date: nobody knows if it comes back by itself (5 answers).** Found 2026-11-11 (the End-dates list, class T4, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; read in source and documents, never tried). After months without power: the web certificate has ended (DB01-6: does Traefik renew at the first start, and what does the household see until then), Debian's package keys may be too old to verify an update (DB16-6), the tunnel client may be below what Cloudflare accepts (DB19-6), a 30-day restore copy is past its date (DB24-6), and the 14-day countdown after a stopped off-site copy ran out while the box was off (DB04-6). Tester 2 has been off since 2026-10-04 and is the natural test. | OPEN — filed 2026-11-11 (end-dates list) | — | When Tester 2 comes back (or with a scratch host and a moved clock on Tier 0, as a task): record what renews by itself and what needs a hand; put the hand steps in a runbook. | CC |
| **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. Decision D79 in `STATUS.md`. | OPEN — filed 2026-11-11 (clock list) | — | After D79: the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC |
| **R-1384** | Box system & updates | P3 | **A verdict or an „already ran” check trusts a clock that can be wrong (30 answers).** Found 2026-11-11 (the Clock list, class W2, `documentation/audits/clock-list-2026-11-11/TABLE.md`). MEASURED: the N100's host is 32.6 s behind and its time service does not run (R-1301); the other two boxes are in step. Nothing checks a box's clock except one hub alarm for a tunnel handshake stamped in the future. The hub takes the box's own time for backup age, off-site age and proof age (CH02-6, CH07-6, CH07-8, CH08-6); a box with a wrong clock refuses every signed job (CA09-6); a stamp from the future blocks the OS-update leg, the trim and the line test until the clock passes it (CA02-6, CA02-8, CA11-6, CA11-8, CA12-6); the crash guard reads the clock at early boot (CA04-6, CA04-8); a hub clock jump moves its own daily jobs (CH03-6, CH04-6, CH06-6, CH09-6, CH16-6, CH18-6, CH20-6, CH24-6). The others: CA05-6, CA05-8, CA06-6, CA07-6, CA10-6, CA20-6, CC04-6, CC05-6, CC14-6, CC18-6. Known parts stay in their rows: R-1189, R-1206, R-1260. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 300). Not built.** **2026-11-11 (afternoon): why the N100's clock is wrong is known (R-1301, `documentation/audits/n100-clock-2026-11-11/README.md`).** Numbers from source: a host clock more than 120 s behind refuses every signed job (`felhom-agent/internal/authz/verifier.go:21`, `:153-162`); at about 1.25 s a day the N100 gets there in about 70 days (mid-January 2027, an estimate from one measurement). | OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): the hub compares the time in each report with its own, alarms the operator above one minute, and does not take a time from the future as „fresh”. | CC |
| **R-1388** | Box system & updates | P3 | **After a kernel went back and forth, the box can load an old security profile for an updated package — on the N100 this stopped the time service for good.** Found 2026-11-11 (`documentation/audits/n100-clock-2026-11-11/README.md`; read only, on three boxes). AppArmor keeps one cache folder for each kernel. A package update writes the new profile only into the folder of the kernel that runs. If the box then starts another kernel that it started once before, that kernel's folder still holds the old profile, and AppArmor uses it (the cached copy looks newer than the package's file). MEASURED on the N100: chrony 4.8 under the profile of chrony 4.6.1; the kernel refuses its write to `/run/systemd/notify`; the unit fails at every start since 2026-10-16. On a household box the product's own kernel rollback (the crash guard) makes the same order, and it is true for every profile, not only chrony's. Not measured: a box where the rollback really ran. Also seen: Tester 1's host still has chrony 4.6.1 while the other two have 4.8 from the OS-update leg; why was not looked at. | OPEN — filed 2026-11-11 (N100 clock diagnosis) | — | Design first, in `11-os-updates.md`: after an OS-update leg and after a kernel rollback, reload the profiles with the cache ignored (`apparmor_parser -r -T -W`), or clear the cache folders of the kernels that do not run. Red-proof on a scratch host: make the order, see the stale profile, see the fix remove it. Then say why Tester 1 did not take chrony 4.8. | CC |
| **R-1004** | Box system & updates | P4 | **After an unclean stop every guest ext4 volume waits ~43 s in multi-mount protection before it mounts, so each restart after a power cut takes ~1.5 min longer.** Kernel: „EXT4-fs warning (device dm-6): ext4_multi_mount_protect: MMP interval 42 higher than expected, please wait" on dm-6/dm-7 and loop0/loop1, demo-hp and Tester 1, every round of `audits/night-power-2026-10-13/` (R1 `repairs.txt`). | **READY** — filed 2026-10-14 | — | CC: find where MMP is switched on for the guest volumes (Proxmox `pct` create, or our own mkfs), whether it protects anything on a one-host disk, and measure the restart without it on a scratch guest | CC |
| **R-1013** | Box system & updates | P4 | **After a power cut the agent's start-up stale-lock pass logs an ERROR „pct unlock failed … no lock found" when Proxmox's own start-all removed the lock in the same second.** The outcome is the goal (no lock), so the line is a false alarm in the log. Seen on demo-hp 2026-10-14 12:07:44–45 UTC (`audits/power-cut-fixes-2026-10-14/E/after-boot.txt`; Proxmox: „removed left over backup lock from '9201'!"). Small, but the agent's release for this session is spent (v0.156.1). | **OPEN** — filed 2026-10-14 | — | CC: treat „no lock found" from `pct unlock` as already clear (INFO), with a test; ride the next agent release | CC |
| **R-1061** | Box system & updates | P4 | **A per-customer floor reached Tester 1 only at its next 15-minute report, not in seconds.** Set 2026-10-15 12:55:51 UTC for tester-1, demo-hp and demo-felhom together (hub 0.153.0, CC's token): demo-hp and demo-felhom ran 0.314.0 within 13 s (the hub's wake); Tester 1 moved at 13:09:35, its regular report (`audits/gate-restore-2026-10-15/release/03-boxes-move.txt`). It had restarted at 12:54:29 onto rc.1 and logged „hub wait channel active". Why the wake did not reach it is not read (the wake line is DEBUG). No harm: the floor arrived. | **OPEN** — filed 2026-10-15 | — | Read Tester 1's DEBUG ring at the next floor change; compare with the hub's bump log | CC |
@@ -331,8 +332,8 @@ stopping line that lies.
| **R-1363** | Monitoring & notifications | P3 | **A household's mail ends with raw JSON: `Megjegyzés: {"avail_gb":5.192043304443359,"band":"warning",...}`.** Seen in the walks of 2026-11-02 (`audits/security-2026-11-02/part5/mail-fill-household.txt`, `mail-crash-household.txt`). `api/handler.go` adds the details as the „Megjegyzés" line; a golden pins it (`mail_golden_test.go`), so it was once wanted. The sentence itself is right and Hungarian. | OPEN — filed 2026-11-03 | — | drop the line from a household's mail, or render known fields in words; the operator's mail keeps it | CC |
| **R-1365** | Monitoring & notifications | P3 | **A security-watch source that stays down mails once and never again; after that only one line of the Monday summary says so.** An open advisory reminds every 7 days, a failed source does not (`08` §11 [DESIGN], „once per streak"). The cost was seen 2026-10-28 → 11-02: one mail, then a week of silence. With the seven pins that can never be read now a stated GAP (R-1334), a streak means a real outage. | WAITING-ON-OPERATOR — D66 | D66 | on yes: `plan()` mails a streak again every 7 days, with a test seen failing; `08` §11's line changes | operator (decision), CC (build) |
| **R-1371** | Monitoring & notifications | P3 | **Seven places have no space alarm, and the host-disk alarm that does exist goes to the household.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 6). (1) **The guest root disk has no watcher.** The controller watches `/` and the system data path, but inside its container both are the data volume (measured on all three boxes, `evidence/extra/controller-fs-view.txt`); the 32 GiB guest root disk, which holds `/opt/docker/stacks`, is not a target. Closed row R-521 says it is — true only before the two volumes were merged; capability-map row C13 („warned before a filesystem fills — per filesystem”) does not hold for this one filesystem (the map's status is not changed here). (2) **The same filesystem is two targets**, and warnings are de-duplicated by path, so two warnings for one full disk are expected (read from source, not proved live; the comment at `cmd/controller/main.go:2694-2707` says the opposite). (3) **`host_disk_critical` and `storage_fill_critical` are mailed to the household by default** (`hub/internal/claim/engine.go:229-243`), but only the operator can free a host disk; the seed list was chosen as „critical only” and the question „can this reader act on it” was not asked. No architecture document names the host-disk event (search S4; `08-alarm-ladder.md` names the thin-pool one). (4) Small places with no alarm: EFI partition, temp folders, guest system log, guest package cache. **Members (answers marked N):** H10-6, H15-6, G1-6, G13-6, G15-6, G16-6, G17-6. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D68 = operator only; `09` §3 decision 287):** the household is not mailed about a host disk it cannot free. **2026-11-10 — parts (2) and (3) FIXED IN SOURCE, NOT RELEASED.** (2) felhom-controller `844f19f1`: one fill target per filesystem (`system.FilesystemKey`; the container's `/` is an overlay with its own device number, so the identity is taken from a Docker-kept file — `documentation/audits/space-list-2026-11-10/evidence/extra/fs-identity.txt`); `TestR1371_*`, red seen three times; not proven on a box. (3) hub source, no build: `host_disk_warning` / `host_disk_critical` are operator-only and the critical is not seeded; `storage_fill_*` reaches a household only for `usb`, `local-dir`, `nfs`, `cifs` (`notify.operatorOnlyFor`; `08-alarm-ladder.md`); `TestD68_*`, red seen three times. Parts (1) and (4) stay open: who watches the guest root disk is a design point in `16-storage-physical-identity.md`. **2026-11-10 (afternoon) — two design points added** (`16-storage-physical-identity.md` §14.4): a drive in the backup role reports as `usb` or `local-dir`, so the household gets the `storage_fill` mail for a drive it must not clean by hand; the hub's first list for a household still holds `storage_fill_critical` until the box sends its own list (not measured on the running hub). Also noted: the closed row R-521 and `felhom-controller/controller/cmd/controller/r521_f7_rootfs_test.go:29,36` call the watched path „the guest's root filesystem”; it is the container's `/`, an overlay on the data volume. The guest's real root disk is the one part (1) says nobody watches. **2026-11-10 (release run) — parts (2) and (3) RELEASED** (`documentation/audits/release-space-fixes-2026-11-10/`). (2) controller 0.352.0, global floor 15:31 UTC: on scratch guest 9202 one fill sent TWO warnings on the version before and ONE on the new one (so the double warning is now seen on a box, not only read from source); Tester 1, demo-hp and demo-felhom each check 2 filesystems where they checked 3. (3) hub 0.180.0 live since 15:12 UTC, Synced/Healthy; proof = the `TestD68_*` tests and the deployed commit. **Not proven on the running hub:** that a household which saved its own alarm list before no longer gets the host-disk type — the pages that show a household's list refuse a token (by design, `05` §17) and no test alarm was made (`16` §14.4). **Still open:** (1) the guest root disk has no watcher (design point); (4) the small places; a real fill on a household's box. | — | Controller: add the guest root disk as a fill-watch target, de-duplicate targets by filesystem, correct the comment; red-proof with two paths on one filesystem. Hub: after the pick, route the two host-disk events. Docs: name the host-disk alarm and its reader in `08-alarm-ladder.md`. | CC |
| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. | OPEN — filed 2026-11-11 (end-dates list) | — | After D77: the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC |
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. Decision D78 in `STATUS.md`. | OPEN — filed 2026-11-11 (clock list) | — | After D78: one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
| **R-1379** | Monitoring & notifications | P3 | **An end date exists and nobody is warned before it (18 answers).** Found 2026-11-11 (the End-dates list, class T1, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; dates measured read-only on three boxes and from public certificates). No alarm fires before any of these ends: the box's web certificate (DB01-5, DB01-7: Traefik renews by itself; a renewal that fails is silent until the day), the hub address certificate (DH20-5), the operator tokens with a 90-day life (DH22-5), a node's Tailscale key (DH30-5), Debian's package keys (DB16-5), the 30-day restore copy (DB24-5), the end of OS security support (DB17-5), app and tunnel-client images (DB18-5, DB19-5), the local-API certificate (DB03-5), and the outside accounts: mail key, domains, Cloudflare, Hetzner, GitHub and Gitea tokens (DH23-5, DH24-5, DH27-5, DH28-5, DH29-5, DH31-5, DH32-5). Decision D77 in `STATUS.md` asks if the product warns 14 days before. **Ruled 2026-11-11 13:17: yes, D77 (`09` §3 decision 298). Not built.** | OPEN — filed 2026-11-11 (end-dates list) | — | Fix shape (ruled, not built): the box reads its web certificate's date, the hub reads its tokens' dates, one operator alarm 14 days before. The outside accounts stay a list the operator reads (`documentation/audits/end-dates-list-2026-11-11/README.md`). | CC |
| **R-1383** | Monitoring & notifications | P3 | **A job on a timer does not run, gives up, or loses its state at a restart, and nobody is told (23 answers).** Found 2026-11-11 (the Clock list, class W1, `documentation/audits/clock-list-2026-11-11/TABLE.md`; read in source, times measured read-only on three boxes). The four worst: the guest-power watchdog gives up with one journal line (CA07-7, CA07-8); a signed job that is refused or runs out stays silent, so an agent update never arrives (CA09-4, CA09-7); the hub's daily checks are skipped with no trace when the hub is down at their minute (CH02-4, CH02-8, CH11-4, CH16-4); an operator mail waiting for a retry is lost at a hub restart, and a household mail is never retried (CH04-4, CH04-7, CH04-8, CH03-7). The others: CA13-7, CC07-7, CC12-4, CC19-7, CC20-4, CH07-4, CH09-4, CH14-4, CH26-7, CH27-4, CH27-8. Beside R-1361 (a lost event) and R-1186 (the silent off-site skip); not the same gaps. **Ruled 2026-11-11 13:17: yes (`09` §3 decision 299). Not built.** | OPEN — filed 2026-11-11 (clock list) | — | Fix shape (ruled, not built): one hub alarm for the four worst cases (agent: watchdog give-up and refused job as events; hub: a start-up check „did my daily jobs run”, and the mail retry kept across a restart). | CC |
| **R-1377** | Monitoring & notifications | P4 | **Two endings tell nobody outside the page.** Found 2026-11-10 (the endings list, `documentation/audits/endings-list-2026-11-10/TABLE.md` endings EC03, EC15; read in source). **Orphan delete** of an app sends the hub a fresh report and no 'app removed' event (`internal/api/router.go:1150-1178`); the normal remove sends one. **Wipe of a drive** sends no event to the hub (`internal/web/storage_handlers.go:911-1000`). Not a member: retiring a drive — the hub does not read that field by ruling (decision 130, closed R-888). **Members (answers marked N):** EC03-6, EC15-6. | OPEN — filed 2026-11-10 (endings list) | — | Controller: send the 'app removed' event from orphan delete; decide in `08-alarm-ladder.md` whether a wipe is an operator event. | CC |
## Hub & operator — 17 rows (P2 1, P3 5, P4 11)
@@ -429,3 +430,5 @@ stopping line that lies.
| **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | **OPEN — (A) BUILT 2026-10-26 evening, closes after the first nightly run (2026-10-27 03:20 UTC) is read.** Both runners proven at once (jobs 2374 + 2375 on `felhom-gates-runner` / `-2`, 17:11:41Z). felhom.eu CI, one push each: **(a) docs only** 810 s (old runner) → 351 s (2 CPUs) → see the close-out; **(b) a gate script** 351 → 203 s (`ecd82aba`, job 2376); **(c) a hub change** 351 → 118 s (`7e57e824`, job 2378). Heavy suites by path + tag + nightly; `test_repo_gates.py` and due-checks group G test the wiring (130 s each → under 3 s). Decoys through CI's entry point with the filter ON: a docs-only register break → red; an inert gate → GREEN at first (a decoy hole, fixed in `e33ec821`) → red. No Go cache: CI has no Go (the 110 s was security-scan, now 27 s). `audits/r1297-ci-trim-2026-10-26/`. (B) DONE by the operator (`09` §3 decision 267). | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC |
| **R-1381** | Process & tooling | P4 | **No document says when eleven things end, or where to read the date (11 answers).** Found 2026-11-11 (the End-dates list, class T3, `documentation/audits/end-dates-list-2026-11-11/TABLE.md`; search S18 found no document that lists end dates). OS support for Debian 13 and Proxmox 9 (DB17-2, DB17-3), the tunnel client's oldest allowed version (DB19-2), the installer ISO (DH19-3), and the outside accounts and tokens (DH23-2, DH24-2, DH27-2, DH28-2, DH29-2, DH31-2, DH32-2). This run could not read these dates (no account call, by its fences). The operator reads them once: the table „Only you can read these dates” in `documentation/audits/end-dates-list-2026-11-11/README.md`. | OPEN — filed 2026-11-11 (end-dates list) | — | When the operator has read the dates: write them in one place (a short table in `documentation/operations/`), and give the due-checks block the ones inside a year. | CC |
| **R-1387** | Process & tooling | P4 | **A document and the source disagree on a time, in four places (4 answers).** Found 2026-11-11 (the Clock list, class W5, `documentation/audits/clock-list-2026-11-11/TABLE.md`). (1) `documentation/architecture/03-host-agent.md` names `Restart=always` and a watchdog for the agent's unit; the unit has `Restart=on-failure` and no watchdog (CA18-6, CA18-8). Which one is the design is not stated, so this session did not edit either. (2) The hub-database copy alarm: 13 hours in one place, 26 in another (CH23-7; a DooPlex procedure, listed for the operator, not edited). (3) Old „24 h / 02:30” text for the hub's daily job (CH17-6). **Seen beside the lists, same kind, not in the count:** (4) the comment at `felhom-controller/controller/cmd/controller/main.go:1412` says 14 days where the code keeps restore copies 30 days (D62); (5) `documentation/architecture/07-backup-architecture.md:455-457` cites `main.go` lines that have moved; (6) `documentation/operations/tailscale.md` says „both nodes” where three test hosts exist, and Tailscale was not running on demo-hp and Tester 1 on 2026-11-11 (not known if that is meant); (7) the threat model and the recovery inventory do not say the same about how often the hub bearer is rotated. | OPEN — filed 2026-11-11 (clock list) | — | Operator or a task: say which side is right for (1) and (2); then one docs commit. (3) is a text fix in the same commit. | CC |
| **R-1389** | Process & tooling | P4 | **From the operator's home network the installer address gives 404.** Found 2026-11-11 (`documentation/tests/golden-0.352.0-2026-11-11/fresh-install/installer-address-404-on-lan.txt`). The home router's DNS (192.168.0.250) answers `felhom.eu` with 192.168.0.192, the site copy on DooPlex that was retired on 2026-11-07 (R-1228). So `https://felhom.eu/scripts/…` gives 404 for every machine that uses that DNS; from DooPlex (public DNS) it gives 200. A household is not touched. A test box on the home network, and Tester 2 if it sits there, cannot get the installer by its public address. | OPEN — filed 2026-11-11 (golden fresh install) | — | The operator removes the `felhom.eu` override from the home DNS (it is his router), then one `curl -sI https://felhom.eu/scripts/` from a machine on that network reads 200. | operator |
| **R-1390** | Process & tooling | P4 | **`sec put` cannot push the store any more, so a rotated token is saved on DooPlex only.** Found 2026-11-11. `sec put` ends with a plain `git push` of the `cc-secrets` clone (`cc-secrets/ccsec/cli.py:51-56`). Since this run that clone's address holds no login, and its helper answers only inside `sec run GITEA_GIT_TOKEN -- …`. A plain push now fails for lack of a login; the new entry is committed in the clone and not pushed. Until it is fixed, the push is a second command: `sec run GITEA_GIT_TOKEN -- git -C /mnt/5_hdd/felhom.eu/git/cc-secrets push origin HEAD:main`. Not tried in this run (no store change was allowed). | OPEN — filed 2026-11-11 (token fix) | — | In `cc-secrets`: `sec put` runs its push through the store's own token (or says plainly „committed, not pushed: run …”); a test that a `put` on a clone with no login in the address ends pushed or says so. | CC |
@@ -0,0 +1,44 @@
# RUNBOOK — the agent cannot reach Proxmox after a certificate change (pin repair)
**PROVEN on demo-hp 2026-11-11** (`documentation/audits/certpin-spike-2026-11-11/README.md`, R-1380). One box, one
run. Design and the open fix: `architecture/03-host-agent.md` §12a.
## When to use it
All three are true:
1. The agent's log has this line: `proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint`
(`journalctl -u felhom-agent -n 50 --no-pager | grep "TLS pin mismatch"`).
2. The operator got the mail `agent_channel_unknown … agentapi: GET /storage: HTTP 502`, or the hub has no new host
report from the box.
3. The guests and the apps are up. (If they are down, this is a different fault.)
## Steps (as root on the Proxmox host)
1. Read the certificate that Proxmox serves, and the pin:
`echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256 -enddate`
`python3 -c "import json;print(json.load(open('/etc/felhom-agent/agent.json'))['proxmox']['tls'])"`
If the two fingerprints are equal, stop: the pin is not the fault.
2. Check that the served certificate is Proxmox's own: its fingerprint must equal the file's.
`openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -fingerprint -sha256`
If they differ, run `systemctl reload pveproxy`, wait 15 seconds, read step 1 again. Do not go on while they differ.
3. Save the config: `cp -p /etc/felhom-agent/agent.json /etc/felhom-agent/agent.json.pre-pin-repair`
4. Write the served fingerprint into `proxmox.tls.fingerprint` in `/etc/felhom-agent/agent.json`. Keep the file's
owner (`felhom-agent`) and mode (`0600`). The script of the proof does this: `tools/setpin.sh fp` in the audit
folder.
5. `systemctl restart felhom-agent` — **one time.** The unit allows 4 starts in 2 minutes; a fifth leaves the agent
stopped until the 2 minutes pass.
6. Proof that it works (a positive sign, not an empty log):
`grep 'felhom-agent@pve' /var/log/pveproxy/access.log | tail -3` shows new lines that end in `200`.
On demo-hp the first one came 2.1 seconds after the restart.
7. At the hub: a new `host-report` from the box in at most 5 minutes, and the event `agent_channel_recovered`.
8. Remove the saved file (it holds the agent's token): `shred -u /etc/felhom-agent/agent.json.pre-pin-repair`
## If the agent does not come back
Put the saved file back (`cp -p …pre-pin-repair agent.json`), wait 2 minutes, restart one time, and read
`journalctl -u felhom-agent -n 80 --no-pager`.
## Not covered
A household box has no path for this today: the steps need root on the host. That is the open part of R-1380.
@@ -123,6 +123,8 @@ R-96 incident record (committed 2026-07-27) — rationale, not directives.
> `cc-secrets/ARCHITECTURE.md` in this workspace. Never `source` or `cat` a credentials file.
> **Deleting a stored secret: operator only (D60, 2026-10-27).** CC never runs `sec rm` or removes anything from the
> store, even after the operator's yes — he runs `sec rm NAME` in his own terminal. The guard refuses it.
> **A repository address never holds a credential (2026-11-11, R-925).** A clone that needs a login uses its own
> credential helper (`felhom.eu/scripts/git-credential-sec.sh`) and `sec run GITEA_GIT_TOKEN -- git …`. Never print a git address.
## Live validation — no browser here
@@ -0,0 +1,8 @@
## boot 2026-11-11T12:54:28Z
drill VM reverted to virgin and cold-booted (RUNBOOK-manual-build §4.0); pve-manager/9.2.2
pveam update: update successful; template debian-13-standard_13.6-1_amd64.tar.zst (checksum verified)
build-golden.sh v3.2.0 from felhom-agent 91a2924ad0, sha256 645b3b65… (the same file as the 0.350.0 bake)
token + registry user copied file->file from sec (GITEA_PKG_TOKEN, GITEA_PKG_USER); unit properties grep for the token value: 0
token in bake.log: 0 (control with the token appended: 1)
registry login in the archive: tar listing for .docker/config.json = 0 (listing control: 3 other lines found)
bake started 12:55:43Z, ended about 13:02Z
@@ -0,0 +1,5 @@
## round trip 2026-11-11T13:02:59Z
GET https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.352.0/golden.tar.zst (anonymous)
HTTP 200 bytes 639712511
d76336054282ec138d8b0c8b9fec63319aa06f192aa421d012890b265b9abced -
bake: d76336054282ec138d8b0c8b9fec63319aa06f192aa421d012890b265b9abced
@@ -0,0 +1,13 @@
== teardown 2026-11-11T13:03:03Z
Logical volume "vm-9100-disk-0" successfully removed.
Logical volume "vm-9100-disk-1" successfully removed.
purging CT 9100 from related configurations..
pct destroy rc=0
shred rc=0
ls: cannot access '/root/.gitea-token': No such file or directory
ls: cannot access '/root/bake.log': No such file or directory
qemu count 0 at 2026-11-11T13:05:04Z
qemu-img snapshot -a virgin rc=0
Snapshot list:
ID TAG VM_SIZE DATE VM_CLOCK ICOUNT
1 virgin 0 B 2026-07-03 14:12:15 0000:00:00.000 0
@@ -0,0 +1,3 @@
read 2026-11-11T13:48:44Z (the hub log prints CET); the hub's Day-0 page shows the same two hashes
2026/11/11 14:48:11 [INFO] Artifact manifest set: agent=0.173.0 golden=0.352.0 min_agent="0.158.1" wrapper_sha=false bundle_sha="51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1"
2026/11/11 14:48:11 [INFO] artifact vouch timing (R-1346): registry probes 166ms, sha lookups 189ms, bundle sha 10ms, fleet check + save 336ms, total 702ms
@@ -0,0 +1,76 @@
# Golden 0.352.0 — bake, publish, one fresh install, 2026-11-11 (afternoon brief, Part 1)
Why: golden 0.350.0 passed the 7-day limit on 2026-11-10, and the gate refused code pushes to `felhom.eu` (R-468).
Procedure: 0.350.0's README, the same recipe. Controller 0.352.0 is the global floor (MinAgent 0.158.1).
Documents read: `architecture/09-update-architecture.md` (§3 decisions 247, 250), `architecture/05-hub-architecture.md`
(Day-0 vouch, D50), `runbooks/publish-train-rules.md`, `runbooks/nightly-rotation.md`, rows R-468, R-242.
| | Previous (`../golden-0.350.0-2026-11-03/`) | This bake |
|---|---|---|
| `build-golden.sh` | v3.2.0 | same file |
| Controller | `felhom-controller:0.350.0` | **`felhom-controller:0.352.0`** (MinAgent **0.158.1**) |
| Traefik baked | v3.7.17 | same |
| Docker engine | the six approved pairs | same |
| Template | `debian-13-standard_13.6-1_amd64.tar.zst` | same |
| Token | from `sec`, file → file | same |
## Pass markers (`bake.log`)
```
[golden] build-golden.sh v3.2.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.352.0
[golden] upload OK (HTTP 201)
GOLDEN_VERSION=0.352.0
GOLDEN_SHA256=d76336054282ec138d8b0c8b9fec63319aa06f192aa421d012890b265b9abced
```
Round trip: the registry's file, downloaded with no login and hashed, equals the bake's sha; 639,712,511 bytes
(`02-round-trip.txt`). Build guest and teardown: `00-boot.txt`, `03-teardown.txt`.
**Golden gate:** `python3 scripts/golden_currency_gate.py` → OK, rc 0. A baked golden with no vouch passes it; no
waiver was written.
## The fresh install (`fresh-install/`)
**The path, exactly.** The hub offers only the vouched golden (0.350.0). So the box ran the public installer 1.36.1
(sha256 `e18cdd17…aa1f1`) with **one added line** after line 479, which names golden 0.352.0 and its sha256
(`installer-the-one-line.diff.txt`). All other steps are the product's own: the hub pairing, the agent download, the
guest build, the setup-code mail, the claim page, the app page.
| Step | Result | Proof |
|---|---|---|
| VM 9384 `golden1111-box` on demo-hp, disks on `nvme-scratch` (never the thin pool), Proxmox from the Felhom ISO | installed | `vm-create.txt`, `install/` |
| Installer run | host `Tester4-d35551` at the hub, agent **0.173.0**, ONLINE, guests 1/1 | `app.txt` (last line) |
| Setup-code mail | arrived 13:07:20Z | the test mailbox |
| Claim page | `POST /claim` → 302 at 13:19:51Z | `claim.txt` |
| Controller version on the box | **0.352.0** | `claim.txt` |
| One app: BentoPDF, through the page's own deploy request | 202 at 13:20:41Z; `running` at 13:21:31Z; its address answers 200; a name with no app answers 404 | `app.txt` |
**The two reads the brief asked for**
1. Time service on the fresh host: `chrony` active, 0.000001 s off, read at 13:05:52Z — **before** the Felhom
installer ran. Not read after it (see below).
2. Proxmox certificate, read from the network: subject `CN=box1111.felhom.test`, issuer `CN=Proxmox Virtual
Environment, O=PVE Cluster Manager CA`, valid 2026-11-10 13:05:27 to **2028-11-09 13:05:27** GMT, sha256
`2D:C1:FE:7C:09:EA:80:F5:8D:89:15:DB:A4:B5:6B:4F:C5:07:0D:95:E5:C5:06:83:2F:B8:EB:7F:BB:79:AB:46`.
**The agent's pin was not read.** Indirect proof only: the agent is ONLINE at the hub and built the guest, and both
need a working call to Proxmox.
**Why not read: CC locked itself out of the host.** The installer closes the host on purpose (new root password into
the hub's vault, SSH by key only). CC put no key on the host before the install. That was CC's mistake, not a product
fault. Lesson, now a line in the proof-VM traps: put your key on the host before you run the installer.
**Seen on the way (not the golden's fault):** from the operator's LAN the name `felhom.eu` answers from the retired
site at 192.168.0.192, so the installer address gives 404 there; from DooPlex it gives 200
(`installer-address-404-on-lan.txt`). Filed as a row. The hub's list showed "0/0 guests" for the new host at 13:21:40Z
and "ONLINE 1/1" at 13:28:30Z: the guest shows at the host's next report (`fresh-install/app.txt`).
## Vouch — done by the operator, 2026-11-11 14:48:11 CET (`04-vouch-readback.txt`)
- golden **0.352.0**, sha256 `d76336054282ec138d8b0c8b9fec63319aa06f192aa421d012890b265b9abced`
- agent **0.173.0**, sha256 `752b2727fb6043417ae5f342acbda52330521b21bb799fda90760b960feef7ea`, bundle
`51ebc1f4c66d87918c87c035b4297c0e40e10ed31e3250c3837dfeaa0002e1c1` (both unchanged)
- min_agent **0.158.1** (unchanged)
## Teardown
`fresh-install/teardown.txt`: VM 9384 stopped and destroyed with its two disks; the helper script removed from demo-hp.
The hub host `Tester4-d35551` and appliance `3fb5fed3-504c-4c2c-a63f-8108d5c15b6e` are the operator's to remove.
@@ -0,0 +1,341 @@
[golden] build-golden.sh v3.2.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.352.0
[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) …
Logical volume "vm-9100-disk-0" created.
Logical volume pve/vm-9100-disk-0 changed.
Creating filesystem with 8388608 4k blocks and 2097152 inodes
Filesystem UUID: 809cd9a8-e042-431b-abcd-d8eb0a100d94
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624
Logical volume "vm-9100-disk-1" created.
Logical volume pve/vm-9100-disk-1 changed.
Creating filesystem with 6291456 4k blocks and 1572864 inodes
Filesystem UUID: 433a7dea-e960-424b-be3c-20f7076e7eda
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst'
Total bytes read: 553512960 (528MiB, 106MiB/s)
Detected container architecture: amd64
Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
done: SHA256:HJsSglsPEot2RL07p81VwRRw4OVfWhi7NVOJnBScUM8 root@felhom-golden
Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
done: SHA256:Lt6rmdjcR5gQ5awqXI/n3JgloNv3jJSb2OTosPz8RVE root@felhom-golden
Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...
done: SHA256:PLM8d/fxjmQEXBgLNuc/cRf9hfI5de1Fh9P2CxsfJoY root@felhom-golden
[golden] starting + installing Docker (official repo, trixie channel) …
[golden] Docker engine set PINNED to the approved release: containerd.io=2.4.2-3~debian.13~trixie docker-buildx-plugin=0.38.1-1~debian.13~trixie docker-ce=5:29.9.2-1~debian.13~trixie docker-ce-cli=5:29.9.2-1~debian.13~trixie docker-ce-rootless-extras=5:29.9.2-1~debian.13~trixie docker-compose-plugin=5.6.1-1~debian.13~trixie
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = (unset),
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = (unset),
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
installed: containerd.io 2.4.2-3~debian.13~trixie
installed: docker-buildx-plugin 0.38.1-1~debian.13~trixie
installed: docker-ce 5:29.9.2-1~debian.13~trixie
installed: docker-ce-cli 5:29.9.2-1~debian.13~trixie
installed: docker-ce-rootless-extras 5:29.9.2-1~debian.13~trixie
installed: docker-compose-plugin 5.6.1-1~debian.13~trixie
[golden] no approved guest release given - the template versions stay; first-night count vs an approved release: n/a
[golden] pending Debian upgrades in the baked guest (what a FUTURE approval may bring): 61
[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …
[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds …
[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
4f55086f7dd0: Pulling fs layer
4f55086f7dd0: Download complete
4f55086f7dd0: Pull complete
Digest: sha256:c567b231f65614688711a75f31518a064258a51d442117a3e771a3a8fdba78ef
Status: Downloaded newer image for hello-world:latest
docker OK (overlay2; data-root /var/lib/docker)
live-restore: on
/var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4
/mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4
both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576
[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.352.0 (no registry cred at deploy) …
WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/
0.352.0: Pulling from admin/felhom-controller
ecc510c1e359: Pulling fs layer
c3f83389ce1b: Pulling fs layer
e81c1fce9826: Pulling fs layer
63a1406b1961: Pulling fs layer
d747c9b6922d: Pulling fs layer
f0be24959d17: Pulling fs layer
153556725a0f: Pulling fs layer
63a1406b1961: Waiting
d747c9b6922d: Waiting
f0be24959d17: Waiting
153556725a0f: Waiting
e81c1fce9826: Verifying Checksum
e81c1fce9826: Download complete
ecc510c1e359: Verifying Checksum
ecc510c1e359: Download complete
63a1406b1961: Verifying Checksum
63a1406b1961: Download complete
d747c9b6922d: Verifying Checksum
d747c9b6922d: Download complete
c3f83389ce1b: Verifying Checksum
c3f83389ce1b: Download complete
f0be24959d17: Verifying Checksum
f0be24959d17: Download complete
153556725a0f: Verifying Checksum
153556725a0f: Download complete
ecc510c1e359: Pull complete
c3f83389ce1b: Pull complete
e81c1fce9826: Pull complete
63a1406b1961: Pull complete
d747c9b6922d: Pull complete
f0be24959d17: Pull complete
153556725a0f: Pull complete
Digest: sha256:3c785dd396b091f19d4c777db9d7a206e75ab018b6238b5d887e1a4c6cfd3b29
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.352.0
gitea.dooplex.hu/admin/felhom-controller:0.352.0
[golden] asking the controller which infra images it manages …
[golden] baking infra images (4): traefik:v3.7.17 cloudflare/cloudflared:2026.10.4 gtstef/filebrowser:1.5.8-stable gitea.dooplex.hu/admin/felhom-samba:1.2.0 …
v3.7.17: Pulling from library/traefik
881fc932939e: Pulling fs layer
c901ba6ff9e0: Pulling fs layer
4f376259b112: Pulling fs layer
7705a4c1b3f8: Pulling fs layer
7705a4c1b3f8: Waiting
881fc932939e: Verifying Checksum
881fc932939e: Download complete
c901ba6ff9e0: Verifying Checksum
c901ba6ff9e0: Download complete
881fc932939e: Pull complete
7705a4c1b3f8: Verifying Checksum
7705a4c1b3f8: Download complete
4f376259b112: Verifying Checksum
4f376259b112: Download complete
c901ba6ff9e0: Pull complete
4f376259b112: Pull complete
7705a4c1b3f8: Pull complete
Digest: sha256:f4bb143d1135a579b630ed491e17ec50f6f7c9c6a30585ff3d385471d2a86f5a
Status: Downloaded newer image for traefik:v3.7.17
docker.io/library/traefik:v3.7.17
2026.10.4: Pulling from cloudflare/cloudflared
2cc7ee286bf3: Pulling fs layer
c172f21841df: Pulling fs layer
218cf840d0d9: Pulling fs layer
f6069939f718: Pulling fs layer
d6b1b89eccac: Pulling fs layer
2780920e5dbf: Pulling fs layer
7c12895b777b: Pulling fs layer
3214acf345c0: Pulling fs layer
52630fc75a18: Pulling fs layer
dd64bf2dd177: Pulling fs layer
b839dfae01f6: Pulling fs layer
ebddc55facdc: Pulling fs layer
a0921f9d8de9: Pulling fs layer
b96fe2995f90: Pulling fs layer
810e2b405a08: Pulling fs layer
bd8962e29291: Pulling fs layer
cac2ae0193cb: Pulling fs layer
eb8da9d09a5e: Pulling fs layer
dd64bf2dd177: Waiting
b839dfae01f6: Waiting
ebddc55facdc: Waiting
a0921f9d8de9: Waiting
b96fe2995f90: Waiting
810e2b405a08: Waiting
bd8962e29291: Waiting
cac2ae0193cb: Waiting
eb8da9d09a5e: Waiting
2780920e5dbf: Waiting
7c12895b777b: Waiting
3214acf345c0: Waiting
52630fc75a18: Waiting
f6069939f718: Waiting
d6b1b89eccac: Waiting
2cc7ee286bf3: Download complete
c172f21841df: Verifying Checksum
c172f21841df: Download complete
218cf840d0d9: Verifying Checksum
218cf840d0d9: Download complete
f6069939f718: Verifying Checksum
f6069939f718: Download complete
2cc7ee286bf3: Pull complete
d6b1b89eccac: Verifying Checksum
d6b1b89eccac: Download complete
2780920e5dbf: Verifying Checksum
2780920e5dbf: Download complete
7c12895b777b: Verifying Checksum
7c12895b777b: Download complete
3214acf345c0: Verifying Checksum
3214acf345c0: Download complete
52630fc75a18: Verifying Checksum
52630fc75a18: Download complete
dd64bf2dd177: Verifying Checksum
dd64bf2dd177: Download complete
b839dfae01f6: Verifying Checksum
b839dfae01f6: Download complete
ebddc55facdc: Verifying Checksum
ebddc55facdc: Download complete
a0921f9d8de9: Download complete
810e2b405a08: Verifying Checksum
810e2b405a08: Download complete
bd8962e29291: Download complete
b96fe2995f90: Verifying Checksum
b96fe2995f90: Download complete
cac2ae0193cb: Verifying Checksum
cac2ae0193cb: Download complete
eb8da9d09a5e: Verifying Checksum
eb8da9d09a5e: Download complete
c172f21841df: Pull complete
218cf840d0d9: Pull complete
f6069939f718: Pull complete
d6b1b89eccac: Pull complete
2780920e5dbf: Pull complete
7c12895b777b: Pull complete
3214acf345c0: Pull complete
52630fc75a18: Pull complete
dd64bf2dd177: Pull complete
b839dfae01f6: Pull complete
ebddc55facdc: Pull complete
a0921f9d8de9: Pull complete
b96fe2995f90: Pull complete
810e2b405a08: Pull complete
bd8962e29291: Pull complete
cac2ae0193cb: Pull complete
eb8da9d09a5e: Pull complete
Digest: sha256:c6c7ae1d5dfdcb094ad344ed68ccffc26635c9c6be80965664c33baf064b0515
Status: Downloaded newer image for cloudflare/cloudflared:2026.10.4
docker.io/cloudflare/cloudflared:2026.10.4
1.5.8-stable: Pulling from gtstef/filebrowser
e2de96513ba9: Pulling fs layer
93c39e448959: Pulling fs layer
b7597036c324: Pulling fs layer
c6591d351da9: Pulling fs layer
4f4fb700ef54: Pulling fs layer
74aec9809d97: Pulling fs layer
9f7b5accdf20: Pulling fs layer
19114688b75e: Pulling fs layer
2067d5df1f91: Pulling fs layer
5669e416ca05: Pulling fs layer
08c322a8a1d4: Pulling fs layer
19114688b75e: Waiting
2067d5df1f91: Waiting
5669e416ca05: Waiting
08c322a8a1d4: Waiting
c6591d351da9: Waiting
4f4fb700ef54: Waiting
74aec9809d97: Waiting
9f7b5accdf20: Waiting
e2de96513ba9: Verifying Checksum
e2de96513ba9: Download complete
c6591d351da9: Verifying Checksum
c6591d351da9: Download complete
4f4fb700ef54: Verifying Checksum
4f4fb700ef54: Download complete
93c39e448959: Verifying Checksum
93c39e448959: Download complete
b7597036c324: Verifying Checksum
b7597036c324: Download complete
e2de96513ba9: Pull complete
9f7b5accdf20: Verifying Checksum
9f7b5accdf20: Download complete
19114688b75e: Verifying Checksum
19114688b75e: Download complete
2067d5df1f91: Verifying Checksum
2067d5df1f91: Download complete
5669e416ca05: Verifying Checksum
5669e416ca05: Download complete
74aec9809d97: Verifying Checksum
74aec9809d97: Download complete
08c322a8a1d4: Verifying Checksum
08c322a8a1d4: Download complete
93c39e448959: Pull complete
b7597036c324: Pull complete
c6591d351da9: Pull complete
4f4fb700ef54: Pull complete
74aec9809d97: Pull complete
9f7b5accdf20: Pull complete
19114688b75e: Pull complete
2067d5df1f91: Pull complete
5669e416ca05: Pull complete
08c322a8a1d4: Pull complete
Digest: sha256:68455d4953bad8e984e1bc5d112ed1e9c918a0a04f104c13c9ece50790308894
Status: Downloaded newer image for gtstef/filebrowser:1.5.8-stable
docker.io/gtstef/filebrowser:1.5.8-stable
1.2.0: Pulling from admin/felhom-samba
e2de96513ba9: Already exists
0ddbc9f3a6b6: Pulling fs layer
ce57a3f93416: Pulling fs layer
fb94eeec2fe1: Pulling fs layer
ce57a3f93416: Verifying Checksum
ce57a3f93416: Download complete
fb94eeec2fe1: Verifying Checksum
fb94eeec2fe1: Download complete
0ddbc9f3a6b6: Verifying Checksum
0ddbc9f3a6b6: Download complete
0ddbc9f3a6b6: Pull complete
ce57a3f93416: Pull complete
fb94eeec2fe1: Pull complete
Digest: sha256:da96f25033827fbb77d0ee398e86fe5b744e2f29be9de898994533340c97acf9
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.2.0
gitea.dooplex.hu/admin/felhom-samba:1.2.0
[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) …
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'.
[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'.
[golden] baking the first-boot SSH host-key regeneration unit (F3) …
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'.
[golden] identity-clean + minimize …
[golden] stop + archive …
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/var/lib/felhom') in backup
INFO: archive file size: 610MB
INFO: Finished Backup of VM 9100 (00:00:27)
[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_11_11-14_00_21.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive)
[golden] publishing golden (639712511 bytes, sha256 d76336054282ec13…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.352.0/golden.tar.zst
[golden] pre-delete existing: HTTP 404 (404/204 expected)
[golden] upload OK (HTTP 201)
GOLDEN_VERSION=0.352.0
GOLDEN_SHA256=d76336054282ec138d8b0c8b9fec63319aa06f192aa421d012890b265b9abced
[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.352.0 / d76336054282ec138d8b0c8b9fec63319aa06f192aa421d012890b265b9abced
[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge)
@@ -0,0 +1,13 @@
2026-11-11T13:20:41Z
{"ok":true,"message":"Telepítés elindítva – az állapot a kártyán követhető"}
HTTP 202
13:20:51 {'deployed': True, 'state': 'deploying'}
13:21:01 {'deployed': True, 'state': 'deploying'}
13:21:11 {'deployed': True, 'state': 'starting'}
13:21:21 {'deployed': True, 'state': 'starting'}
13:21:31 {'deployed': True, 'state': 'running'}
route: 200 at 2026-11-11T13:21:40Z
negative control (no such app name): 404
Tester4-d35551 Tester4 0.173.0 floor: declared MinAgent 9.2.2 7.0.2-6-pve ONLINE 0/0 0% 18% 16% unknown (no running customer guest) 16%
2026-11-11T13:28:30Z Tester4-d35551 Tester4 0.173.0 floor: declared MinAgent 9.2.2 7.0.2-6-pve ONLINE 1/1 0% 21% 21% absent (no cloudflared container in the guest (no tunnel set up on this box)) 21% local-lvm
@@ -0,0 +1,3 @@
GET /claim HTTP/2 200
csrf_len 64 cookie_present yes
POST /claim HTTP/2 302 location: /
Binary file not shown.

After

Width:  |  Height:  |  Size: 155 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 136 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 129 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 133 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 148 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 131 KiB

@@ -0,0 +1 @@
Install pressed 13:02:31Z
@@ -0,0 +1,17 @@
== the installer address from the box and from DooPlex, 2026-11-11T13:06:42Z
-- box
2606:4700:3030::6815:12da felhom.eu
2606:4700:3037::ac43:b765 felhom.eu
search felhom.test
nameserver 192.168.0.250
404 192.168.0.192 scripts
404 192.168.0.192 root
-- dooplex
2a06:98c1:3121::3 felhom.eu
2a06:98c1:3120::3 felhom.eu
nameserver 8.8.8.8
nameserver 1.1.1.1
200 188.114.97.3 scripts
200 188.114.97.3 root
server: cloudflare
cf-ray: a58d840b9a5f28d0-VIE
@@ -0,0 +1,8 @@
e18cdd17823911e7d87da12fcb8d48b6827f1706a84b3168114fe19fe30aa1f1 /root/hi-orig.sh
197:SCRIPT_VERSION="1.36.1" # the SINGLE version source (F-1): -h and the run banners follow it.
line: 479
479a480
> ART_GOLDEN_VER=0.352.0; ART_GOLDEN_SHA=d76336054282ec138d8b0c8b9fec63319aa06f192aa421d012890b265b9abced
syntax rc=0
2026-11-11T13:06:42Z
Running as unit: install1111.service; invocation ID: 8f11a38ee7c44269a2f521571357c14d
@@ -0,0 +1,40 @@
== teardown 2026-11-11T13:48:56Z
name: golden1111-box
scsi0: nvme-scratch:9384/vm-9384-disk-0.raw,size=64G
scsi1: nvme-scratch:9384/vm-9384-disk-1.raw,size=48G
scsihw: virtio-scsi-single
stop rc=0
purging VM 9384 from related configurations..
destroy rc=0
VMID NAME STATUS
341 night1004-tester1 running
341
9202
9401
341.conf
removed '/root/vm-create-1111.sh'
9201-pkgs-after-partd.txt
9201-pkgs-before-partd.txt
e-crash.sh
e-pass.log
felhom-agent.sudoers.bak-pre-osapply
felhom-os-apply.bak-0.140.0
felhom-os-apply.bak-0.141.1
grub.default.bak-2026-10-04
spike-bootnext-setup.sh
spike-bootnext-undo.sh
spike-cleanup.sh
spike-setup.sh
spike-undo.sh
spike-watchdog-setup.sh
spike-watchdog-undo.sh
Name Type Status
felhom-pbs pbs active
local dir active
local-lvm lvmthin active
nvme-scratch dir active
4096 Nov 11 .
4096 Aug 21 ..
1705324544 Nov 11 felhom-installer-1.30.0-pve9.2-1.iso
removed '/var/lib/vz/template/iso/felhom-installer-1.30.0-pve9.2-1.iso'
@@ -0,0 +1,24 @@
update VM 9384: -scsi0 nvme-scratch:64,format=raw -scsi1 nvme-scratch:48,format=raw
Formatting '/mnt/hdd_1/images/9384/vm-9384-disk-0.raw', fmt=raw size=68719476736 preallocation=off
scsi0: successfully created disk 'nvme-scratch:9384/vm-9384-disk-0.raw,size=64G'
Formatting '/mnt/hdd_1/images/9384/vm-9384-disk-1.raw', fmt=raw size=51539607552 preallocation=off
scsi1: successfully created disk 'nvme-scratch:9384/vm-9384-disk-1.raw,size=48G'
update VM 9384: -ide2 local:iso/felhom-installer-1.30.0-pve9.2-1.iso,media=cdrom
update VM 9384: -boot order=scsi0;ide2
boot: order=scsi0;ide2
cores: 6
cpu: host
ide2: local:iso/felhom-installer-1.30.0-pve9.2-1.iso,media=cdrom,size=1665356K
memory: 10240
meta: creation-qemu=11.1.3,ctime=1794401854
name: golden1111-box
net0: virtio=BC:24:11:98:E4:72,bridge=vmbr0
ostype: l26
scsi0: nvme-scratch:9384/vm-9384-disk-0.raw,size=64G
scsi1: nvme-scratch:9384/vm-9384-disk-1.raw,size=48G
scsihw: virtio-scsi-single
smbios1: uuid=3fb5fed3-504c-4c2c-a63f-8108d5c15b6e
vga: std
vmgenid: cc62c7e4-c0d6-4766-abbf-8b12ef8f83c5
STARTED
2026-11-11T12:57:41Z
+23
View File
@@ -0,0 +1,23 @@
#!/bin/sh
# git-credential-sec.sh — git credential helper for CC's clones on DooPlex (Part C, 2026-11-11; R-925).
#
# A repository address never holds a credential. Git asks this helper for the password at the moment it needs one,
# and the helper answers from the environment that the secret store sets for one command:
#
# sec run GITEA_GIT_TOKEN -- git push origin main
#
# Per clone (no secret in any of the three settings):
# git config --local credential.helper ""
# git config --local --add credential.helper /mnt/5_hdd/felhom.eu/git/felhom.eu/scripts/git-credential-sec.sh
# git config --local credential.username <login>
#
# Without `sec run` the helper gives no password and tells git to stop asking (no prompt, no other helper, no
# clear-text credentials file). A public repository still fetches, because the server asks for nothing.
# The helper writes the value only to git's pipe: never to a file, the terminal or a log.
[ "${1:-}" = get ] || exit 0
if [ -z "${GITEA_GIT_TOKEN:-}" ]; then
printf 'git-credential-sec: no token in the environment. Run: sec run GITEA_GIT_TOKEN -- git <command>\n' >&2
printf 'quit=true\n'
exit 0
fi
printf 'password=%s\n' "$GITEA_GIT_TOKEN"
+5 -1
View File
@@ -93,7 +93,8 @@ Runs as the NON-ROOT `felhom-agent` user: `/usr/local/bin/felhom-agent --config
> **Release and delivery today (corrected 2026-10-13, R-998 session — the copy-and-restart block below is the
> pre-R-840 path):** release with `scripts/release-agent.sh <ver>` (push the code first, the CHANGELOG entry after;
> package creds = the gitea entry of `~/.docker/config.json`, read into env, never printed). A box takes it ONLY by a
> package creds = the gitea entry of `~/.docker/config.json`, read into env, never printed). **Since 2026-11-11 the felhom-agent
> clone's address holds no login:** a push, a pull or the release script's tag push runs inside `sec run GITEA_GIT_TOKEN -- …`. A box takes it ONLY by a
> signed job per box: `felhom-opsign -op agent_update -host <host_id> -key-id felhom-op-1 -key <operational key>
> -agent-version <ver> -sha256 <sha> -upload http://<hub ClusterIP>:8080 -hub-key "$REPORT_API_KEY"`, and when
> `configs/` changed also `-op agent_config_update -agent-version <ver> -bundle-sha256 <bundle sha>` (root files ride
@@ -190,6 +191,9 @@ curl -fsSL -o /tmp/rt.iso https://iso.felhom.eu/felhom-installer-<VER>-pve<PVE>.
> never appeared in the command. `%{url_effective}` and `%{referer}` do the same. Print
> `%{http_code}` alone; read the destination from `-D-` headers if you need it.
The verb is **`/discard`**, POST only (`hub/internal/web/server.go:345`); `/delete` 404s.
- **Put your SSH key on the proof host BEFORE you run the Felhom installer.** The installer closes the host on
purpose (new root password into the hub's vault, SSH by key only). With no key you are locked out and cannot read
the agent's config or logs (2026-11-11, golden 0.352.0's fresh install).
- Venue: `demo-hp`, scratch `dir` storage at **`/mnt/nvme-1tb` root** (a subdirectory reads
`disconnected` forever — the agent's `exactMount` check). Never `local-lvm`. Remove the storage at teardown.