Golden 0.352.0 vouched; token fix finished; why the N100 keeps no time; certificate pin spike on demo-hp
gates / gates (push) Successful in 3m34s

- Golden 0.352.0: bake, publish, one fresh install, the operator's vouch (R-468).
- No clone holds a login in its address; the credential helper script is in the repository (R-925, R-1390).
- N100 clock: a stale cached AppArmor profile stops chrony (R-1301, new R-1388). Read only.
- Certificate pin spike on demo-hp (R-1380): the break, the proven hand repair (runbook), the CA-file test,
  03-host-agent.md 12a, decision D80.
- Rulings of 13:17 written down (09 decisions 297-302). Register 245 -> 248 (R-1388, R-1389, R-1390).
- Docs, evidence and one helper script. No release, no floor change, no hub deploy.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-11 15:15:20 +01:00
1 parent 978666dde3
commit d4c7e84c22
74 files changed
+3376 -14

No files matched your search

@@ -0,0 +1,68 @@
# The Proxmox certificate changes and the agent's pin does not follow — spike on demo-hp, 2026-11-11
Row R-1380. Box: demo-hp only. No agent code, no release. Files: `evidence/` (26 files), `tools/` (the scripts).
Documents read: `architecture/01-topology-and-trust.md`, `03-host-agent.md`, `04-control-plane-authorization.md`;
the End-dates list entry DB02 and its `evidence/box/`. **No architecture document named this pin before today**
(a search of 01, 03 and 04 found no line; now `03-host-agent.md` §12a).
## What was done, in order (times in UTC)
| Time | Step | Result | File |
|---|---|---|---|
| 13:49:16 | Before | served certificate = file = agent's pin (`07:B5:72:5D…07:BB`, ends 2028-08-19); agent calls answer 200 | `01`, `05` |
| — | Who else trusts it | port 8006's log: only the agent, from 127.0.0.1 (75,374 lines). No file outside the box holds the fingerprint | `02` |
| 13:49:23 | `pvecm updatecerts --force` | new file (`21:D6:B5:E0…84:12`); **Proxmox still serves the old one, the agent still works** | `06`, `07` |
| 13:50:23 | `systemctl reload pveproxy` | the new certificate is served about 10 s later | `08` |
| 13:50:39 | The agent | first `TLS pin mismatch` line; every call to Proxmox fails from here | `09`, `10` |
| 13:59:25 | Hand repair | new fingerprint into `agent.json`, restart: **a working call 2.1 s later** | `16`, `17` |
| 13:59:39 | CA file in place of the fingerprint | the agent connects (2.2 s) | `18`, `19` |
| 13:59:54 | Second forced certificate (`1F:75:D3:51…0E:74`) + reload | **the agent goes on, no restart, no hand step, 0 TLS lines** | `20` |
| 14:01:02 | Control: a CA file that did not sign it | `x509: certificate signed by unknown authority` (so the CA file is really checked) | `25` |
| 14:02:46 | Back to the standard | pin = served certificate; 77 of 77 capabilities; hub report at 14:02:53 | `22`, `24`, `26` |
## What broke while the pin was wrong (8 min 56 s, by choice)
- **The agent's line:** `proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint`, as a
WARN in each loop. It does not say "the certificate changed" and does not say what to do.
- **Failed:** the guest list, the storage view, the guest config, the pool read, the node status. So: reconcile
(`pass failed`), the controller supervisor, the guest-power watchdog, the guest network sweep, the stale-lock
recovery, and the local API's `/storage` and `/disks` (502 to the controller). From the code, not tried: backup,
restore, snapshot, start/stop, resize all use the same client, so they fail too.
- **The hub:** the agent sent **no host report** (`hub: collect failed; skipping this cycle's report`). The Hosts page
still read `ONLINE 1/2` eleven minutes after the last report. Not measured: when it turns to offline.
- **The alarm:** one mail to the operator after 94 s, `agent_channel_unknown (warning) — Controller→agent channel
down: agentapi: GET /storage: HTTP 502`. It names a symptom, not the cause. A recovery event came 39 s after the
repair.
- **Stayed up:** both guests, Tester 1's VM, all 26 containers in the guest, the enrolled drive.
## Three facts the fix must know
1. **Proxmox's own renewal reloads the web service by itself** (`pveupdate`, daily timer): node certificate at most
2 × 365 days, renewed when it ends within 14 days. So on a real box the break comes the same night (`03`).
2. **The agent cannot read Proxmox's certificate files.** It runs as user `felhom-agent`; `/etc/pve/pve-root-ca.pem`
and the node certificate are `0640 root:www-data`. The CA test used a copy in `/etc/felhom-agent/` (`18`). The
ruled fix shape ("the agent reads the certificate file on its own host") needs a root step for the same reason.
3. **The fingerprint wins.** Config key `proxmox.tls.ca_file` exists, but it is used only when
`proxmox.tls.fingerprint` is empty (`felhom-agent/internal/proxmox/tls.go`).
The root CA lasts 10 years (demo-hp: to 2036-08-17). Proxmox renews it when less than two years are left (about
August 2034) and **keeps its key and its name**, so new node certificates still pass a copy of the old CA file until
that copy's own end date. Read from Proxmox's code (`03`), not forced.
## The two ways, and the pick → `03-host-agent.md` §12a, decision D80 in `STATUS.md`
**CA file** (CC's pick): proven here, no agent code. **Follow the fingerprint**: needs new agent code and a root
helper. Both need one root step to reach the boxes that exist.
## CC's mistakes in this run
- The first control ran under 1 s (the waiter matched an older call): void. The repeat met systemd's start limit
(4 starts in 2 minutes): **the agent was stopped for 92 s** (14:01:14 to 14:02:46). The failure hook ran and did
nothing ("no pending update"). The proof for the control comes from the 1-second run's own log (`23`, `24`, `25`).
- The scripts rewrote `agent.json` with a new layout. The content is equal to the saved copy (`24`: `rest equal: True`).
## How demo-hp was left
Fingerprint pin equal to the served certificate (`1F:75:D3:51…0E:74`, ends 2028-11-09). Agent active, 0 restarts,
calls answer 200. The CA copy and the saved config are removed. **The certificate itself is new**: that is the one
lasting change, and it moves demo-hp's end date from 2028-08-19 to 2028-11-09.
@@ -0,0 +1,187 @@
== demo-hp 2026-11-11T13:26:29Z pve pve-manager/9.2.26/41a0e2e09b9e6f9d (running kernel: 7.0.14-28-pve)
== served certificate (network read, 127.0.0.1:8006)
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Aug 20 15:44:42 2026 GMT
notAfter=Aug 19 15:44:42 2028 GMT
sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB
X509v3 Subject Alternative Name:
IP Address:127.0.0.1, IP Address:0:0:0:0:0:0:0:1, DNS:localhost, IP Address:192.168.0.104, DNS:demo-hp, DNS:demo-hp.local
== files
-rw------- 1 root www-data 3272 2026-08-21 17:44:42.000000000 +0200 /etc/pve/priv/pve-root-ca.key
-rw-r----- 1 root www-data 2094 2026-08-21 17:44:42.000000000 +0200 /etc/pve/pve-root-ca.pem
/etc/pve/local/:
total 2
drwxr-xr-x 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 .
drwxr-xr-x 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 ..
-rw-r----- 1 root www-data 83 2026-11-11 14:26:26.000000000 +0100 lrm_status
drwxr-xr-x 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 lxc
drwxr-xr-x 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 openvz
drwx------ 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 priv
-rw-r----- 1 root www-data 1704 2026-08-21 17:44:42.000000000 +0200 pve-ssl.key
-rw-r----- 1 root www-data 1785 2026-08-21 17:44:42.000000000 +0200 pve-ssl.pem
drwxr-xr-x 2 root www-data 0 2026-08-21 17:44:42.000000000 +0200 qemu-server
-rw-r----- 1 root www-data 560 2026-11-10 15:43:03.000000000 +0100 ssh_known_hosts
-- /etc/pve/local/pve-ssl.pem
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Aug 20 15:44:42 2026 GMT
notAfter=Aug 19 15:44:42 2028 GMT
sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB
-- /etc/pve/pve-root-ca.pem
subject=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Aug 20 15:44:42 2026 GMT
notAfter=Aug 17 15:44:42 2036 GMT
sha256 Fingerprint=E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7
ls: cannot access '/etc/pve/local/pveproxy-ssl.pem': No such file or directory
== agent
felhom-agent 0.173.0
active
Restart=on-failure
NRestarts=0
ActiveEnterTimestamp=Tue 2026-11-10 15:58:17 CET
proxmox keys: ['endpoint', 'node', 'tls', 'token']
endpoint : https://127.0.0.1:8006
node : demo-hp
token : <present>
tls: {'fingerprint': '07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB', 'insecure_skip_verify': False}
total 28
drwxr-xr-x 2 root root 4096 2026-09-28 16:25:22.412907166 +0200 .
drwxr-xr-x 102 root root 4096 2026-11-08 02:49:07.920613811 +0100 ..
-rw------- 1 felhom-agent felhom-agent 2454 2026-09-28 16:25:22.443523132 +0200 agent.json
-rw------- 1 felhom-agent felhom-agent 2498 2026-09-24 15:33:45.307471226 +0200 agent.json.night-0925-off
-rw------- 1 felhom-agent felhom-agent 2451 2026-09-24 21:56:34.103084862 +0200 agent.json.pre-a4-retention
-rw------- 1 felhom-agent felhom-agent 2451 2026-09-24 21:57:43.930322856 +0200 agent.json.pre-d44
-rw------- 1 root root 2451 2026-08-21 18:09:15.470472229 +0200 agent.json.pre-r672
8:# allowlist in /etc/sudoers.d/felhom-agent (configs/felhom-agent.sudoers). The User= here and the
39:User=felhom-agent
41:ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json
== agent log, last proxmox lines
4422
pam_unix(sudo:session): session closed for user root
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- cat /etc/network/interfaces
pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
pam_unix(sudo:session): session closed for user root
pam_unix(sudo:session): session closed for user root
felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/pct exec 9201 -- pgrep -x dhclient
pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
pam_unix(sudo:session): session closed for user root
== who calls port 8006 (pveproxy access log, client addresses, last 2 days)
75375 ::ffff:127.0.0.1
-- users seen
75374 ::ffff:127.0.0.1 felhom-agent@pve!agent
1 ::ffff:127.0.0.1 -
== listening / other TLS users of this cert
LISTEN 0 4096 127.0.0.1:85 0.0.0.0:* users:(("pvedaemon worke",pid=235161,fd=6),("pvedaemon worke",pid=168847,fd=6),("pvedaemon w
LISTEN 0 4096 *:8006 *:* users:(("pveproxy worker",pid=316496,fd=6),("pveproxy worker",pid=308404,fd=6),("pveproxy wo
LISTEN 0 4096 *:3128 *:* users:(("spiceproxy work",pid=1733482,fd=6),("spiceproxy",pid=1399,fd=6))
== in flight
running tasks: []
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
341 night1004-tester1 running 8192 200.00 1950
9384 golden1111-box running 10240 64.00 313884
VMID Status Lock Name
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
== tools' own help
USAGE: pvecm updatecerts [OPTIONS]
Update node certificates (and generate all needed files/directories).
--force <boolean>
Force generation of new SSL certificate.
--silent <boolean>
Ignore errors (i.e. when cluster has no quorum).
--unmerge-known-hosts <boolean> (default=0)
Unmerge legacy SSH known hosts.
USAGE: pvenode cert delete [<restart>]
DELETE custom certificate chain and key.
<restart> <boolean> (default=0)
Restart pveproxy.
USAGE: pvenode cert info [FORMAT_OPTIONS]
Get information about node's certificates.
USAGE: pvenode cert set <certificates> [<key>] [OPTIONS] [FORMAT_OPTIONS]
Upload or update custom certificate chain and key.
<certificates> <string>
PEM encoded certificate (chain).
<key> <string>
PEM encoded private key.
--force <boolean> (default=0)
Overwrite existing custom or ACME certificate files.
--restart <boolean> (default=0)
Restart pveproxy.
== renew rule in code
83:my $pveca_cert_days = 3650;
85:my $pvessl_cert_max_days = 2 * 365;
86:my $pvessl_cert_days_before_ca = 2;
438: '-days',
439: $pveca_cert_days,
478:# certificate chain up to the renewed one, and clients trusting the previous one accept new node
479:# certificates until it expires. Returns the expiry of the previous one if it got renewed.
480:sub renew_pveca_cert {
483: my $renewal_reason = sub {
484: my $min_days = $pvessl_cert_max_days + $pvessl_cert_days_before_ca;
485: if (PVE::Certificate::check_expiry($pveca_cert_fn, time() + $min_days * 24 * 60 * 60)) {
486: return 'expires within two years';
501: my $reason = $renewal_reason->() or return undef;
503: my $old_expiry = PVE::Cluster::cfs_lock_domain(
507: return undef if !$renewal_reason->(); # another node renewed it in the meantime
519: '-days', $pveca_cert_days,
528: die "subject of renewed certificate differs\n";
535: die "renewing pve root certificate failed:\n$err";
543: if ($old_expiry) {
544: my $deadline = localtime($old_expiry);
545: print "Renewed cluster CA certificate, as the previous one $reason. Clients that trust"
546: . " the previous one must import the renewed $pveca_cert_fn before $deadline\n";
549: return $old_expiry;
579:# and its expiry. A certificate from another issuer was placed there by an admin and must be kept.
685: # get ca expiry
687: my $daysleft = int(($cainfo->{notafter} - time()) / (24 * 60 * 60));
689: if ($daysleft < 14) {
690: die "CA expires in less than 2 weeks, unable to generate certificate.\n";
693: $daysleft -= $pvessl_cert_days_before_ca;
695: if ($daysleft >= $pvessl_cert_max_days) {
/usr/bin/pveupdate:86: my $lead_time = PVE::Certificate::get_cert_renew_lead_time($cert);
/usr/bin/pveupdate:88: if (PVE::Certificate::check_expiry($cert, time() + $lead_time)) {
/usr/bin/pveupdate:89: PVE::API2::ACME->renew_certificate({ node => $nodename });
/usr/bin/pveupdate:93: 'Custom certificate does not expire soon, skipping ACME renewal.',
/usr/bin/pveupdate:99: 'ACME config found for node, but no custom certificate exists. Skipping ACME renewal until initial certificate has been deployed.',
/usr/bin/pveupdate:104:syslog('err', "Renewing ACME certificate failed: $@") if $@;
/usr/bin/pveupdate:106:eval { PVE::Cluster::Setup::renew_pveca_cert() };
/usr/bin/pveupdate:107:syslog('err', "Renewing cluster CA certificate failed: $@") if $@;
/usr/bin/pveupdate:131: my $renew = sub {
/usr/bin/pveupdate:142: print "Renewed PVE certificate is valid until $expiry\n";
/usr/bin/pveupdate:144: print "Restarting pveproxy after renewing certificate\n";
/usr/bin/pveupdate:152: $renew->("was issued by a replaced cluster CA, renewing...", 1);
/usr/bin/pveupdate:153: } elsif (PVE::Certificate::check_expiry($certpath)) {
/usr/bin/pveupdate:155: $renew->("expired, renewing...");
/usr/bin/pveupdate:156: } elsif (PVE::Certificate::check_expiry($certpath, time() + 14 * 24 * 60 * 60)) {
/usr/bin/pveupdate:158: $renew->("expires soon, renewing...");
/usr/bin/pveupdate:159: } elsif (!PVE::Certificate::check_expiry($certpath, time() + 2 * 365 * 24 * 60 * 60)) {
/usr/bin/pveupdate:161: $renew->(
/usr/bin/pveupdate:162: "expires in more than 2 years, renewing to reduce certificate life-span for client compatibility..."
/usr/bin/pveupdate:166:syslog('err', "Checking/Renewing SSL certificate failed: $@") if $@;
uid=999(felhom-agent) gid=990(felhom-agent) groups=990(felhom-agent),999(systemd-journal)
agent CANNOT read the root CA file
agent CANNOT read the leaf file
755 root:www-data /etc/pve
640 root:www-data /etc/pve/pve-root-ca.pem
@@ -0,0 +1,20 @@
== who else holds demo-hp's fingerprint? 2026-11-11T13:26:49Z
-- workspace files (colon form, plain form), names only, audits' evidence left out
(end of list)
-- positive control: the same search with audits in
felhom.eu/documentation/audits/end-dates-list-2026-11-11/evidence/box/demo-hp-host-dates.txt
felhom.eu/documentation/audits/evidence-fixes-first-tester-2026-09-30/partC/agent-0138-live.txt
-- hub code: does the hub store or use a box's Proxmox fingerprint?
felhom.eu/hub/internal/osupdates/service.go:704: if rel, _ := s.Store.LatestOSRelease(LayerPVE); rel != nil && rel.Fingerprint == fp {
(end)
-- agent: where the pin is written
felhom-agent/internal/localapi/cert.go:29:// is absent. It returns the tls.Certificate to serve and the leaf's SHA-256 fingerprint (the
felhom-agent/internal/localapi/cert.go:38:func EnsureLeaf(certPath, keyPath, bridgeHost string) (cert tls.Certificate, fingerprint string, generated bool, err error) {
felhom-agent/internal/config/config.go:916: if v := os.Getenv("FELHOM_AGENT_PROXMOX_TLS_FINGERPRINT"); v != "" {
felhom-agent/internal/config/config.go:917: cfg.Proxmox.TLS.Fingerprint = v
felhom-agent/internal/proxmox/tls.go:55: return fmt.Errorf("proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint")
felhom-agent/cmd/felhom-agent/main.go:340: Fingerprint: cfg.Proxmox.TLS.Fingerprint,
felhom.eu/scripts/felhom-host-install.sh:3182: # TLS pin: the SERVED leaf cert fingerprint (not pvesh node info — may differ)
felhom.eu/scripts/felhom-host-install.sh:3185: [[ -n "$fp" ]] || log_warn " could not compute TLS fingerprint (leaving empty — agent will use system trust)"
felhom.eu/scripts/felhom-host-install.sh:3206: log_dry "write $AGENT_CONFIG (0600): proxmox{endpoint,node=$NODE,token=<secret>,tls.fingerprint=$fp} hub{url=$HUB_URL,host_id=$HOST_ID,a
felhom.eu/scripts/felhom-host-install.sh:3335: "tls": {"fingerprint": os.environ['FP'], "insecure_skip_verify": False},
@@ -0,0 +1,318 @@
2026-11-11T13:27:23Z
libpve-cluster-perl 9.1.9
pve-cluster 9.1.9
== Setup.pm 470-560
gen_pveca_key();
$generate_pveca_cert->($pveca_key_fn, $pveca_cert_fn);
return 1;
}
# Keeping the key and the exact encoded subject lets node certificates issued by the previous CA
# certificate chain up to the renewed one, and clients trusting the previous one accept new node
# certificates until it expires. Returns the expiry of the previous one if it got renewed.
sub renew_pveca_cert {
return undef if !-f $pveca_key_fn || !-f $pveca_cert_fn;
my $renewal_reason = sub {
my $min_days = $pvessl_cert_max_days + $pvessl_cert_days_before_ca;
if (PVE::Certificate::check_expiry($pveca_cert_fn, time() + $min_days * 24 * 60 * 60)) {
return 'expires within two years';
}
my $key_usage;
my $cmd = ['openssl', 'x509', '-noout', '-ext', 'keyUsage', '-in', $pveca_cert_fn];
PVE::Tools::run_command(
$cmd,
outfunc => sub { $key_usage = 1 },
errfunc => sub { },
timeout => $pveca_lock_cmd_timeout,
);
return 'lacks the keyUsage extension that strict TLS clients require' if !$key_usage;
return undef;
};
my $reason = $renewal_reason->() or return undef;
my $old_expiry = PVE::Cluster::cfs_lock_domain(
'pve-root-ca',
undef,
sub {
return undef if !$renewal_reason->(); # another node renewed it in the meantime
my $old = PVE::Certificate::get_certificate_info($pveca_cert_fn);
my $tmp_fn = "$pveca_cert_fn.tmp.$$";
my $cmd = [
[
'openssl', 'x509', '-x509toreq', # keeps the exact encoded subject
'-in', $pveca_cert_fn, '-key', $pveca_key_fn,
],
[
'faketime', 'yesterday', # NOTE: wrap openssl with faketime to prevent bug #904
'openssl', 'req', '-x509', '-in', '/dev/stdin', '-key', $pveca_key_fn,
'-days', $pveca_cert_days,
'-addext', 'keyUsage=critical,keyCertSign,cRLSign',
'-out', $tmp_fn,
],
];
eval {
run_silent_cmd($cmd, timeout => $pveca_lock_cmd_timeout);
my $new = PVE::Certificate::get_certificate_info($tmp_fn);
if ($new->{subject} ne $old->{subject}) {
die "subject of renewed certificate differs\n";
}
rename($tmp_fn, $pveca_cert_fn)
or die "replacing '$pveca_cert_fn' failed - $!\n";
};
if (my $err = $@) {
unlink $tmp_fn;
die "renewing pve root certificate failed:\n$err";
}
return $old->{notafter};
},
);
die $@ if $@;
if ($old_expiry) {
my $deadline = localtime($old_expiry);
print "Renewed cluster CA certificate, as the previous one $reason. Clients that trust"
. " the previous one must import the renewed $pveca_cert_fn before $deadline\n";
}
return $old_expiry;
}
sub rotate_pveca {
my $tmp_key_fn = "$pveca_key_fn.tmp.$$";
my $tmp_cert_fn = "$pveca_cert_fn.tmp.$$";
eval {
$generate_pveca_key->($tmp_key_fn);
$generate_pveca_cert->($tmp_key_fn, $tmp_cert_fn);
PVE::Cluster::cfs_lock_domain(
'pve-root-ca',
== Setup.pm 575-720
}
}
# Returns who issued the node certificate, the 'current-ca', a 'replaced-ca' or an 'other' issuer,
# and its expiry. A certificate from another issuer was placed there by an admin and must be kept.
sub pve_ssl_cert_status {
my ($nodename) = @_;
my $pvessl_cert_fn = "$pmxcfs_base_dir/nodes/$nodename/pve-ssl.pem";
return undef if !-f $pvessl_cert_fn;
my $cert = PVE::Certificate::get_certificate_info($pvessl_cert_fn);
my $issuer = $cert->{issuer};
my $issued_by = 'other';
if ($issuer eq PVE::Certificate::get_certificate_info($pveca_cert_fn)->{subject}) {
$issued_by = 'current-ca';
} elsif ($issuer =~ m!^/CN=Proxmox Virtual Environment/OU=[^/]+/O=PVE Cluster Manager CA$!) {
$issued_by = 'replaced-ca';
}
return { 'issued-by' => $issued_by, notafter => $cert->{notafter} };
}
sub gen_pve_ssl_key {
my ($nodename) = @_;
die "no node name specified" if !$nodename;
my $pvessl_key_fn = "$pmxcfs_base_dir/nodes/$nodename/pve-ssl.key";
return if -f $pvessl_key_fn;
eval { run_silent_cmd(['openssl', 'genrsa', '-out', $pvessl_key_fn, '2048']); };
die "unable to generate pve ssl key for node '$nodename':\n$@" if $@;
}
sub gen_pve_www_key {
return if -f $pvewww_key_fn;
eval { run_silent_cmd(['openssl', 'genrsa', '-out', $pvewww_key_fn, '2048']); };
die "unable to generate pve www key:\n$@" if $@;
}
sub update_serial {
my ($serial) = @_;
PVE::Tools::file_set_contents($pveca_srl_fn, $serial);
}
sub gen_pve_ssl_cert {
my ($force, $nodename, $ip) = @_;
die "no node name specified" if !$nodename;
die "no IP specified" if !$ip;
my $pvessl_cert_fn = "$pmxcfs_base_dir/nodes/$nodename/pve-ssl.pem";
return if !$force && -f $pvessl_cert_fn;
my $names = "IP:127.0.0.1,IP:::1,DNS:localhost,IP:${ip},DNS:${nodename}";
my $fqdn = $nodename;
my $rc = PVE::INotify::read_file('resolvconf');
if ($rc && $rc->{search}) {
$fqdn .= ".$rc->{search}";
$names .= ",DNS:$fqdn";
}
my $reqfn = "/run/pve-cluster/pvecertreq-$$.tmp";
unlink $reqfn;
my $pvessl_key_fn = "$pmxcfs_base_dir/nodes/$nodename/pve-ssl.key";
eval {
run_silent_cmd([
'openssl',
'req',
'-batch',
'-new',
'-key',
$pvessl_key_fn,
'-out',
$reqfn,
'-subj',
"/OU=PVE Cluster Node/O=Proxmox Virtual Environment/CN=$fqdn",
'-addext',
'basicConstraints=CA:FALSE',
'-addext',
'extendedKeyUsage=serverAuth',
'-addext',
"subjectAltName=$names",
]);
};
if (my $err = $@) {
unlink $reqfn;
die "unable to generate pve certificate request:\n$err";
}
# the lock also covers the serial file that all nodes share
PVE::Cluster::cfs_lock_domain(
'pve-root-ca',
undef,
sub {
update_serial("0000000000000000") if !-f $pveca_srl_fn;
# get ca expiry
my $cainfo = PVE::Certificate::get_certificate_info($pveca_cert_fn);
my $daysleft = int(($cainfo->{notafter} - time()) / (24 * 60 * 60));
if ($daysleft < 14) {
die "CA expires in less than 2 weeks, unable to generate certificate.\n";
}
$daysleft -= $pvessl_cert_days_before_ca;
if ($daysleft >= $pvessl_cert_max_days) {
$daysleft = $pvessl_cert_max_days;
}
my $cmd = [
'faketime', 'yesterday', # NOTE: wrap openssl with faketime to prevent bug #904
'openssl', 'x509', '-req', '-in', $reqfn, '-days', $daysleft, '-out',
$pvessl_cert_fn,
'-CAkey', $pveca_key_fn, '-CA', $pveca_cert_fn, '-CAserial', $pveca_srl_fn,
'-copy_extensions', 'copyall',
];
run_silent_cmd($cmd, timeout => $pveca_lock_cmd_timeout);
},
);
my $err = $@;
unlink $reqfn or $!{ENOENT} or warn "failed to clean up '$reqfn' - $!";
die "unable to generate pve ssl certificate:\n$err" if $err;
}
sub gen_pve_node_files {
my ($nodename, $ip, $opt_force) = @_;
gen_local_dirs($nodename);
== pveupdate 100-170
);
}
}
};
syslog('err', "Renewing ACME certificate failed: $@") if $@;
eval { PVE::Cluster::Setup::renew_pveca_cert() };
syslog('err', "Renewing cluster CA certificate failed: $@") if $@;
eval {
my $certpath = PVE::CertHelpers::default_cert_path_prefix($nodename) . ".pem";
my $capath = "/etc/pve/pve-root-ca.pem";
my $check_issued_by_ca = sub {
# get CA info
my $cainfo = PVE::Certificate::get_certificate_info($capath);
# get cert and check issuer and chain metadata
my $certinfo = PVE::Certificate::get_certificate_info($certpath);
if ($certinfo->{issuer} ne $cainfo->{subject}) {
die "SSL certificate ($certpath) is not issued by root CA ($capath)!\n";
}
# check if cert is really signed by the ca
# TODO: replace by low level ssleay interface if version 1.86 is available
my $cmd = [
'/usr/bin/openssl', 'verify', '-no_check_time', '-CAfile', $capath, '--', $certpath,
];
PVE::Tools::run_command($cmd);
};
my $renew = sub {
my ($msg, $replaced_ca) = @_;
# one from a replaced cluster CA fails these checks, but is still ours to replace
$check_issued_by_ca->() if !$replaced_ca;
print "PVE certificate $msg\n";
# create new certificate
my $ip = PVE::Cluster::remote_node_ip($nodename);
PVE::Cluster::Setup::gen_pve_ssl_cert(1, $nodename, $ip);
my $expiry = localtime(PVE::Certificate::get_expiration_as_epoch($certpath));
print "Renewed PVE certificate is valid until $expiry\n";
print "Restarting pveproxy after renewing certificate\n";
PVE::Tools::run_command(['systemctl', 'reload-or-restart', 'pveproxy']);
};
my $status = eval { PVE::Cluster::Setup::pve_ssl_cert_status($nodename) };
syslog('err', "Checking SSL certificate failed: $@") if $@;
if ($status && $status->{'issued-by'} eq 'replaced-ca') {
$renew->("was issued by a replaced cluster CA, renewing...", 1);
} elsif (PVE::Certificate::check_expiry($certpath)) {
# already expired
$renew->("expired, renewing...");
} elsif (PVE::Certificate::check_expiry($certpath, time() + 14 * 24 * 60 * 60)) {
# expires in next 2 weeks
$renew->("expires soon, renewing...");
} elsif (!PVE::Certificate::check_expiry($certpath, time() + 2 * 365 * 24 * 60 * 60)) {
# expires in more than 2 years
$renew->(
"expires in more than 2 years, renewing to reduce certificate life-span for client compatibility..."
);
}
};
syslog('err', "Checking/Renewing SSL certificate failed: $@") if $@;
sub cleanup_tasks {
my $taskdir = "/var/log/pve/tasks";
== timer
NEXT LEFT LAST PASSED UNIT ACTIVATES
Thu 2026-11-12 01:46:13 CET 11h Wed 2026-11-11 02:22:58 CET 12h ago pve-daily-update.timer pve-daily-update.service
@@ -0,0 +1,7 @@
== preconditions 2026-11-11T13:49:13Z
demo-hp running tasks: []
(end of process list)
config saved on the box (0600, stays on the box)
57f9650d1ca60fcc
tester1 running tasks: []
(end)
@@ -0,0 +1,20 @@
== state 2026-11-11T13:49:16Z
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Aug 20 15:44:42 2026 GMT
notAfter=Aug 19 15:44:42 2028 GMT
sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB
agent tls config: {'fingerprint': '07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB', 'insecure_skip_verify': False}
agent: active since Tue 2026-11-10 15:58:17 CET restarts=0
-- agent calls in pveproxy's access log, last 4 (time, call, answer code)
[11/11/2026:14:49:07 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
[11/11/2026:14:49:07 +0100] "GET /api2/json/storage 200
[11/11/2026:14:49:08 +0100] "GET /api2/json/nodes/demo-hp/storage 200
[11/11/2026:14:49:08 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6)
0
-- guests
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
341 night1004-tester1 running
@@ -0,0 +1,10 @@
== force 2026-11-11T13:49:23Z
before: sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB
(re)generate node files
generate new node certificate
merge authorized SSH keys
updatecerts rc=0
file now: sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12 notAfter=Nov 9 13:49:24 2028 GMT
served 5 s later, no restart: sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB
pveproxy: Tue 2026-11-10 15:43:04 CET reloads:
done 2026-11-11T13:49:29Z
@@ -0,0 +1,20 @@
== state 2026-11-11T13:50:14Z
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Aug 20 15:44:42 2026 GMT
notAfter=Aug 19 15:44:42 2028 GMT
sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB
agent tls config: {'fingerprint': '07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB', 'insecure_skip_verify': False}
agent: active since Tue 2026-11-10 15:58:17 CET restarts=0
-- agent calls in pveproxy's access log, last 4 (time, call, answer code)
[11/11/2026:14:50:07 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
[11/11/2026:14:50:07 +0100] "GET /api2/json/storage 200
[11/11/2026:14:50:08 +0100] "GET /api2/json/nodes/demo-hp/storage 200
[11/11/2026:14:50:08 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6)
0
-- guests
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
341 night1004-tester1 running
@@ -0,0 +1,24 @@
== the step Proxmox's own nightly job does after a renewal: systemctl reload-or-restart pveproxy; 2026-11-11T13:50:23Z
reload rc=0
served: sha256 Fingerprint=07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB notAfter=Aug 19 15:44:42 2028 GMT
2026-11-11T13:50:28Z
2026-11-11T13:50:34Z
served now: sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12
2026-11-11T14:50:24+01:00 demo-hp systemd[1]: Reloaded pveproxy.service - PVE API Proxy Server.
2026-11-11T14:50:25+01:00 demo-hp pveproxy[1293]: restarting server
2026-11-11T14:50:25+01:00 demo-hp pveproxy[1293]: starting 3 worker(s)
2026-11-11T14:50:25+01:00 demo-hp pveproxy[1293]: worker 494020 started
2026-11-11T14:50:25+01:00 demo-hp pveproxy[1293]: worker 494021 started
2026-11-11T14:50:25+01:00 demo-hp pveproxy[1293]: worker 494022 started
2026-11-11T14:50:30+01:00 demo-hp pveproxy[316496]: worker exit
2026-11-11T14:50:30+01:00 demo-hp pveproxy[308404]: worker exit
2026-11-11T14:50:30+01:00 demo-hp pveproxy[203540]: worker exit
2026-11-11T14:50:31+01:00 demo-hp pveproxy[1293]: worker 316496 finished
2026-11-11T14:50:31+01:00 demo-hp pveproxy[1293]: worker 203540 finished
2026-11-11T14:50:31+01:00 demo-hp pveproxy[1293]: worker 308404 finished
PID STARTED CMD
1293 Tue Nov 10 15:43:04 2026 pveproxy
1293 Tue Nov 10 15:43:04 2026 pveproxy
494020 Wed Nov 11 14:50:25 2026 pveproxy worker
494021 Wed Nov 11 14:50:25 2026 pveproxy worker
494022 Wed Nov 11 14:50:25 2026 pveproxy worker
@@ -0,0 +1,26 @@
== state 2026-11-11T13:52:21Z
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Nov 10 13:49:24 2026 GMT
notAfter=Nov 9 13:49:24 2028 GMT
sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12
agent tls config: {'fingerprint': '07:B5:72:5D:5E:C1:23:51:4D:58:B9:38:E2:0E:6E:AA:4E:D3:54:10:44:61:0C:27:5E:44:07:83:65:05:07:BB', 'insecure_skip_verify': False}
agent: active since Tue 2026-11-10 15:58:17 CET restarts=0
-- agent calls in pveproxy's access log, last 4 (time, call, answer code)
[11/11/2026:14:50:19 +0100] "GET /api2/json/pools/felhom 200
[11/11/2026:14:50:19 +0100] "GET /api2/json/nodes/demo-hp/lxc 200
[11/11/2026:14:50:19 +0100] "GET /api2/json/pools/felhom 200
[11/11/2026:14:50:19 +0100] "GET /api2/json/nodes/demo-hp/storage 200
-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6)
22
time=2026-11-11T14:52:05.144+01:00 level=WARN msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" vmid=9201 target=felhom-pbs err="proxmox: GET /nodes/demo-hp/storage/felhom-pbs/content: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/storage/felhom-pbs/content\": pr
time=2026-11-11T14:52:05.505+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:52:19.211+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:52:19.230+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:52:19.393+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:52:19.394+01:00 level=WARN msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
-- guests
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
341 night1004-tester1 running
@@ -0,0 +1,37 @@
== all agent log lines since the new certificate was served (no sudo noise), first 25, cut
2026-11-11T14:50:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:50:39.237+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:50:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:50:39.323+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
2026-11-11T14:50:49+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:50:49.395+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:50:59+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:50:59.230+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:50:59+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:50:59.310+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
2026-11-11T14:51:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:05.501+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:51:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:19.212+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:51:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:19.231+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:51:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:19.313+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
2026-11-11T14:51:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:19.395+01:00 level=WARN msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:51:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:19.395+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:51:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:39.233+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:51:39+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:39.310+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
2026-11-11T14:51:49+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:49.395+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:51:59+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:59.231+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:51:59+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:51:59.313+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.111+01:00 level=WARN msg="local-api: storage view unavailable for the tier presence report" target=local err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.116+01:00 level=WARN msg="local-api: storage view unavailable for the tier presence report" target=felhom-pbs err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.124+01:00 level=WARN msg="local-api: storage view unavailable for the backup-target presence check — assuming present" target=local err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.129+01:00 level=WARN msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" vmid=9201 target=local err="proxmox: GET /nodes/demo-hp/storage/local/content: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/storage/local/content\": proxmox: TLS pin mismatch: server cert sha256 does n
2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.136+01:00 level=WARN msg="local-api: storage view unavailable for the backup-target presence check — assuming present" target=felhom-pbs err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.144+01:00 level=WARN msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record" vmid=9201 target=felhom-pbs err="proxmox: GET /nodes/demo-hp/storage/felhom-pbs/content: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/storage/felhom-pbs/content\": proxmox: TLS pin mismatch: server cer
2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.144+01:00 level=INFO msg="local-api: backup storage unreadable — due-check uses the last success saved on disk (R-894)" vmid=9201 target=felhom-pbs saved=2026-11-05T20:50:35Z
2026-11-11T14:52:05+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:05.505+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
2026-11-11T14:52:19+01:00 demo-hp felhom-agent[66863]: time=2026-11-11T14:52:19.211+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
== count by message
6 msg="reconcile: storage view unavailable — using mount-table only"
6 msg="reconcile: enrolled drive bound under shared parent (live, no reboot)"
4 msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)"
2 msg="local-api: storage view unavailable for the tier presence report"
2 msg="local-api: storage view unavailable for the backup-target presence check — assuming present"
2 msg="local-api: /storage guest config"
2 msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record"
2 msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)"
2 msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)"
1 msg="local-api: backup storage unreadable — due-check uses the last success saved on disk (R-894)"
@@ -0,0 +1,17 @@
2026-11-11T13:54:07Z
HUB: .173.0 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 23% 37% running (connected) 37% local demo-hp-bb76ea Demo HP 0.173.0 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 73% 45% running (connected) 59% local-lvm tester-1-d70be4 Tester 1 0.173.0 9.2.2 7.0.14-22-pve ONLINE 1/2 8% 29% 33% running (connected) 33% local Felhom Hub 0.180.0
--
== guest 9201 containers: running count / not running
26
== agent report lines to the hub since 14:50
time=2026-11-11T14:52:05.111+01:00 level=WARN msg="local-api: storage view unavailable for the tier presence report" target=local err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configure
time=2026-11-11T14:52:05.116+01:00 level=WARN msg="local-api: storage view unavailable for the tier presence report" target=felhom-pbs err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match conf
== agent local API (what the controller asks) last lines
2026/11/11 13:54:05 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping bookstack — last check 3m0s ago, effective interval 5m0s, healthy=true
2026/11/11 13:54:05 healthprobe.go:64: [DEBUG] [stacks] RunHealthProbes: skipping home-assistant — last check 3m0s ago, effective interval 5m0s, healthy=true
2026/11/11 13:54:05 scheduler.go:450: [INFO] [scheduler] Running job: agent-channel-health
2026/11/11 13:54:05 scheduler.go:85: [DEBUG] [scheduler] job agent-channel-health: execution starting
2026/11/11 13:54:05 client.go:1288: [DEBUG] [agentapi] GET /storage -> 502 (4ms)
2026/11/11 13:54:05 scheduler.go:467: [INFO] [scheduler] Job agent-channel-health completed (took 5ms)
2026/11/11 13:54:06 client.go:1288: [DEBUG] [agentapi] GET /disks -> 502 (6ms)
2026/11/11 13:54:06 client.go:1288: [DEBUG] [agentapi] GET /disks -> 502 (5ms)
@@ -0,0 +1,9 @@
36
Forbidden: missing scope hosts:read
pin 0
TLS 0
certificate 0
proxmox: 0
unavailable 0
degraded 0
Storage 0
@@ -0,0 +1,14 @@
2026-11-11T13:54:34Z
== hub log lines naming demo-hp since 13:45Z (count, then last 15, cut)
9
2026/11/11 14:43:30 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 22057 bytes)
2026/11/11 14:43:30 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1)
2026/11/11 14:47:07 [INFO] DR-recipe app-half stored for customer demo-hp (v1)
2026/11/11 14:47:07 [INFO] Received report from demo-hp (10182 bytes)
2026/11/11 14:48:15 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 22040 bytes)
2026/11/11 14:48:15 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1)
2026/11/11 14:52:05 [INFO] Event from demo-hp: agent_channel_unknown (warning) — Controller→agent channel down: agentapi: GET /storage: HTTP 502
2026/11/11 14:52:05 [INFO] Operator email sent for demo-hp/agent_channel_unknown
2026/11/11 14:54:27 [WARN] access token "cc-releases-2" refused: missing scope hosts:read — GET /hosts/demo-hp-bb76ea → 403
== any hub line with pin/tls/certificate
0
@@ -0,0 +1,21 @@
2026-11-11T13:59:16Z
== hub log lines naming demo-hp, last 12 min
2026/11/11 14:48:15 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 22040 bytes)
2026/11/11 14:48:15 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1)
2026/11/11 14:52:05 [INFO] Event from demo-hp: agent_channel_unknown (warning) — Controller→agent channel down: agentapi: GET /storage: HTTP 502
2026/11/11 14:52:05 [INFO] Operator email sent for demo-hp/agent_channel_unknown
HUB LINE: demo-hp-bb76ea Demo HP 0.173.0 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 73% 45% running (connected) 59% local-lvm tester-1-d70be4 Tester
== agent lines about the hub report since 14:50 CET
26 msg="reconcile: storage view unavailable — using mount-table only"
26 msg="reconcile: enrolled drive bound under shared parent (live, no reboot)"
17 msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)"
9 msg="local-api: /storage guest config"
8 msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)"
8 msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)"
4 msg="local-api: storage view unavailable for the tier presence report"
4 msg="local-api: storage view unavailable for the backup-target presence check — assuming present"
4 msg="local-api: could not read the backup storage for the due-check — falling back to the in-memory record"
2 msg="local-api: backup storage unreadable — due-check uses the last success saved on disk (R-894)"
1 msg="stale-lock: guest list unavailable — skipping recovery"
1 msg="reconcile: pass failed"
1 msg="hub: collect failed; skipping this cycle's report"
@@ -0,0 +1,3 @@
time=2026-11-11T14:58:18.520+01:00 level=ERROR msg="reconcile: pass failed" err="reconcile: actual state: proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:58:19.394+01:00 level=WARN msg="stale-lock: guest list unavailable — skipping recovery" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:58:24.067+01:00 level=WARN msg="hub: collect failed; skipping this cycle's report" err="hub: NodeStatus failed (no useful report): proxmox: GET /nodes/demo-hp/status: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/status\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
@@ -0,0 +1,7 @@
== repair start 2026-11-11T13:59:25Z
== setpin fp start 2026-11-11T13:59:25Z
written tls: {'fingerprint': '21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12', 'insecure_skip_verify': False}
first agent call after restart: [11/11/2026:14:59:27 "GET /api2/json/pools/felhom 200
seconds from start to a working call: 2.149922736 (loops: 2; 60 = no call seen)
agent: active
== repair end 2026-11-11T13:59:28Z
@@ -0,0 +1,22 @@
== state 2026-11-11T13:59:29Z
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Nov 10 13:49:24 2026 GMT
notAfter=Nov 9 13:49:24 2028 GMT
sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12
agent tls config: {'fingerprint': '21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12', 'insecure_skip_verify': False}
agent: active since Wed 2026-11-11 14:59:25 CET restarts=0
-- agent calls in pveproxy's access log, last 4 (time, call, answer code)
[11/11/2026:14:59:27 +0100] "GET /api2/json/storage 200
[11/11/2026:14:59:27 +0100] "GET /api2/json/storage 200
[11/11/2026:14:59:28 +0100] "GET /api2/json/nodes/demo-hp/lxc 200
[11/11/2026:14:59:28 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6)
33
time=2026-11-11T14:58:59.229+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:05.502+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.213+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.232+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.394+01:00 level=WARN msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.394+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
-- guests
@@ -0,0 +1,15 @@
== CA test start 2026-11-11T13:59:39Z
-rw-r----- 1 root felhom-agent 2094 Nov 11 14:59 /etc/felhom-agent/pve-root-ca.pem
-rw-r----- 1 root www-data 2094 Aug 21 17:44 /etc/pve/pve-root-ca.pem
agent user can read the copy
agent user CANNOT read the original
subject=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Aug 20 15:44:42 2026 GMT
notAfter=Aug 17 15:44:42 2036 GMT
sha256 Fingerprint=E2:2E:E9:F0:E3:3F:76:4A:11:38:87:E8:EF:67:24:C5:83:3A:9F:68:C1:20:BD:64:F4:40:54:93:22:29:DC:F7
== setpin ca start 2026-11-11T13:59:39Z
written tls: {'ca_file': '/etc/felhom-agent/pve-root-ca.pem', 'fingerprint': '', 'insecure_skip_verify': False}
first agent call after restart: [11/11/2026:14:59:41 "GET /api2/json/storage 200
seconds from start to a working call: 2.160335125 (loops: 2; 60 = no call seen)
agent: active
@@ -0,0 +1,26 @@
== state 2026-11-11T13:59:43Z
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Nov 10 13:49:24 2026 GMT
notAfter=Nov 9 13:49:24 2028 GMT
sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12
agent tls config: {'ca_file': '/etc/felhom-agent/pve-root-ca.pem', 'fingerprint': '', 'insecure_skip_verify': False}
agent: active since Wed 2026-11-11 14:59:39 CET restarts=0
-- agent calls in pveproxy's access log, last 4 (time, call, answer code)
[11/11/2026:14:59:41 +0100] "GET /api2/json/storage 200
[11/11/2026:14:59:41 +0100] "GET /api2/json/storage 200
[11/11/2026:14:59:41 +0100] "GET /api2/json/nodes/demo-hp/lxc 200
[11/11/2026:14:59:41 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6)
32
time=2026-11-11T14:58:59.229+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:05.502+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.213+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.232+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.394+01:00 level=WARN msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.394+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
-- guests
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
341 night1004-tester1 running
@@ -0,0 +1,43 @@
== force 2 start 2026-11-11T13:59:54Z
== force 2026-11-11T13:59:54Z
before: sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12
(re)generate node files
generate new node certificate
merge authorized SSH keys
updatecerts rc=0
file now: sha256 Fingerprint=1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74 notAfter=Nov 9 13:59:54 2028 GMT
served 5 s later, no restart: sha256 Fingerprint=21:D6:B5:E0:12:4C:67:04:DB:BB:98:32:95:FA:C6:B4:0A:3A:44:61:D2:BE:D0:8C:B7:EB:7B:9F:6E:25:84:12
pveproxy: Tue 2026-11-10 15:43:04 CET reloads:
done 2026-11-11T13:59:59Z
reload rc=0 at 2026-11-11T14:00:01Z
== state 2026-11-11T14:00:46Z
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Nov 10 13:59:54 2026 GMT
notAfter=Nov 9 13:59:54 2028 GMT
sha256 Fingerprint=1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74
agent tls config: {'ca_file': '/etc/felhom-agent/pve-root-ca.pem', 'fingerprint': '', 'insecure_skip_verify': False}
agent: active since Wed 2026-11-11 14:59:39 CET restarts=0
-- agent calls in pveproxy's access log, last 4 (time, call, answer code)
[11/11/2026:15:00:41 +0100] "GET /api2/json/nodes/demo-hp/lxc 200
[11/11/2026:15:00:41 +0100] "GET /api2/json/pools/felhom 200
[11/11/2026:15:00:41 +0100] "GET /api2/json/pools/felhom 200
[11/11/2026:15:00:42 +0100] "GET /api2/json/nodes/demo-hp/storage 200
-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6)
18
time=2026-11-11T14:58:59.229+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:05.502+01:00 level=ERROR msg="local-api: /storage guest config" vmid=9201 err="proxmox: GET /nodes/demo-hp/lxc/9201/config: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc/9201/config\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.213+01:00 level=WARN msg="guestnet: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.232+01:00 level=WARN msg="reconcile: storage view unavailable — using mount-table only" err="storage: ListStorage: proxmox: GET /storage: Get \"https://127.0.0.1:8006/api2/json/storage\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.394+01:00 level=WARN msg="guest-power: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
time=2026-11-11T14:59:19.394+01:00 level=WARN msg="controller-supervisor: guest list unavailable — skipping sweep (ownership unproven)" err="proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": proxmox: TLS pin mismatch: server cert sha256 does not match configured fingerprint"
-- guests
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
341 night1004-tester1 running
== agent start time and restarts (must be unchanged: 14:59:39)
NRestarts=0
ActiveEnterTimestamp=Wed 2026-11-11 14:59:39 CET
== agent lines with x509/pin/tls since 14:59:45
0
@@ -0,0 +1,13 @@
== negative control: a CA file that did NOT sign the certificate. Start 2026-11-11T14:01:01Z
== setpin ca start 2026-11-11T14:01:01Z
written tls: {'ca_file': '/etc/ssl/certs/ISRG_Root_X1.pem', 'fingerprint': '', 'insecure_skip_verify': False}
first agent call after restart: [11/11/2026:15:01:02 "GET /api2/json/nodes/demo-hp/storage 500
seconds from start to a working call: .112020654 (loops: 0; 60 = no call seen)
agent: active
x509 lines: 0
== back to the standard: fingerprint pin. 2026-11-11T14:01:02Z
== setpin fp start 2026-11-11T14:01:02Z
written tls: {'fingerprint': '1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74', 'insecure_skip_verify': False}
first agent call after restart: [11/11/2026:15:01:02 "GET /api2/json/nodes/demo-hp/storage 500
seconds from start to a working call: .718520877 (loops: 0; 60 = no call seen)
agent: active
@@ -0,0 +1,36 @@
== the first control was void: the waiter matched a call of the same second; the agent ran under 1 s on the wrong file. Second try.
== state now 2026-11-11T14:01:14Z
[11/11/2026:15:01:07 "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
[11/11/2026:15:01:08 "GET /api2/json/nodes/demo-hp/storage 200
[11/11/2026:15:01:08 "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
::ffff:127.0.0.1 - felhom-agent@pve!agent [11/11/2026:15:01:02 +0100] "GET /api2/json/nodes/demo-hp/storage HTTP/1.1" 500 -
== wrong CA, 30 s
Job for felhom-agent.service failed because start of the service was attempted too often.
See "systemctl status felhom-agent.service" and "journalctl -xeu felhom-agent.service" for details.
To force a start use "systemctl reset-failed felhom-agent.service"
followed by "systemctl start felhom-agent.service" again.
== setpin ca start 2026-11-11T14:01:14Z
written tls: {'ca_file': '/etc/ssl/certs/ISRG_Root_X1.pem', 'fingerprint': '', 'insecure_skip_verify': False}
seconds from start to a working call: 61.364869823 (loops: 60; 60 = no call seen)
x509 lines in 32 s: 0
agent calls answered 200 in the last 25 s: 0
== back to the standard 2026-11-11T14:02:46Z
== setpin fp start 2026-11-11T14:02:46Z
written tls: {'fingerprint': '1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74', 'insecure_skip_verify': False}
first agent call after restart: [11/11/2026:15:02:48 "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
seconds from start to a working call: 2.148034679 (loops: 2; 60 = no call seen)
agent: active
== state 2026-11-11T14:03:09Z
subject=OU=PVE Cluster Node, O=Proxmox Virtual Environment, CN=demo-hp.local
issuer=CN=Proxmox Virtual Environment, OU=bd229733-676d-4c57-a395-a33de3263545, O=PVE Cluster Manager CA
notBefore=Nov 10 13:59:54 2026 GMT
notAfter=Nov 9 13:59:54 2028 GMT
sha256 Fingerprint=1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74
agent tls config: {'fingerprint': '1F:75:D3:51:DA:ED:2C:00:7C:F3:DF:E0:6A:58:1B:CE:95:DB:E1:55:D1:40:72:F2:09:AF:CA:DC:9D:D1:0E:74', 'insecure_skip_verify': False}
agent: active since Wed 2026-11-11 15:02:46 CET restarts=0
-- agent calls in pveproxy's access log, last 4 (time, call, answer code)
[11/11/2026:15:03:08 +0100] "GET /api2/json/nodes/demo-hp/storage 200
[11/11/2026:15:03:08 +0100] "GET /api2/json/nodes/demo-hp/lxc/9201/config 200
[11/11/2026:15:03:08 +0100] "GET /api2/json/storage 200
[11/11/2026:15:03:09 +0100] "GET /api2/json/nodes/demo-hp/storage 200
-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6)
@@ -0,0 +1,50 @@
2026-11-11T14:03:22Z
Restart=on-failure
RestartUSec=5s
Result=success
NRestarts=0
ActiveState=active
SubState=running
StartLimitIntervalUSec=2min
StartLimitBurst=4
2026-11-11T15:01:02+01:00 demo-hp systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)...
2026-11-11T15:01:02+01:00 demo-hp systemd[1]: felhom-agent.service: Deactivated successfully.
2026-11-11T15:01:02+01:00 demo-hp systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-11T15:01:02+01:00 demo-hp systemd[1]: felhom-agent.service: Consumed 28.460s CPU time, 348.1M memory peak.
2026-11-11T15:01:02+01:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-11T15:01:02+01:00 demo-hp systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)...
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.135+01:00 level=WARN msg="hub: collect failed; skipping this cycle's report" err="hub: NodeStatus failed (no useful rep
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="reconcile: pass failed" err="reconcile: actual state: proxmox: GET /nodes/demo-hp/lxc: Get \
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.136+01:00 level=WARN msg="reconcile: shared parent ensure failed — binds may not propagate live" err="shared-parent:
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.138+01:00 level=ERROR msg="reconcile: AttachDrive failed" vmid=9201 where=/mnt/hdd_1 err="guest-attach: namespace /mnt/
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.139+01:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.139+01:00 level=WARN msg="selfheal: node check FAILED" heal=networking detail="networking.service active=false, default
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.139+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_windo
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.139+01:00 level=ERROR msg="node self-heal: networking start attempt failed" attempt=1 err="context canceled: "
2026-11-11T15:01:03+01:00 demo-hp felhom-agent[527553]: time=2026-11-11T15:01:03.139+01:00 level=WARN msg="lanresolver: EnsureDnsmasq failed — will retry" err="enable dnsmasq: context canceled"
2026-11-11T15:01:03+01:00 demo-hp systemd[1]: felhom-agent.service: Deactivated successfully.
2026-11-11T15:01:03+01:00 demo-hp systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-11T15:01:03+01:00 demo-hp systemd[1]: felhom-agent.service: Consumed 1.212s CPU time, 23.5M memory peak.
2026-11-11T15:01:03+01:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-11T15:01:05+01:00 demo-hp felhom-agent[527856]: time=2026-11-11T15:01:05.230+01:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3
2026-11-11T15:01:05+01:00 demo-hp felhom-agent[527856]: time=2026-11-11T15:01:05.230+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_windo
2026-11-11T15:01:14+01:00 demo-hp felhom-agent[527856]: time=2026-11-11T15:01:14.815+01:00 level=WARN msg="pbs: verify task wait failed; reporting current snapshot state" datastore=felhom-offsite err=
2026-11-11T15:01:14+01:00 demo-hp felhom-agent[527856]: time=2026-11-11T15:01:14.815+01:00 level=WARN msg="pbs: snapshot list failed" datastore=felhom-offsite err="pbs: GET /admin/datastore/felhom-off
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: Stopping felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog)...
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent.service: Deactivated successfully.
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: Stopped felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent.service: Consumed 15.975s CPU time, 327.3M memory peak.
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent.service: Start request repeated too quickly.
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent.service: Failed with result 'start-limit-hit'.
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: Failed to start felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent.service: Triggering OnFailure= dependencies.
2026-11-11T15:02:46+01:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
2026-11-11T15:02:49+01:00 demo-hp felhom-agent[535281]: time=2026-11-11T15:02:49.047+01:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_windo
2026-11-11T15:02:49+01:00 demo-hp felhom-agent[535281]: time=2026-11-11T15:02:49.047+01:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3
== tests in the agent repo for the CA path
5:func TestNormalizeFingerprint(t *testing.T) {
20:func TestNormalizeFingerprint_Bad(t *testing.T) {
28:func TestTLSConfig_Build(t *testing.T) {
@@ -0,0 +1,26 @@
2026-11-11T14:03:37Z
== the 1-second run on the WRONG CA file (pid 527553): its proxmox errors
time=2026-11-11T15:01:03.135+01:00 level=WARN msg="hub: collect failed; skipping this cycle's report" err="hub: NodeStatus failed (no useful report): proxmox: GET /nodes/demo-hp/status: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/status\": terminated signal received"
time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="reconcile: pass failed" err="reconcile: actual state: proxmox: GET /nodes/demo-hp/lxc: Get \"https://127.0.0.1:8006/api2/json/nodes/demo-hp/lxc\": terminated signal received"
== x509 count for that pid: 3
== OnFailure unit
OnFailure=felhom-agent-rollback.service
Result=success
ExecMainStartTimestamp=Wed 2026-11-11 15:01:14 CET
ActiveState=inactive
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: Starting felhom-agent-rollback.service - Felhom agent self-update auto-rollback (pending-guarded; fires per crash, no-ops without a pending update)...
2026-11-11T15:01:14+01:00 demo-hp felhom-selfupdate-guarded[529488]: felhom-selfupdate-guarded: rollback: no pending update — no-op
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: felhom-agent-rollback.service: Deactivated successfully.
2026-11-11T15:01:14+01:00 demo-hp systemd[1]: Finished felhom-agent-rollback.service - Felhom agent self-update auto-rollback (pending-guarded; fires per crash, no-ops without a pending update).
== now: capability / attach lines since 15:02:46
time=2026-11-11T15:02:48.053+01:00 level=INFO msg="capabilities self-check" ok=77 total=77 degraded=0 inactive=0
time=2026-11-11T15:02:48.851+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
time=2026-11-11T15:03:09.617+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
time=2026-11-11T15:03:29.562+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/felhom-drives/hdd_1 durable_id=uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae
(end)
== drive in guest 9201
/mnt/felhom-drives/hdd_1 /dev/nvme0n1[/felhom-data] ext4
/mnt/felhom-drives/hdd_1 /dev/nvme0n1[/felhom-data] ext4
== config: same as the backup but for tls?
rest equal: True | tls keys now: ['fingerprint', 'insecure_skip_verify'] | before: ['fingerprint', 'insecure_skip_verify']
-rw------- felhom-agent felhom-agent
@@ -0,0 +1,17 @@
2026-11-11T14:03:44Z
time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=pve:pool-read feature="stale-lock recovery scoping (pool ownership check)" reason="proxmox: GET /pools/felhom: Get \"https://127.0.0.1:8006/api2/json/pools/felhom\": tls: failed to verify certificate: x509: certificate signed by unknown authority" critical=false
time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=pve:store-grant:local feature="backup tier local readable by the agent (archive listing, restore-test candidacy)" reason="could not read own permissions: proxmox: GET /access/permissions?path=%2Fstorage%2Flocal: Get \"https://127.0.0.1:8006/api2/json/access/permissions?path=%2Fstorage%
time=2026-11-11T15:01:03.135+01:00 level=ERROR msg="capability DEGRADED — privileged grant missing (feature impaired until fixed)" capability=pve:store-grant:felhom-pbs feature="backup tier felhom-pbs readable by the agent (archive listing, restore-test candidacy)" reason="could not read own permissions: proxmox: GET /access/permissions?path=%2Fstorage%2Ffelhom-pbs: Get \"https://127.0.0.1:8006/api2/json/access/permissions?p
== hub: demo-hp lines since the repair
2026/11/11 14:59:32 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 21349 bytes)
2026/11/11 14:59:32 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1)
2026/11/11 14:59:47 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 21349 bytes)
2026/11/11 14:59:47 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1)
2026/11/11 15:00:06 [INFO] Event from demo-hp: agent_channel_recovered (info) — Controller→agent channel recovered — local-API reachable again.
2026/11/11 15:01:10 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 21351 bytes)
2026/11/11 15:01:10 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1)
2026/11/11 15:02:07 [INFO] DR-recipe app-half stored for customer demo-hp (v1)
2026/11/11 15:02:07 [INFO] Received report from demo-hp (14341 bytes)
2026/11/11 15:02:07 [INFO] managed floor SERVED for demo-hp: floor 0.352.0, agent requirement "0.158.1" from manifest (golden 0.352.0)
2026/11/11 15:02:53 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 2 backups, 2 restore-tests, 2 pbs-snapshots, 21349 bytes)
2026/11/11 15:02:53 [INFO] DR-recipe host-half stored for customer demo-hp (host demo-hp-bb76ea, v1)
@@ -0,0 +1,18 @@
2026-11-11T14:03:55Z
0
(0 = both files gone)
active
active
NRestarts=0
ActiveEnterTimestamp=Wed 2026-11-11 15:02:46 CET
[11/11/2026:15:03:49 "GET /api2/json/pools/felhom 200
[11/11/2026:15:03:49 "GET /api2/json/nodes/demo-hp/storage 200
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
341 night1004-tester1 running
26
Host key verification failed.
active
9201 running
2026-11-11T14:04:07Z
@@ -0,0 +1,41 @@
main() {
echo "== $(hostname) $(date -u +%FT%TZ) pve $(pveversion | head -1)"
echo "== served certificate (network read, 127.0.0.1:8006)"
echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -subject -issuer -dates -fingerprint -sha256 -ext subjectAltName 2>/dev/null
echo "== files"
ls -la --time-style=full-iso /etc/pve/local/ /etc/pve/pve-root-ca.pem /etc/pve/priv/pve-root-ca.key 2>&1 | cut -c1-140
for f in /etc/pve/local/pve-ssl.pem /etc/pve/pve-root-ca.pem; do echo "-- $f"; openssl x509 -in $f -noout -subject -issuer -dates -fingerprint -sha256; done
ls /etc/pve/local/pveproxy-ssl.pem 2>&1
echo "== agent"
/usr/local/bin/felhom-agent --version; systemctl is-active felhom-agent; systemctl show felhom-agent -p ActiveEnterTimestamp -p NRestarts -p Restart
python3 - <<'PY'
import json
c=json.load(open('/etc/felhom-agent/agent.json'))
p=c.get('proxmox',{})
print('proxmox keys:',sorted(p.keys()))
for k,v in p.items():
if k=='tls': print('tls:',v)
elif 'secret' in k.lower() or 'token' in k.lower(): print(k,': <present>' if v else ': <empty>')
else: print(k,':',v)
PY
ls -la --time-style=full-iso /etc/felhom-agent/ | cut -c1-120
systemctl cat felhom-agent | grep -i -n 'Environment\|ExecStart\|User='
echo "== agent log, last proxmox lines"
journalctl -u felhom-agent --since '-20 min' --no-pager -o cat | grep -i -c .
journalctl -u felhom-agent --since '-20 min' --no-pager -o cat | cut -c1-230 | tail -8
echo "== who calls port 8006 (pveproxy access log, client addresses, last 2 days)"
for f in /var/log/pveproxy/access.log /var/log/pveproxy/access.log.1; do [ -f $f ] && awk '{print $1}' $f; done | sort | uniq -c | sort -rn | head
echo "-- users seen"
for f in /var/log/pveproxy/access.log /var/log/pveproxy/access.log.1; do [ -f $f ] && awk '{print $1, $3}' $f; done | sort | uniq -c | sort -rn | head
echo "== listening / other TLS users of this cert"
ss -ltnp | grep -E ':8006|:3128|:8007|:85 ' | cut -c1-150
echo "== in flight"
pvesh get /cluster/tasks --output-format json 2>/dev/null | python3 -c "import json,sys; t=json.load(sys.stdin); r=[x for x in t if not x.get('endtime')]; print('running tasks:',[(x['type'],x.get('id')) for x in r])"
qm list; pct list
echo "== tools' own help"
pvecm help updatecerts 2>&1 | head -20
pvenode help cert 2>&1 | head -30
echo "== renew rule in code"
grep -n -i 'expir\|renew\|check_expiry\|2 \* 365\|730\|days' /usr/share/perl5/PVE/Cluster/Setup.pm | head -30
grep -rn -i 'renew\|check_expiry\|updatecerts' /usr/share/perl5/PVE/Service/pveproxy.pm /usr/bin/pve-daily-update /usr/share/perl5/PVE/CLI/pveupdate.pm /usr/bin/pveupdate 2>/dev/null | head -20
}; main </dev/null
@@ -0,0 +1,11 @@
# force a new node certificate the documented way: pvecm updatecerts --force
main() {
echo "== force $(date -u +%FT%TZ)"
echo "before: $(echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256)"
pvecm updatecerts --force; echo "updatecerts rc=$?"
echo "file now: $(openssl x509 -in /etc/pve/local/pve-ssl.pem -noout -fingerprint -sha256 -enddate | tr '\n' ' ')"
sleep 5
echo "served 5 s later, no restart: $(echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256)"
echo "pveproxy: $(systemctl show pveproxy -p ActiveEnterTimestamp --value) reloads: $(journalctl -u pveproxy --since '-1 min' --no-pager -o cat | tail -3 | tr '\n' ';')"
echo "done $(date -u +%FT%TZ)"
}; main </dev/null
@@ -0,0 +1,19 @@
# usage: setpin.sh fp | setpin.sh ca — writes the agent's proxmox.tls block, restarts the agent
main() {
mode=$1; t0=$(date +%s.%N); echo "== setpin $mode start $(date -u +%FT%TZ)"
FP=$(echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256 | cut -d= -f2)
MODE=$mode FP=$FP python3 - <<'PY'
import json,os
f='/etc/felhom-agent/agent.json'; c=json.load(open(f)); st=os.stat(f)
if os.environ['MODE']=='fp': c['proxmox']['tls']={'fingerprint':os.environ['FP'],'insecure_skip_verify':False}
else: c['proxmox']['tls']={'ca_file':'/etc/ssl/certs/ISRG_Root_X1.pem','fingerprint':'','insecure_skip_verify':False}
tmp=f+'.tmp'; fd=os.open(tmp,os.O_WRONLY|os.O_CREAT|os.O_TRUNC,0o600)
with os.fdopen(fd,'w') as o: json.dump(c,o,indent=2); o.write('\n')
os.chown(tmp,st.st_uid,st.st_gid); os.rename(tmp,f); print('written tls:',c['proxmox']['tls'])
PY
systemctl restart felhom-agent
n=0; mark=$(date '+%d/%m/%Y:%H:%M:%S'); until [ $n -ge 60 ]; do l=$(grep 'felhom-agent@pve' /var/log/pveproxy/access.log | tail -1); ts=$(echo "$l" | awk '{print $4}' | tr -d '[');
if [ "$(date -d "$(echo $ts | sed 's#\([0-9]*\)/\([0-9]*\)/\([0-9]*\):#\3-\2-\1 #')" +%s 2>/dev/null || echo 0)" -ge "${t0%.*}" ]; then echo "first agent call after restart: $(echo "$l" | awk '{print $4,$6,$7,$9}')"; break; fi; n=$((n+1)); sleep 1; done
t1=$(date +%s.%N); echo "seconds from start to a working call: $(echo "$t1 - $t0" | bc) (loops: $n; 60 = no call seen)"
echo "agent: $(systemctl is-active felhom-agent)"
}; main "$@" </dev/null
@@ -0,0 +1,19 @@
# usage: setpin.sh fp | setpin.sh ca — writes the agent's proxmox.tls block, restarts the agent
main() {
mode=$1; t0=$(date +%s.%N); echo "== setpin $mode start $(date -u +%FT%TZ)"
FP=$(echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256 | cut -d= -f2)
MODE=$mode FP=$FP python3 - <<'PY'
import json,os
f='/etc/felhom-agent/agent.json'; c=json.load(open(f)); st=os.stat(f)
if os.environ['MODE']=='fp': c['proxmox']['tls']={'fingerprint':os.environ['FP'],'insecure_skip_verify':False}
else: c['proxmox']['tls']={'ca_file':'/etc/felhom-agent/pve-root-ca.pem','fingerprint':'','insecure_skip_verify':False}
tmp=f+'.tmp'; fd=os.open(tmp,os.O_WRONLY|os.O_CREAT|os.O_TRUNC,0o600)
with os.fdopen(fd,'w') as o: json.dump(c,o,indent=2); o.write('\n')
os.chown(tmp,st.st_uid,st.st_gid); os.rename(tmp,f); print('written tls:',c['proxmox']['tls'])
PY
systemctl restart felhom-agent
n=0; mark=$(date '+%d/%m/%Y:%H:%M:%S'); until [ $n -ge 60 ]; do l=$(grep 'felhom-agent@pve' /var/log/pveproxy/access.log | tail -1); ts=$(echo "$l" | awk '{print $4}' | tr -d '[');
if [ "$(date -d "$(echo $ts | sed 's#\([0-9]*\)/\([0-9]*\)/\([0-9]*\):#\3-\2-\1 #')" +%s 2>/dev/null || echo 0)" -ge "${t0%.*}" ]; then echo "first agent call after restart: $(echo "$l" | awk '{print $4,$6,$7,$9}')"; break; fi; n=$((n+1)); sleep 1; done
t1=$(date +%s.%N); echo "seconds from start to a working call: $(echo "$t1 - $t0" | bc) (loops: $n; 60 = no call seen)"
echo "agent: $(systemctl is-active felhom-agent)"
}; main "$@" </dev/null
@@ -0,0 +1,15 @@
# read-only state: served cert, agent pin, agent calls in the access log, agent log lines
main() {
echo "== state $(date -u +%FT%TZ)"
echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -subject -issuer -dates -fingerprint -sha256 2>/dev/null
python3 -c "
import json;c=json.load(open('/etc/felhom-agent/agent.json'));print('agent tls config:',c['proxmox']['tls'])"
echo "agent: $(systemctl is-active felhom-agent) since $(systemctl show felhom-agent -p ActiveEnterTimestamp --value) restarts=$(systemctl show felhom-agent -p NRestarts --value)"
echo "-- agent calls in pveproxy's access log, last 4 (time, call, answer code)"
grep 'felhom-agent@pve' /var/log/pveproxy/access.log | tail -4 | awk '{print $4,$5,$6,$7,$9}'
echo "-- agent log, last 3 min: lines with tls/x509/pin/proxmox error (count, then the last 6)"
journalctl -u felhom-agent --since '-3 min' --no-pager -o cat | grep -v 'pam_unix\|COMMAND=' | grep -i -c 'pin mismatch\|x509\|tls'
journalctl -u felhom-agent --since '-3 min' --no-pager -o cat | grep -v 'pam_unix\|COMMAND=' | grep -i 'pin mismatch\|x509\|tls' | cut -c1-330 | tail -6
echo "-- guests"
pct list | awk 'NR>1{print $1,$2,$NF}'; qm list | awk 'NR>1{print $1,$2,$3}'
}; main </dev/null
@@ -0,0 +1,77 @@
# Why the N100 keeps no time — 2026-11-11
Read only. Nothing on the N100, on Tester 1 or on demo-hp was changed. Files: `evidence/`, `tools/` (the read scripts).
Documents read: `architecture/11-os-updates.md`, `architecture/03-host-agent.md`, the Clock list
(`audits/clock-list-2026-11-11/`, entries CA03, CA04, CA09), rows R-1301, R-1384, R-1206.
## The cause, in three lines
1. The time service (`chrony`) tells systemd "I am ready" through the socket `/run/systemd/notify`.
2. On the N100 the kernel **refuses that one write**, because the security profile (AppArmor) it loaded for chrony is
an **old copy from a cache** — the copy made for the older chrony, which did not talk to that socket.
3. chrony treats the refused write as fatal and stops. So the unit fails 2 seconds into every start.
## The evidence (each line is a read, not a theory)
| What | N100 | demo-hp | Tester 1 host | File |
|---|---|---|---|---|
| chrony package | `4.8-4~bpo13+2` | the same | `4.6.1-3+deb13u1` (old) | `*-stale-proof.txt` |
| unit, drop-ins, `/etc/default/chrony`, config, profile file on disk | identical (profile sha256 `214e509a…`) | identical | not compared | `n100-host.txt`, `demo-hp-host.txt` |
| unit state | **failed**, 8 fatal lines = 4 starts | active, 0 fatal lines | active, 0 | `*-host-2.txt` |
| profile the kernel has loaded (sha256) | **`dbf67f16…`** | `83417e83…` | `dbf67f16…` | `*-stale-proof.txt` |
| a fresh compile of the profile file on disk | `83417e83…` | `83417e83…` | `dbf67f16…` | the same |
| loaded = fresh compile? | **no** | yes | yes | the same |
| the kernel's own line | `apparmor="DENIED" operation="sendmsg" profile="/usr/sbin/chronyd" name="/run/systemd/notify" … denied_mask="w"` (2026-10-16 05:09:25) | none | none | `n100-host-2.txt:13` |
`dbf67f16…` is the profile of chrony 4.6.1 (Tester 1, which still runs 4.6.1, compiles exactly that). `83417e83…` is
the profile of chrony 4.8. **The N100 runs chrony 4.8 under the profile of chrony 4.6.1.**
chrony's source agrees with the log: `main.c` `notify_system_manager()` ends in
`LOG_FATAL("Could not send notification to $NOTIFY_SOCKET")` on any failed send, and sends once at start and once at
exit. That is the same line twice for each start.
## How the old copy got there (dates from the box)
AppArmor keeps one cache folder for each kernel feature set. Both boxes have two: `ae1a4050.0` (kernel 7.0.2) and
`f6da0254.0` (the 7.0.14 kernels).
| When | N100 | demo-hp |
|---|---|---|
| the 7.0.14 folder was made, with the old chrony profile in it | **2026-10-07 12:11** (a kernel test of that day booted the new kernel once, then went back) | 2026-10-04 09:44 |
| chrony went to 4.8 (apt, requested by `felhom-agent`, the OS-update leg) | 2026-10-07 14:15, **on the old kernel** → the new profile went into the 7.0.2 folder only | 2026-10-10 04:40, on a 7.0.14 kernel → the new profile went into the 7.0.14 folder |
| next start on a 7.0.14 kernel | the 7.0.14 folder still holds the old profile; it is loaded | the new profile is loaded |
Why AppArmor did not see that the copy was old: the cached file carries the time 2026-07-18 18:15:53; the new profile
file from the package carries 2026-07-01 22:28:16. The cache looks newer than its source, so it is used. (The time
rule is AppArmor's documented cache check; this run did not read its source.)
A control on the method: the same check names `lxc-containers` as "differs" on all three boxes, the healthy ones too.
That one is not a signal. `usr.sbin.chronyd` differs on the N100 only.
## The three questions
1. **Is the cause the product's?** In part. The product did not write the profile or the unit. But the product's
OS-update leg installed the new chrony, and the bad order is: *start a new kernel once → go back to the old kernel →
a package with a changed profile is updated → start the new kernel again.* On the N100 a test made that order. On a
household box the **kernel rollback** (the crash guard goes back to the last good kernel) makes the same order. It
is true for every AppArmor profile, not only chrony's.
2. **When did it start?** The oldest failed start in the journal is 2026-10-16 05:09:25. The cause dates from
2026-10-07 14:15. The journal has no start between those two dates.
3. **The smallest repair.** One hand step on the N100 (in `STATUS.md`, not done, **not tested on a box**):
`apparmor_parser -r -T -W /etc/apparmor.d/usr.sbin.chronyd && systemctl restart chrony && chronyc tracking`.
For a household box no path delivers it today. Proposed, not built: after an OS-update leg `felhom-os-apply`
reloads the profiles with the cache ignored (`-T -W`), and the kernel rollback does the same. Filed as a row.
**The fresh box of Part 1 cannot show this.** It has one cache folder and no history. It showed only that a new
install keeps time (chrony active, 0.000001 s off, before the Felhom installer ran).
## What a wrong clock breaks (numbers from source)
- **Signed jobs** (the only way an agent update arrives): valid from issue minus 2 minutes (`DefaultClockSkew`,
`felhom-agent/internal/authz/verifier.go:21`) to expiry; a window over 24 hours is refused (`MaxWindow`, `:26`). A
host clock **more than 120 s behind** refuses every job as "not yet valid" (`verifier.go:153-162`).
- The N100 was 32.6 s behind this morning, after 26 days with no time service: about 1.25 s a day. At that rate it
passes 120 s in **about 70 days (mid-January 2027)**. This is an estimate from one measurement.
- The guest has no clock of its own; it reads the host's. So the backup age, the off-site age and the proof age that
the hub trusts, the night slots and the crash guard all read this clock (Clock list, class W2, 30 cells, R-1384).
- Who notices today: nobody. The hub alarms only on a tunnel handshake from the future.
@@ -0,0 +1,77 @@
#### demo-hp 2026-11-11T13:16:04Z
## fresh compile of the on-disk profile, to stdout only (no load, no cache write)
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -
apparmor_parser: Too many actions given on the command line.
## loaded raw policy sha256
83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1
## cache dirs
total 16
drwxr-xr-x 4 root root 4096 2026-10-04 09:44:36.114533363 +0200 .
drwxr-xr-x 13 root root 4096 2026-08-21 17:42:55.201857862 +0200 ..
drwx------ 2 root root 4096 2026-08-21 17:44:37.686947147 +0200 ae1a4050.0
drwx------ 2 root root 4096 2026-11-08 02:48:51.017396841 +0100 f6da0254.0
/var/cache/apparmor/ae1a4050.0/ features-sha=0e6b37757d22af1e files=106
-rw-r--r-- 1 root root 3177 2026-08-21 17:44:37.159696460 +0200 /var/cache/apparmor/ae1a4050.0//.features
-rw------- 1 root root 295529 2026-08-21 17:42:38.360866076 +0200 /var/cache/apparmor/ae1a4050.0//usr.sbin.chronyd
/var/cache/apparmor/f6da0254.0/ features-sha=611413ba507a5c19 files=106
-rw-r--r-- 1 root root 3219 2026-10-04 09:44:36.115533363 +0200 /var/cache/apparmor/f6da0254.0//.features
-rw------- 1 root root 296497 2026-08-21 17:42:38.360866076 +0200 /var/cache/apparmor/f6da0254.0//usr.sbin.chronyd
## kernel features now
yes yes yes yes yes
611413ba507a5c19
## newest source file the profile depends on
2025-07-21 12:21:49.000000000 /etc/apparmor.d/abi/kernel-5.4-vanilla
2025-07-21 12:21:49.000000000 /etc/apparmor.d/abstractions/base
2025-07-21 12:21:49.000000000 /etc/apparmor.d/abstractions/nameservice
2025-07-21 12:21:49.000000000 /etc/apparmor.d/tunables/global
2026-05-21 13:32:03.000000000 /etc/apparmor.d/local/usr.sbin.chronyd
2026-07-01 22:28:16.000000000 /etc/apparmor.d/usr.sbin.chronyd
## files in /etc/apparmor.d changed 2026-07-18 or 2026-08-21
/etc/apparmor.d/lxc
/etc/apparmor.d/abi
/etc/apparmor.d/abstractions
/etc/apparmor.d/abstractions/lxc
/etc/apparmor.d/abstractions/ubuntu-browsers.d
/etc/apparmor.d/abstractions/apparmor_api
/etc/apparmor.d/local
/etc/apparmor.d/local/usr.bin.tcpdump
/etc/apparmor.d/local/lxc-containers
/etc/apparmor.d/local/usr.sbin.dhclient
## versions
apparmor 4.1.1-pmx1
chrony 4.8-4~bpo13+2
AppArmor parser version 4.1.1
## dpkg log
2026-08-21 15:41:16 install apparmor:amd64 <none> 4.1.1-pmx1
2026-08-21 15:41:58 install proxmox-kernel-7.0.2-6-pve-signed:amd64 <none> 7.0.2-6
2026-10-04 09:42:20 install proxmox-kernel-7.0.14-20-pve-signed:amd64 <none> 7.0.14-20
2026-10-10 04:40:04 upgrade chrony:amd64 4.6.1-3+deb13u1 4.8-4~bpo13+2
2026-10-10 04:41:43 install proxmox-kernel-7.0.14-22-pve-signed:amd64 <none> 7.0.14-22
2026-10-11 04:42:12 install proxmox-kernel-7.0.14-23-pve-signed:amd64 <none> 7.0.14-23
2026-10-16 04:48:58 install proxmox-kernel-7.0.14-24-pve-signed:amd64 <none> 7.0.14-24
2026-10-21 04:45:13 install proxmox-kernel-7.0.14-25-pve-signed:amd64 <none> 7.0.14-25
2026-10-29 05:06:43 install proxmox-kernel-7.0.14-26-pve-signed:amd64 <none> 7.0.14-26
2026-11-10 02:50:13 install proxmox-kernel-7.0.14-28-pve-signed:amd64 <none> 7.0.14-28
## kernels booted
IDX BOOT ID FIRST ENTRY LAST ENTRY
-5 182af91122a941ec97ac7ee247a4d470 Sun 2026-10-04 21:42:08 CEST Sun 2026-10-04 21:43:33 CEST
-4 bb68aee4240244718d8a7f39d999a3a8 Wed 2026-11-04 21:06:10 CET Tue 2026-11-10 02:53:28 CET
-3 b6e0edea2b5647dfa36f5bb3825adc3a Tue 2026-11-10 02:53:57 CET Tue 2026-11-10 13:56:35 CET
-2 d1167d9431bf459abf2a569e8997a569 Tue 2026-11-10 13:57:04 CET Tue 2026-11-10 15:32:31 CET
-1 e2cc9e3e60f14bde8949f01d7ba35ccf Tue 2026-11-10 15:32:59 CET Tue 2026-11-10 15:42:26 CET
0 bed106f79e804eb1ace314a26da7f049 Tue 2026-11-10 15:42:54 CET Wed 2026-11-11 14:16:04 CET
br-1359a945d146: port 1(vethf5b125d) entered disabled state
Linux version 7.0.14-28-pve (build@proxmox) (gcc (Debian 14.
Linux version 7.0.14-28-pve (build@proxmox) (gcc (Debian 14.
Linux version 7.0.14-26-pve (build@proxmox) (gcc (Debian 14.
Linux version 7.0.14-28-pve (build@proxmox) (gcc (Debian 14.
## chronyd confinement now
/usr/sbin/chronyd (enforce) 780 ? 00:00:00 chronyd
/usr/sbin/chronyd (enforce) 903 ? 00:00:00 chronyd
## what installed chrony 4.8
Start-Date: 2026-10-10 04:39:59
Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --only-upgrade --no-install-recommends frr=10.6.1-1+pve3 libradosstriper1=19.2.6-pve4 librgw2=19.2.6-pve4 ceph-common=19.2.6-pve4 librbd1=19.2.6-pve4 libra
Requested-By: felhom-agent (999)
Upgrade: librados2:amd64 (19.2.3-pve4, 19.2.6-pve4), pve-docs:amd64 (9.2.1, 9.2.13), pve-edk2-firmware-ovmf:amd64 (4.2025.05-2, 4.2026.08-1), ceph-fuse:amd64 (19.2.3-pve4, 19.2.6-pve4), proxmox-widget-toolkit:amd64 (5.2.2, 5.2.10), corosync:amd64 (3.1.10-pve2,
Start-Date: 2026-10-10 04:41:43
@@ -0,0 +1,232 @@
#### demo-hp 2026-11-11T13:11:00Z
net.unix.max_dgram_qlen = 512
kernel.printk_ratelimit = 5
## cmdline
BOOT_IMAGE=/boot/vmlinuz-7.0.14-28-pve root=/dev/mapper/pve-root ro quiet
## apparmor denied this boot: 4
Nov 10 15:43:11 demo-hp kernel: audit: type=1400 audit(1794321791.730:129): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=2094 comm="vgs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
Nov 10 15:43:11 demo-hp kernel: audit: type=1400 audit(1794321791.782:130): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=2095 comm="lvs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
Nov 10 15:43:11 demo-hp kernel: audit: type=1400 audit(1794321791.923:131): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=2111 comm="vgs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
Nov 10 15:43:11 demo-hp kernel: audit: type=1400 audit(1794321791.986:132): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=2112 comm="lvs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
## audit lines with chrony, all boots
## the second around the chrony start, this boot
start: 2026-11-10T15:42:59+01:00
[ 8.825037] demo-hp kernel: input: HD-Audio Generic HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:08.1/0000:05:00.1/sound/card0/input9
[ 8.826027] demo-hp kernel: input: HD-Audio Generic HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:08.1/0000:05:00.1/sound/card0/input10
[ 8.826114] demo-hp kernel: input: HD-Audio Generic HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:08.1/0000:05:00.1/sound/card0/input11
[ 8.827023] demo-hp kernel: input: HD-Audio Generic HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:08.1/0000:05:00.1/sound/card0/input12
[ 8.831933] demo-hp systemd[1]: Reached target sound.target - Sound Card.
[ 8.837028] demo-hp kernel: amdgpu: Virtual CRAT table created for CPU
[ 8.837135] demo-hp kernel: amdgpu: Topology: Add CPU node
[ 8.837161] demo-hp kernel: amdgpu 0000:05:00.0: enabling device (0106 -> 0107)
[ 8.837625] demo-hp kernel: amdgpu 0000:05:00.0: initializing kernel modesetting (RAVEN 0x1002:0x15DD 0x103C:0x8522 0x82).
[ 8.838111] demo-hp kernel: amdgpu 0000:05:00.0: register mmio base: 0xFCB00000
[ 8.838441] demo-hp kernel: amdgpu 0000:05:00.0: register mmio size: 524288
[ 8.840015] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 0 <common_v2_0_0> (soc15_common)
[ 8.840395] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 1 <gmc_v9_0_0> (gmc_v9_0)
[ 8.840735] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 2 <ih_v4_0_0> (vega10_ih)
[ 8.841064] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 3 <psp_v10_0_0> (psp)
[ 8.841387] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 4 <smu_v1_0_0> (powerplay)
[ 8.841727] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 5 <dce_v1_0_0> (dm)
[ 8.842073] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 6 <gfx_v9_0_0> (gfx_v9_0)
[ 8.842434] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 7 <sdma_v4_0_0> (sdma_v4_0)
[ 8.842772] demo-hp kernel: amdgpu 0000:05:00.0: detected ip block number 8 <vcn_v1_0_0> (vcn_v1_0)
[ 8.843155] demo-hp kernel: amdgpu 0000:05:00.0: Fetched VBIOS from VFCT
[ 8.843500] demo-hp kernel: amdgpu 0000:05:00.0: [drm] ATOM BIOS: 113-RAVEN-111
[ 8.843847] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: ALC256: picked fixup (pin match)
[ 8.844233] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: autoconfig for ALC256: line_outs=1 (0x14/0x0/0x0/0x0/0x0) type:speaker
[ 8.844547] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: speaker_outs=0 (0x0/0x0/0x0/0x0/0x0)
[ 8.844874] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: hp_outs=1 (0x21/0x0/0x0/0x0/0x0)
[ 8.845180] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: mono: mono_out=0x0
[ 8.845473] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: inputs:
[ 8.845756] demo-hp kernel: snd_hda_codec_alc269 hdaudioC1D0: Mic=0x19
[ 8.870030] demo-hp kernel: Console: switching to colour dummy device 80x25
[ 8.871013] demo-hp kernel: amdgpu 0000:05:00.0: vgaarb: deactivate vga console
[ 8.871457] demo-hp kernel: amdgpu 0000:05:00.0: Trusted Memory Zone (TMZ) feature enabled
[ 8.871771] demo-hp kernel: amdgpu 0000:05:00.0: vm size is 262144 GB, 4 levels, block size is 9-bit, fragment size is 9-bit
[ 8.872113] demo-hp kernel: amdgpu 0000:05:00.0: VRAM: 2048M 0x000000F400000000 - 0x000000F47FFFFFFF (2048M used)
[ 8.872452] demo-hp kernel: amdgpu 0000:05:00.0: GART: 1024M 0x0000000000000000 - 0x000000003FFFFFFF
[ 8.872757] demo-hp kernel: amdgpu 0000:05:00.0: [drm] Detected VRAM RAM=2048M, BAR=2048M
[ 8.873106] demo-hp kernel: amdgpu 0000:05:00.0: [drm] RAM width 128bits DDR4
[ 8.873432] demo-hp kernel: amdgpu 0000:05:00.0: 2048M of VRAM memory ready
[ 8.873746] demo-hp kernel: amdgpu 0000:05:00.0: 14997M of GTT memory ready.
[ 8.874109] demo-hp kernel: amdgpu 0000:05:00.0: [drm] GART: num cpu pages 262144, num gpu pages 262144
[ 8.874443] demo-hp kernel: amdgpu 0000:05:00.0: [drm] PCIE GART of 1024M enabled.
[ 8.874765] demo-hp kernel: amdgpu 0000:05:00.0: [drm] PTB located at 0x000000F400A00000
[ 8.875104] demo-hp kernel: amdgpu: hwmgr_sw_init smu backed is smu10_smu
[ 8.875136] demo-hp kernel: amdgpu 0000:05:00.0: [VCN instance 0] Found VCN firmware Version ENC: 1.15 DEC: 3 VEP: 0 Revision: 0
[ 8.882025] demo-hp kernel: input: HD-Audio Generic Mic as /devices/pci0000:00/0000:00:08.1/0000:05:00.6/sound/card1/input13
[ 8.882116] demo-hp kernel: input: HD-Audio Generic Front Headphone as /devices/pci0000:00/0000:00:08.1/0000:05:00.6/sound/card1/input14
[ 8.895018] demo-hp kernel: amdgpu 0000:05:00.0: reserve 0x400000 from 0xf47fc00000 for PSP TMR
[ 8.974285] demo-hp kernel: amdgpu 0000:05:00.0: RAS: optional ras ta ucode is not available
[ 8.982279] demo-hp kernel: amdgpu 0000:05:00.0: RAP: optional rap ta ucode is not available
[ 8.982905] demo-hp kernel: amdgpu 0000:05:00.0: SECUREDISPLAY: optional securedisplay ta ucode is not available
[ 8.990029] demo-hp kernel: [drm] DM_PPLIB: values for F clock
[ 8.990101] demo-hp kernel: [drm] DM_PPLIB: 400000 in kHz, 3599 in mV
[ 8.990150] demo-hp kernel: [drm] DM_PPLIB: 933000 in kHz, 3724 in mV
[ 8.990192] demo-hp kernel: [drm] DM_PPLIB: 1067000 in kHz, 3924 in mV
[ 8.990225] demo-hp kernel: [drm] DM_PPLIB: 1200000 in kHz, 4074 in mV
[ 8.990252] demo-hp kernel: [drm] DM_PPLIB: values for DCF clock
[ 8.990295] demo-hp kernel: [drm] DM_PPLIB: 300000 in kHz, 3599 in mV
[ 8.990330] demo-hp kernel: [drm] DM_PPLIB: 600000 in kHz, 3724 in mV
[ 8.990379] demo-hp kernel: [drm] DM_PPLIB: 626000 in kHz, 3924 in mV
[ 8.990415] demo-hp kernel: [drm] DM_PPLIB: 654000 in kHz, 4074 in mV
[ 8.990720] demo-hp kernel: amdgpu 0000:05:00.0: [drm] Display Core v3.2.369 initialized on DCN 1.0
[ 8.994046] demo-hp kernel: snd_hda_intel 0000:05:00.1: bound 0000:05:00.0 (ops amdgpu_dm_audio_component_bind_ops [amdgpu])
[ 9.036409] demo-hp kernel: amdgpu 0000:05:00.0: kiq ring mec 2 pipe 1 q 0
[ 9.053051] demo-hp kernel: kfd kfd: Allocated 3969056 bytes on gart
[ 9.053414] demo-hp kernel: kfd kfd: Total number of KFD nodes to be created: 1
[ 9.053588] demo-hp kernel: amdgpu: Virtual CRAT table created for GPU
[ 9.053620] demo-hp kernel: amdgpu: Topology: Add GPU node [0x1002:0x15dd]
[ 9.053643] demo-hp kernel: kfd kfd: added device 1002:15dd
[ 9.053792] demo-hp kernel: amdgpu 0000:05:00.0: SE 1, SH per SE 1, CU per SH 11, active_cu_number 8
[ 9.054099] demo-hp kernel: amdgpu 0000:05:00.0: ring gfx uses VM inv eng 0 on hub 0
[ 9.054373] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.0.0 uses VM inv eng 1 on hub 0
[ 9.054628] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.1.0 uses VM inv eng 4 on hub 0
[ 9.054881] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.2.0 uses VM inv eng 5 on hub 0
[ 9.055139] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.3.0 uses VM inv eng 6 on hub 0
[ 9.055384] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.0.1 uses VM inv eng 7 on hub 0
[ 9.055641] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.1.1 uses VM inv eng 8 on hub 0
[ 9.055897] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.2.1 uses VM inv eng 9 on hub 0
[ 9.056158] demo-hp kernel: amdgpu 0000:05:00.0: ring comp_1.3.1 uses VM inv eng 10 on hub 0
[ 9.056432] demo-hp kernel: amdgpu 0000:05:00.0: ring kiq_0.2.1.0 uses VM inv eng 11 on hub 0
[ 9.056680] demo-hp kernel: amdgpu 0000:05:00.0: ring sdma0 uses VM inv eng 0 on hub 8
[ 9.056932] demo-hp kernel: amdgpu 0000:05:00.0: ring vcn_dec uses VM inv eng 1 on hub 8
[ 9.057198] demo-hp kernel: amdgpu 0000:05:00.0: ring vcn_enc0 uses VM inv eng 4 on hub 8
[ 9.057469] demo-hp kernel: amdgpu 0000:05:00.0: ring vcn_enc1 uses VM inv eng 5 on hub 8
[ 9.057727] demo-hp kernel: amdgpu 0000:05:00.0: ring jpeg_dec uses VM inv eng 6 on hub 8
[ 9.060024] demo-hp kernel: amdgpu: pp_dpm_get_sclk_od was not implemented.
[ 9.060097] demo-hp kernel: amdgpu: pp_dpm_get_mclk_od was not implemented.
[ 9.060121] demo-hp kernel: amdgpu 0000:05:00.0: Runtime PM not available
[ 9.061030] demo-hp kernel: amdgpu 0000:05:00.0: [drm] Registered 4 planes with drm panic
[ 9.068049] demo-hp kernel: [drm] Initialized amdgpu 3.64.0 for 0000:05:00.0 on minor 1
[ 9.081054] demo-hp kernel: amdgpu 0000:05:00.0: [drm] Failed to setup vendor infoframe on connector DP-1: -22
[ 9.086060] demo-hp kernel: fbcon: amdgpudrmfb (fb0) is primary device
[ 9.131047] demo-hp kernel: Console: switching to colour frame buffer device 240x67
[ 9.147045] demo-hp kernel: amdgpu 0000:05:00.0: [drm] fb0: amdgpudrmfb frame buffer device
[ 9.170506] demo-hp systemd[1]: Finished ifupdown2-pre.service - Helper to synchronize boot up for ifupdown.
[ 9.171027] demo-hp systemd[1]: Finished systemd-udev-settle.service - Wait for udev To Complete Device Initialization.
[ 9.171890] demo-hp systemd[1]: zfs-import-cache.service - Import ZFS pools by cache file skipped, unmet condition check ConditionFileNotEmpty=/etc/zfs/zpool.cache
[ 9.172055] demo-hp systemd[1]: Reached target zfs-import.target - ZFS pool import target.
[ 9.179591] demo-hp systemd[1]: Starting zfs-mount.service - Mount ZFS filesystems...
[ 9.181286] demo-hp systemd[1]: Starting zfs-volume-wait.service - Wait for ZFS Volume (zvol) links in /dev...
[ 9.207137] demo-hp zvol_wait[623]: No zvols found, nothing to do.
[ 9.207944] demo-hp systemd[1]: Finished zfs-mount.service - Mount ZFS filesystems.
[ 9.208529] demo-hp systemd[1]: Finished zfs-volume-wait.service - Wait for ZFS Volume (zvol) links in /dev.
[ 9.209473] demo-hp systemd[1]: Reached target local-fs.target - Local File Systems.
[ 9.209606] demo-hp systemd[1]: Reached target zfs-volumes.target - ZFS volumes are ready.
[ 9.211049] demo-hp systemd[1]: Listening on systemd-sysext.socket - System Extension Image Management.
[ 9.212846] demo-hp systemd[1]: Starting apparmor.service - Load AppArmor profiles...
[ 9.214592] demo-hp systemd[1]: Starting console-setup.service - Set console font and keymap...
[ 9.216370] demo-hp systemd[1]: Starting felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit)...
[ 9.218403] demo-hp systemd[1]: Starting networking.service - Network initialization...
[ 9.220586] demo-hp systemd[1]: Starting pvebanner.service - Proxmox VE Login Banner...
[ 9.222876] demo-hp systemd[1]: Starting pvefw-logger.service - Proxmox VE firewall logger...
[ 9.225689] demo-hp systemd[1]: Starting pvenetcommit.service - Commit Proxmox VE network changes...
[ 9.228763] demo-hp systemd[1]: Starting systemd-binfmt.service - Set Up Additional Binary Formats...
[ 9.230821] demo-hp systemd[1]: Starting systemd-tmpfiles-setup.service - Create System Files and Directories...
[ 9.233511] demo-hp systemd[1]: Finished console-setup.service - Set console font and keymap.
[ 9.242393] demo-hp apparmor.systemd[630]: Restarting AppArmor
[ 9.242393] demo-hp apparmor.systemd[630]: Reloading AppArmor profiles
[ 9.248120] demo-hp systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 640 (systemd-binfmt)
[ 9.250462] demo-hp systemd-tmpfiles[641]: /etc/tmpfiles.d/felhom-privsep.conf:15: Duplicate line for path "/run/sshd", ignoring.
[ 9.250689] demo-hp systemd-tmpfiles[641]: /usr/lib/tmpfiles.d/legacy.conf:14: Duplicate line for path "/run/lock", ignoring.
[ 9.251172] demo-hp systemd[1]: Mounting proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System...
[ 9.251836] demo-hp systemd[1]: Finished pvenetcommit.service - Commit Proxmox VE network changes.
[ 9.253408] demo-hp networking[633]: networking: Configuring network interfaces
[ 9.272681] demo-hp pvefw-logger[668]: starting pvefw logger
[ 9.274360] demo-hp systemd[1]: Started pvefw-logger.service - Proxmox VE firewall logger.
[ 9.276036] demo-hp kernel: audit: type=1400 audit(1794321779.088:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="Discord" pid=660 comm="apparmor_parser"
[ 9.276172] demo-hp kernel: audit: type=1400 audit(1794321779.088:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name=4D6F6E676F444220436F6D70617373 pid=661 comm="apparmor_pars
[ 9.276224] demo-hp kernel: audit: type=1400 audit(1794321779.088:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="QtWebEngineProcess" pid=662 comm="apparmor_parser"
[ 9.277127] demo-hp kernel: audit: type=1400 audit(1794321779.089:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="1password" pid=659 comm="apparmor_parser"
[ 9.277255] demo-hp kernel: audit: type=1400 audit(1794321779.090:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="balena-etcher" pid=664 comm="apparmor_parser"
[ 9.277292] demo-hp kernel: audit: type=1400 audit(1794321779.090:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="buildah" pid=666 comm="apparmor_parser"
[ 9.277340] demo-hp kernel: audit: type=1400 audit(1794321779.090:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="brave" pid=665 comm="apparmor_parser"
[ 9.279225] demo-hp kernel: audit: type=1400 audit(1794321779.091:11): apparmor="STATUS" operation="profile_load" profile="unconfined" name="cam" pid=672 comm="apparmor_parser"
[ 9.279392] demo-hp kernel: audit: type=1400 audit(1794321779.092:12): apparmor="STATUS" operation="profile_load" profile="unconfined" name="ch-checkns" pid=673 comm="apparmor_parser"
[ 9.279919] demo-hp systemd[1]: Finished systemd-tmpfiles-setup.service - Create System Files and Directories.
[ 9.282035] demo-hp kernel: audit: type=1400 audit(1794321779.094:13): apparmor="STATUS" operation="profile_load" profile="unconfined" name="busybox" pid=671 comm="apparmor_parser"
[ 9.284316] demo-hp systemd[1]: Mounting run-rpc_pipefs.mount - RPC Pipe File System...
[ 9.284460] demo-hp systemd[1]: ldconfig.service - Rebuild Dynamic Linker Cache skipped, no trigger condition checks were met.
[ 9.287648] demo-hp systemd[1]: Starting rpcbind.service - RPC bind portmap service...
[ 9.287715] demo-hp systemd[1]: systemd-firstboot.service - First Boot Wizard skipped, unmet condition check ConditionFirstBoot=yes
[ 9.287753] demo-hp systemd[1]: first-boot-complete.target - First Boot Complete skipped, unmet condition check ConditionFirstBoot=yes
[ 9.287841] demo-hp systemd[1]: systemd-journal-catalog-update.service - Rebuild Journal Catalog skipped, unmet condition check ConditionNeedsUpdate=/var
[ 9.287922] demo-hp systemd[1]: systemd-machine-id-commit.service - Save Transient machine-id to Disk skipped, unmet condition check ConditionPathIsMountPoint=/etc/machine-id
[ 9.288596] demo-hp systemd[1]: systemd-update-done.service - Update is Completed skipped, no trigger condition checks were met.
[ 9.290860] demo-hp systemd[1]: Mounted proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System.
[ 9.303595] demo-hp systemd[1]: Finished systemd-binfmt.service - Set Up Additional Binary Formats.
[ 9.317151] demo-hp systemd[1]: Started rpcbind.service - RPC bind portmap service.
[ 9.317289] demo-hp systemd[1]: Reached target rpcbind.target - RPC Port Mapper.
[ 9.364582] demo-hp felhom-crash-guard[632]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10
[ 9.375481] demo-hp felhom-crash-guard[775]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10
## chrony lines per boot (all boots)
-- boot -5
-- boot -4
Nov 10 02:50:56 demo-hp chronyd[774]: chronyd exiting
Nov 10 02:50:56 demo-hp systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Nov 10 02:50:56 demo-hp systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
-- boot -3
Nov 10 02:54:02 demo-hp systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Nov 10 02:54:02 demo-hp systemd[1]: Started chrony.service - chrony, an NTP client/server.
Nov 10 13:54:26 demo-hp chronyd[778]: chronyd exiting
Nov 10 13:54:26 demo-hp systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Nov 10 13:54:26 demo-hp systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
-- boot -2
Nov 10 13:57:09 demo-hp systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Nov 10 13:57:09 demo-hp systemd[1]: Started chrony.service - chrony, an NTP client/server.
Nov 10 15:30:24 demo-hp chronyd[778]: chronyd exiting
Nov 10 15:30:24 demo-hp systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Nov 10 15:30:24 demo-hp systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
-- boot -1
Nov 10 15:33:04 demo-hp systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Nov 10 15:33:04 demo-hp systemd[1]: Started chrony.service - chrony, an NTP client/server.
Nov 10 15:40:09 demo-hp chronyd[780]: chronyd exiting
Nov 10 15:40:09 demo-hp systemd[1]: Stopping chrony.service - chrony, an NTP client/server...
Nov 10 15:40:09 demo-hp systemd[1]: Stopped chrony.service - chrony, an NTP client/server.
-- boot 0
Nov 10 15:42:59 demo-hp systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Nov 10 15:42:59 demo-hp systemd[1]: Started chrony.service - chrony, an NTP client/server.
## sandbox env of pid1
total 0
drwxr-xr-x 20 root root 540 Nov 11 14:07 .
drwxr-xr-x 36 root root 1520 Nov 11 14:11 ..
drwxr-xr-x 2 root root 40 Nov 10 15:42 ask-password
drwxr-xr-x 6 root root 260 Nov 10 15:58 generator
dr-xr-xr-x 3 root root 160 Nov 10 15:42 inaccessible
drwxr-xr-x 2 root root 40 Nov 10 15:42 incoming
drwxr-xr-x 2 root root 40 Nov 10 15:42 inhibit
srw-rw-rw- 1 root root 0 Nov 10 15:42 io.systemd.Credentials
srw-rw-rw- 1 root root 0 Nov 10 15:42 io.systemd.Hostname
srw-rw-rw- 1 root root 0 Nov 10 15:42 io.systemd.ManagedOOM
srw------- 1 root root 0 Nov 10 15:42 io.systemd.sysext
drwxr-xr-x 3 root root 200 Nov 10 15:42 journal
drwxr-xr-x 2 root root 40 Nov 10 15:42 machines
dr-xr-xr-x 2 root root 40 Nov 10 15:42 mount-rootfs
drwxr-xr-x 4 systemd-network systemd-network 80 Nov 10 15:42 netif
srwxrwxrwx 1 root root 0 Nov 10 15:42 notify
srwx------ 1 root root 0 Nov 10 15:42 private
drwxr-xr-x 7 root root 140 Nov 11 14:08 propagate
drwxr-xr-x 2 root root 60 Nov 10 15:42 seats
drwxr-xr-x 2 root root 120 Nov 11 14:11 sessions
drwxr-xr-x 2 root root 40 Nov 10 15:42 shutdown
drwxr-xr-x 2 root root 40 Nov 10 15:42 system
-r--r--r-- 1 root root 0 Nov 10 15:58 systemd-units-load
drwxr-xr-x 2 root root 100 Nov 11 14:10 transient
drwxr-xr-x 2 root root 1940 Nov 11 14:11 units
drwxr-xr-x 2 root root 60 Nov 10 15:42 userdb
drwxr-xr-x 2 root root 60 Nov 11 14:11 users
## tailscale
inactive
## rtc
ls: cannot access '/dev/pps*': No such file or directory
/dev/ptp0
/dev/ptp1
/dev/ptp2
/dev/ptp3
/dev/rtc
/dev/rtc0
## cpu
8
model name : AMD Ryzen Embedded V1756B with Radeon Vega Gfx
@@ -0,0 +1,273 @@
#### host demo-hp 2026-11-11T13:07:29Z local 14:07:29+0100
## versions
pve-manager/9.2.26/41a0e2e09b9e6f9d (running kernel: 7.0.14-28-pve)
ii chrony 4.8-4~bpo13+2
ii systemd 257.13-1~deb13u1
un systemd-timesyncd <none>
7.0.14-28-pve
## unit state
Type=notify
PIDFile=/run/chrony/chronyd.pid
NotifyAccess=main
Result=success
NRestarts=0
ExecMainCode=0
ExecMainStatus=0
ActiveState=active
SubState=running
FragmentPath=/usr/lib/systemd/system/chrony.service
DropInPaths=
UnitFileState=enabled
ActiveEnterTimestamp=Tue 2026-11-10 15:42:59 CET
InactiveEnterTimestamp=
## unit cat
# /usr/lib/systemd/system/chrony.service
[Unit]
Description=chrony, an NTP client/server
Documentation=man:chronyd(8) man:chronyc(1) man:chrony.conf(5)
Conflicts=openntpd.service ntp.service ntpsec.service
ConditionVirtualization=|!container
ConditionVirtualization=|wsl
ConditionCapability=CAP_SYS_TIME
[Service]
Type=notify
PIDFile=/run/chrony/chronyd.pid
EnvironmentFile=-/etc/default/chrony
User=_chrony
# Daemon is started as root, but still sandboxed
ExecStart=!/usr/sbin/chronyd -n $DAEMON_OPTS
CapabilityBoundingSet=~CAP_AUDIT_CONTROL CAP_AUDIT_READ CAP_AUDIT_WRITE
CapabilityBoundingSet=~CAP_BLOCK_SUSPEND CAP_KILL CAP_LEASE CAP_LINUX_IMMUTABLE
CapabilityBoundingSet=~CAP_MAC_ADMIN CAP_MAC_OVERRIDE CAP_MKNOD CAP_SYS_ADMIN
CapabilityBoundingSet=~CAP_SYS_BOOT CAP_SYS_CHROOT CAP_SYS_MODULE CAP_SYS_PACCT
CapabilityBoundingSet=~CAP_SYS_PTRACE CAP_SYS_RAWIO CAP_SYS_TTY_CONFIG CAP_WAKE_ALARM
DeviceAllow=char-pps rw
DeviceAllow=char-ptp rw
DeviceAllow=char-rtc rw
DevicePolicy=closed
LockPersonality=yes
MemoryDenyWriteExecute=yes
NoNewPrivileges=yes
PrivateTmp=yes
ProcSubset=pid
ProtectControlGroups=yes
ProtectHome=yes
ProtectHostname=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectProc=invisible
ProtectSystem=strict
# Used for gps refclocks
ReadWritePaths=/run
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=yes
RestrictSUIDSGID=yes
SystemCallArchitectures=native
SystemCallFilter=~@cpu-emulation @debug @module @mount @obsolete @raw-io @reboot @swap
ConfigurationDirectory=chrony
RuntimeDirectory=chrony
RuntimeDirectoryMode=0700
# See dumpdir in chrony.conf(5)
RuntimeDirectoryPreserve=restart
StateDirectory=chrony
StateDirectoryMode=0750
LogsDirectory=chrony
LogsDirectoryMode=0750
# Adjust restrictions for /usr/sbin/sendmail (mailonchange directive)
NoNewPrivileges=no
ReadWritePaths=-/var/spool
RestrictAddressFamilies=AF_NETLINK
[Install]
Alias=chronyd.service
WantedBy=multi-user.target
## files
ls: cannot access '/etc/systemd/system/chrony.service.d': No such file or directory
ls: cannot access '/etc/systemd/system/chronyd.service.d': No such file or directory
ls: cannot access '/etc/systemd/system/chrony.service': No such file or directory
lrwxrwxrwx 1 root root 38 May 21 13:32 /etc/systemd/system/chronyd.service -> /usr/lib/systemd/system/chrony.service
-rw-r--r-- 1 root root 1929 Jul 1 22:28 /lib/systemd/system/chrony.service
-rw-r--r-- 1 root root 1929 Jul 1 22:28 /usr/lib/systemd/system/chrony.service
## dpkg verify chrony
rc=0
## conf
/etc/chrony:
total 24
drwxr-xr-x 4 root root 4096 May 21 13:32 .
drwxr-xr-x 102 root root 4096 Nov 8 02:49 ..
-rw-r--r-- 1 root root 1450 Feb 16 2026 chrony.conf
-rw-r----- 1 root _chrony 481 Feb 16 2026 chrony.keys
drwxr-xr-x 2 root root 4096 Oct 10 04:40 conf.d
drwxr-xr-x 2 root root 4096 Oct 10 04:40 sources.d
/etc/chrony/conf.d:
total 12
drwxr-xr-x 2 root root 4096 Oct 10 04:40 .
drwxr-xr-x 4 root root 4096 May 21 13:32 ..
-rw-r--r-- 1 root root 422 Feb 16 2026 README
/etc/chrony/sources.d:
total 12
drwxr-xr-x 2 root root 4096 Oct 10 04:40 .
drwxr-xr-x 4 root root 4096 May 21 13:32 ..
-rw-r--r-- 1 root root 498 Feb 16 2026 README
pool 2.debian.pool.ntp.org iburst
sourcedir /run/chrony-dhcp
sourcedir /etc/chrony/sources.d
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
ntsdumpdir /var/lib/chrony
logdir /var/log/chrony
maxupdateskew 100.0
rtcsync
makestep 1 3
leapseclist /usr/share/zoneinfo/leap-seconds.list
confdir /etc/chrony/conf.d
-- /etc/chrony/conf.d/README
Files found under the /etc/chrony/conf.d directory with the .conf suffix are
parsed in the lexicographical order of the file names when chronyd starts up.
This enables a fragmented configuration of chronyd.
Although those files can contain any directives listed in chrony.conf(5),
it would be wiser to add NTP sources in the /etc/chrony/sources.d
directory. Please read /etc/chrony/sources.d/README for more information.
-- /etc/chrony/sources.d/README
Only NTP sources can be specified in the /etc/chrony/sources.d directory.
Files in this directory must end with the ".sources" suffix. They can only
contain the "peer", "pool" and "server" directives and must have all
lines terminated by a trailing newline.
There is no need to restart chronyd for these time sources to be usable,
running 'chronyc reload sources' is sufficient.
Example:
## default
DAEMON_OPTS="-F 1"
## apparmor
/usr/sbin/chronyd
/usr/sbin/chronyd (780)
/usr/sbin/chronyd (903)
-rw-r--r-- 1 root root 0 May 21 13:32 /etc/apparmor.d/local/usr.sbin.chronyd
-rw-r--r-- 1 root root 2837 Jul 1 22:28 /etc/apparmor.d/usr.sbin.chronyd
/etc/apparmor.d/disable:
total 8
drwxr-xr-x 2 root root 4096 Jul 21 2025 .
drwxr-xr-x 9 root root 4096 Nov 8 02:48 ..
## apparmor profile notify lines
81: # Allow systemd Type=notify using sd_notify's $NOTIFY_SOCKET
82: @{run}/systemd/notify w,
## kernel audit lines for chronyd (last 20)
## journal chrony this boot
Nov 10 15:42:59 demo-hp systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Nov 10 15:42:59 demo-hp chronyd[780]: chronyd version 4.8 starting (+CMDMON +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +NTS +SECHASH +IPV6 -DEBUG)
Nov 10 15:42:59 demo-hp chronyd[780]: Loaded 0 symmetric keys
Nov 10 15:42:59 demo-hp chronyd[780]: Using leap second list /usr/share/zoneinfo/leap-seconds.list
Nov 10 15:42:59 demo-hp chronyd[780]: Frequency -12.230 +/- 1.327 ppm read from /var/lib/chrony/chrony.drift
Nov 10 15:42:59 demo-hp chronyd[780]: Loaded seccomp filter (level 1)
Nov 10 15:42:59 demo-hp systemd[1]: Started chrony.service - chrony, an NTP client/server.
Nov 10 15:43:10 demo-hp chronyd[780]: Selected source 81.0.124.253 (2.debian.pool.ntp.org)
Nov 10 15:43:10 demo-hp chronyd[780]: System clock TAI offset set to 37 seconds
Nov 10 15:43:11 demo-hp chronyd[780]: Selected source 37.221.209.24 (2.debian.pool.ntp.org)
Nov 10 15:44:16 demo-hp chronyd[780]: Selected source 195.111.92.55 (2.debian.pool.ntp.org)
Nov 10 15:46:26 demo-hp chronyd[780]: Selected source 37.221.209.24 (2.debian.pool.ntp.org)
## first failure ever in the journal
count: 0
## oldest journal line
Oct 04 21:42:08 demo-hp systemd[779137]: Queued start job for default target def
## last good start in the journal
Nov 10 15:43:11 demo-hp chronyd[780]: Selected source 37.221.209.24 (2.debian.pool.ntp.org)
Nov 10 15:44:16 demo-hp chronyd[780]: Selected source 195.111.92.55 (2.debian.pool.ntp.org)
Nov 10 15:46:26 demo-hp chronyd[780]: Selected source 37.221.209.24 (2.debian.pool.ntp.org)
## boots
-5 182af91122a941ec97ac7ee247a4d470 Sun 2026-10-04 21:42:08 CEST Sun 2026-10-04 21:43:33 CEST
-4 bb68aee4240244718d8a7f39d999a3a8 Wed 2026-11-04 21:06:10 CET Tue 2026-11-10 02:53:28 CET
-3 b6e0edea2b5647dfa36f5bb3825adc3a Tue 2026-11-10 02:53:57 CET Tue 2026-11-10 13:56:35 CET
-2 d1167d9431bf459abf2a569e8997a569 Tue 2026-11-10 13:57:04 CET Tue 2026-11-10 15:32:31 CET
-1 e2cc9e3e60f14bde8949f01d7ba35ccf Tue 2026-11-10 15:32:59 CET Tue 2026-11-10 15:42:26 CET
0 bed106f79e804eb1ace314a26da7f049 Tue 2026-11-10 15:42:54 CET Wed 2026-11-11 14:07:29 CET
## what starts with it
chrony.service
● ├─-.mount
● ├─system.slice
● ├─systemd-journald.socket
● ├─systemd-remount-fs.service
● ├─systemd-tmpfiles-setup.service
● ├─tmp.mount
● ├─basic.target
● │ ├─-.mount
○ │ ├─systemd-pcrphase-sysinit.service
● │ ├─tmp.mount
● │ ├─paths.target
● │ │ ├─systemd-ask-password-console.path
● │ │ └─systemd-ask-password-wall.path
● │ ├─slices.target
● │ │ ├─-.slice
● │ │ ├─system-xfs_scrub.slice
● │ │ ├─system.slice
● │ │ └─user.slice
● │ ├─sockets.target
● │ │ ├─dbus.socket
● │ │ ├─iscsid.socket
● │ │ ├─netavark-dhcp-proxy.socket
● │ │ ├─rrdcached.socket
● │ │ ├─sshd-unix-local.socket
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.
chrony.service +174ms
└─basic.target @5.760s
└─sockets.target @5.760s
└─systemd-hostnamed.socket @5.760s
└─sysinit.target @5.757s
└─felhom-crash-guard.service @5.583s +173ms
└─local-fs.target @5.576s
└─etc-pve.mount @7.233s
└─local-fs-pre.target @761ms
## other time services
inactive inactive inactive inactive
Local time: Wed 2026-11-11 14:07:32 CET
Universal time: Wed 2026-11-11 13:07:32 UTC
RTC time: Wed 2026-11-11 13:07:33
Time zone: Europe/Budapest (CET, +0100)
System clock synchronized: yes
NTP service: active
RTC in local TZ: no
## chronyc
Reference ID : 25DDD118 (37.221.209.24)
Stratum : 3
Ref time (UTC) : Wed Nov 11 12:51:20 2026
System time : 0.000228395 seconds fast of NTP time
Last offset : +0.000003178 seconds
RMS offset : 0.000066409 seconds
Frequency : 12.927 ppm slow
Residual freq : +0.000 ppm
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^- cebece.pomaz.hu 2 10 377 978 -7115us[-7112us] +/- 32ms
^* 37.221.209.24 2 10 377 971 -2744us[-2741us] +/- 15ms
^+ mail.deployis.eu 4 10 377 758 -3122us[-3122us] +/- 24ms
^+ ntp2.niif.hu 2 10 377 377 -4436us[-4436us] +/- 39ms
## dpkg history chrony/apparmor/systemd
2026-08-21 15:41:16 install apparmor:amd64 <none> 4.1.1-pmx1
2026-08-21 15:42:37 configure apparmor:amd64 4.1.1-pmx1 <none>
2026-10-10 04:40:04 upgrade chrony:amd64 4.6.1-3+deb13u1 4.8-4~bpo13+2
2026-10-10 04:40:23 configure chrony:amd64 4.8-4~bpo13+2 <none>
## felhom files that name chrony
## ps
780 _chrony /usr/sbin/chronyd -n -F 1
903 _chrony /usr/sbin/chronyd -n -F 1
## run dir
srwxrwxrwx 1 root root 0 Nov 10 15:42 /run/systemd/notify
/run/chrony:
total 4
drwx------ 2 _chrony _chrony 80 Nov 11 14:07 .
drwxr-xr-x 36 root root 1520 Nov 11 14:07 ..
-rw-r--r-- 1 root root 4 Nov 10 15:42 chronyd.pid
srwxr-xr-x 1 _chrony _chrony 0 Nov 10 15:42 chronyd.sock
## uptime
2026-11-10 15:42:49
@@ -0,0 +1,26 @@
#### demo-hp 2026-11-11T13:16:38Z
## fresh compile of the on-disk profile to stdout (no load, cache not read, not written)
83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1 -
## loaded
83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1
## cache
dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 /var/cache/apparmor/ae1a4050.0/usr.sbin.chronyd
83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1 /var/cache/apparmor/f6da0254.0/usr.sbin.chronyd
drwx------ 2 root root 4096 2026-08-21 17:44:37.686947147 +0200 /var/cache/apparmor/ae1a4050.0/
-rw-r--r-- 1 root root 3177 2026-08-21 17:44:37.159696460 +0200 /var/cache/apparmor/ae1a4050.0/.features
drwx------ 2 root root 4096 2026-11-08 02:48:51.017396841 +0100 /var/cache/apparmor/f6da0254.0/
-rw-r--r-- 1 root root 3219 2026-10-04 09:44:36.115533363 +0200 /var/cache/apparmor/f6da0254.0/.features
## state
active
chrony 4.8-4~bpo13+2
7.0.14-28-pve
System time : 0.000003214 seconds slow of NTP time
Leap status : Normal
1794402998.136055491
## chrony fatal lines
0
## stale check, every cached profile: cache differs from a fresh compile
current dir: /var/cache/apparmor/f6da0254.0/
STALE: lxc-containers
checked=106 stale=1
@@ -0,0 +1,76 @@
#### demo-felhom 2026-11-11T13:15:31Z
## fresh compile of the on-disk profile, to stdout only (no load, no cache write)
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 -
apparmor_parser: Too many actions given on the command line.
## loaded raw policy sha256
dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647
## cache dirs
total 16
drwxr-xr-x 4 root root 4096 2026-10-07 12:11:32.040268492 +0200 .
drwxr-xr-x 14 root root 4096 2026-07-19 13:08:02.082082737 +0200 ..
drwx------ 2 root root 4096 2026-10-07 14:16:03.051992948 +0200 ae1a4050.0
drwx------ 2 root root 4096 2026-11-08 05:32:24.065142783 +0100 f6da0254.0
/var/cache/apparmor/ae1a4050.0/ features-sha=0e6b37757d22af1e files=106
-rw-r--r-- 1 root root 3177 2026-07-18 18:17:02.251100923 +0200 /var/cache/apparmor/ae1a4050.0//.features
-rw------- 1 root root 296497 2026-07-18 18:15:53.043202621 +0200 /var/cache/apparmor/ae1a4050.0//usr.sbin.chronyd
/var/cache/apparmor/f6da0254.0/ features-sha=611413ba507a5c19 files=106
-rw-r--r-- 1 root root 3219 2026-10-07 12:11:32.040268492 +0200 /var/cache/apparmor/f6da0254.0//.features
-rw------- 1 root root 295529 2026-07-18 18:15:53.043202621 +0200 /var/cache/apparmor/f6da0254.0//usr.sbin.chronyd
## kernel features now
yes yes yes yes yes
611413ba507a5c19
## newest source file the profile depends on
2025-07-21 12:21:49.000000000 /etc/apparmor.d/abi/kernel-5.4-vanilla
2025-07-21 12:21:49.000000000 /etc/apparmor.d/abstractions/base
2025-07-21 12:21:49.000000000 /etc/apparmor.d/abstractions/nameservice
2025-07-21 12:21:49.000000000 /etc/apparmor.d/tunables/global
2026-05-21 13:32:03.000000000 /etc/apparmor.d/local/usr.sbin.chronyd
2026-07-01 22:28:16.000000000 /etc/apparmor.d/usr.sbin.chronyd
## files in /etc/apparmor.d changed 2026-07-18 or 2026-08-21
/etc/apparmor.d/tunables
/etc/apparmor.d/tunables/home.d
/etc/apparmor.d/tunables/home.d/ubuntu
/etc/apparmor.d/tunables/multiarch.d
/etc/apparmor.d/tunables/xdg-user-dirs.d
/etc/apparmor.d/tunables/xdg-user-dirs.d/site.local
/etc/apparmor.d/lxc
/etc/apparmor.d/local
/etc/apparmor.d/local/lsb_release
/etc/apparmor.d/local/usr.bin.swtpm
## versions
apparmor 4.1.1-pmx1
chrony 4.8-4~bpo13+2
AppArmor parser version 4.1.1
## dpkg log
2026-07-18 16:14:58 install apparmor:amd64 <none> 4.1.1-pmx1
2026-07-18 16:15:24 install proxmox-kernel-7.0.2-6-pve-signed:amd64 <none> 7.0.2-6
2026-10-07 12:09:03 install proxmox-kernel-7.0.14-20-pve-signed:amd64 <none> 7.0.14-20
2026-10-07 14:15:40 upgrade chrony:amd64 4.6.1-3+deb13u1 4.8-4~bpo13+2
2026-10-09 04:44:23 install proxmox-kernel-7.0.14-22-pve-signed:amd64 <none> 7.0.14-22
2026-10-10 04:44:06 install proxmox-kernel-7.0.14-23-pve-signed:amd64 <none> 7.0.14-23
2026-10-16 05:06:43 install proxmox-kernel-7.0.14-24-pve-signed:amd64 <none> 7.0.14-24
2026-10-21 05:21:10 install proxmox-kernel-7.0.14-25-pve-signed:amd64 <none> 7.0.14-25
2026-10-29 04:45:31 install proxmox-kernel-7.0.14-26-pve-signed:amd64 <none> 7.0.14-26
2026-11-08 05:32:59 install proxmox-kernel-7.0.14-28-pve-signed:amd64 <none> 7.0.14-28
## kernels booted
IDX BOOT ID FIRST ENTRY LAST ENTRY
-5 46081f567f384ab88591d4e604555a48 Sun 2026-10-04 21:43:33 CEST Sun 2026-10-04 21:43:46 CEST
-4 e82176ae7cf644518507468fc609eaa9 Wed 2026-10-14 15:43:07 CEST Fri 2026-10-16 05:09:06 CEST
-3 ddd3d7305e244b2a9eafb5cf229476f1 Fri 2026-10-16 05:09:23 CEST Wed 2026-10-21 05:23:34 CEST
-2 56db71938b304a57bdedf30f89860e1f Wed 2026-10-21 05:23:48 CEST Thu 2026-10-29 04:47:58 CET
-1 4f6d8b388e224b8ca8ed45e06a2f96b9 Thu 2026-10-29 04:48:11 CET Sun 2026-11-08 05:35:13 CET
0 72899d2625244e069e8e8099775d1e51 Sun 2026-11-08 05:35:21 CET Wed 2026-11-11 14:15:31 CET
docker0: port 1(vethbb70052) entered disabled state
Linux version 7.0.14-24-pve (build@proxmox) (gcc (Debian 14.
Linux version 7.0.14-25-pve (build@proxmox) (gcc (Debian 14.
Linux version 7.0.14-26-pve (build@proxmox) (gcc (Debian 14.
Linux version 7.0.14-28-pve (build@proxmox) (gcc (Debian 14.
## chronyd confinement now
no chronyd process
## what installed chrony 4.8
Start-Date: 2026-10-07 14:15:37
Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --only-upgrade --no-install-recommends frr=10.6.1-1+pve3 librgw2=19.2.6-pve4 libradosstriper1=19.2.6-pve4 ceph-common=19.2.6-pve4 librbd1=19.2.6-pve4 libra
Requested-By: felhom-agent (999)
Upgrade: librados2:amd64 (19.2.3-pve4, 19.2.6-pve4), pve-docs:amd64 (9.2.1, 9.2.13), ceph-fuse:amd64 (19.2.3-pve4, 19.2.6-pve4), proxmox-widget-toolkit:amd64 (5.2.2, 5.2.10), corosync:amd64 (3.1.10-pve2, 3.1.10-pve3), frr:amd64 (10.6.1-1+pve2, 10.6.1-1+pve3),
Start-Date: 2026-10-08 04:37:54
@@ -0,0 +1,236 @@
#### demo-felhom 2026-11-11T13:08:39Z
net.unix.max_dgram_qlen = 512
kernel.printk_ratelimit = 5
## cmdline
BOOT_IMAGE=/boot/vmlinuz-7.0.14-28-pve root=/dev/mapper/pve-root ro quiet softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10
## apparmor denied this boot: 36
Nov 08 05:35:31 demo-felhom kernel: audit: type=1400 audit(1794112531.870:128): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=1529 comm="vgs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
Nov 08 05:35:31 demo-felhom kernel: audit: type=1400 audit(1794112531.898:129): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=1530 comm="lvs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
Nov 08 05:35:31 demo-felhom kernel: audit: type=1400 audit(1794112531.979:130): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=1536 comm="vgs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
Nov 08 05:35:32 demo-felhom kernel: audit: type=1400 audit(1794112532.008:131): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=1537 comm="lvs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
Nov 08 06:06:23 demo-felhom kernel: audit: type=1400 audit(1794114383.198:133): apparmor="DENIED" operation="getattr" class="posix_mqueue" profile="/usr/bin/lxc-start" name="/" pid=43146 comm="vgs" requested="getattr" denied="getattr" class="posix_mqueue" fsuid=0 ouid=0 olabel="unconfined"
## audit lines with chrony, all boots
Oct 16 05:09:25 demo-felhom kernel: audit: type=1400 audit(1792120165.400:121): apparmor="DENIED" operation="sendmsg" class="file" profile="/usr/sbin/chronyd" name="/run/systemd/notify" pid=692 comm="chronyd" requested_mask="w" denied_mask="w" fsuid=100 ouid=0
## the second around the chrony start, this boot
start: 2026-11-08T05:35:23+01:00
[ 3.980022] demo-felhom kernel: intel_pmc_core INT33A1:00: initialized
[ 3.988439] demo-felhom systemd-modules-load[310]: Inserted module 'zfs'
[ 3.989049] demo-felhom kernel: ZFS: Loaded module v2.4.4-pve1, ZFS pool version 5000, ZFS filesystem version 5
[ 3.989820] demo-felhom systemd[1]: Finished systemd-modules-load.service - Load Kernel Modules.
[ 3.991237] demo-felhom systemd[1]: Starting systemd-sysctl.service - Apply Kernel Variables...
[ 4.004023] demo-felhom kernel: usbcore: registered new interface driver btusb
[ 4.011027] demo-felhom kernel: Bluetooth: hci0: RTL: examining hci_ver=0b hci_rev=000b lmp_ver=0b lmp_subver=8852
[ 4.011156] demo-felhom kernel: Bluetooth: hci0: RTL: rom_version status=0 version=1
[ 4.011353] demo-felhom systemd[1]: Starting systemd-rfkill.service - Load/Save RF Kill Switch Status...
[ 4.013369] demo-felhom kernel: Bluetooth: hci0: RTL: btrtl_initialize: key id 0
[ 4.013474] demo-felhom kernel: Bluetooth: hci0: RTL: loading rtl_bt/rtl8852bu_fw.bin
[ 4.018168] demo-felhom systemd[1]: Reached target bluetooth.target - Bluetooth Support.
[ 4.058030] demo-felhom kernel: Bluetooth: hci0: RTL: loading rtl_bt/rtl8852bu_config.bin
[ 4.060122] demo-felhom kernel: Bluetooth: hci0: RTL: cfg_sz 6, total sz 62539
[ 4.069224] demo-felhom systemd[1]: Started systemd-rfkill.service - Load/Save RF Kill Switch Status.
[ 4.072229] demo-felhom systemd[1]: Finished systemd-sysctl.service - Apply Kernel Variables.
[ 4.094018] demo-felhom kernel: usb 1-6: Warning! Unlikely big volume range (=511), cval->res is probably wrong.
[ 4.094444] demo-felhom kernel: usb 1-6: [5] FU [Mic Capture Volume] ch = 1, val = -8192/-1/16
[ 4.100058] demo-felhom kernel: usb 1-6: Warning! Unlikely big volume range (=896), cval->res is probably wrong.
[ 4.100362] demo-felhom kernel: usb 1-6: [6] FU [PCM Playback Volume] ch = 2, val = -16384/-2048/16
[ 4.100548] demo-felhom kernel: usbcore: registered new interface driver snd-usb-audio
[ 4.106727] demo-felhom systemd[1]: Reached target sound.target - Sound Card.
[ 4.116015] demo-felhom kernel: rtw89_8852be 0000:02:00.0: loaded firmware rtw89/rtw8852b_fw-1.bin
[ 4.116308] demo-felhom kernel: rtw89_8852be 0000:02:00.0: enabling device (0000 -> 0003)
[ 4.124022] demo-felhom kernel: rtw89_8852be 0000:02:00.0: Firmware version 0.29.29.15 (6fb3ec41), cmd version 0, type 5
[ 4.124318] demo-felhom kernel: rtw89_8852be 0000:02:00.0: Firmware version 0.29.29.15 (6fb3ec41), cmd version 0, type 3
[ 4.137907] demo-felhom systemd[1]: Mounting boot-efi.mount - /boot/efi...
[ 4.139530] demo-felhom systemd[1]: mnt-hdd_1.mount: Directory /mnt/hdd_1 to mount over is not empty, mounting anyway.
[ 4.141158] demo-felhom systemd[1]: Mounting mnt-hdd_1.mount - Felhom storage mount 47a3361a-91e0-4831-a69d-27f540ed3f48...
[ 4.142916] demo-felhom systemd[1]: Mounting tmp.mount - Temporary Directory /tmp...
[ 4.267728] demo-felhom systemd[1]: Mounted tmp.mount - Temporary Directory /tmp.
[ 4.268510] demo-felhom systemd[1]: Mounted boot-efi.mount - /boot/efi.
[ 4.421015] demo-felhom kernel: Bluetooth: hci0: RTL: fw version 0x127cfd78
[ 4.475042] demo-felhom kernel: rtw89_8852be 0000:02:00.0: chip info CID: 0, CV: 1, AID: 0, ACV: 1, RFE: 5
[ 4.476021] demo-felhom kernel: rtw89_8852be 0000:02:00.0: rfkill hardware state changed to enable
[ 4.494046] demo-felhom kernel: intel_tcc_cooling: TCC Offset locked
[ 4.499095] demo-felhom kernel: rtw89_8852be 0000:02:00.0 wlp2s0: renamed from wlan0
[ 4.499354] demo-felhom kernel: i915 0000:00:02.0: [drm] Found alderlake_p/alderlake_n (device ID 46d1) integrated display version 13.00 stepping D0
[ 4.499561] demo-felhom kernel: i915 0000:00:02.0: [drm] VT-d active for gfx access
[ 4.499846] demo-felhom kernel: i915 0000:00:02.0: vgaarb: deactivate vga console
[ 4.500020] demo-felhom kernel: i915 0000:00:02.0: [drm] Using Transparent Hugepages
[ 4.501210] demo-felhom kernel: i915 0000:00:02.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[ 4.505030] demo-felhom kernel: i915 0000:00:02.0: [drm] Finished loading DMC firmware i915/adlp_dmc.bin (v2.20)
[ 4.505306] demo-felhom kernel: Bluetooth: hci0: AOSP extensions version v1.00
[ 4.515016] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: GuC firmware i915/tgl_guc_70.bin version 70.49.4
[ 4.515287] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: HuC firmware i915/tgl_huc.bin version 7.9.3
[ 4.517022] demo-felhom kernel: intel_rapl_msr: PL4 support detected.
[ 4.517113] demo-felhom kernel: intel_rapl_common: Found RAPL domain package
[ 4.517134] demo-felhom kernel: intel_rapl_common: Found RAPL domain core
[ 4.517152] demo-felhom kernel: intel_rapl_common: Found RAPL domain uncore
[ 4.520023] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: HuC: authenticated for all workloads
[ 4.520252] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: GUC: submission enabled
[ 4.520417] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: GUC: SLPC enabled
[ 4.520574] demo-felhom kernel: i915 0000:00:02.0: [drm] GT0: GUC: RC enabled
[ 4.522017] demo-felhom kernel: i915 0000:00:02.0: [drm] Protected Xe Path (PXP) protected content support initialized
[ 4.522233] demo-felhom kernel: i915 0000:00:02.0: [drm] Registered 3 planes with drm panic
[ 4.530018] demo-felhom kernel: [drm] Initialized i915 1.6.0 for 0000:00:02.0 on minor 0
[ 4.531012] demo-felhom kernel: ACPI: video: Video Device [GFX0] (multi-head: yes rom: no post: no)
[ 4.531087] demo-felhom kernel: input: Video Bus as /devices/pci0000:00/acpi.video_bus.0/input/input4
[ 4.536037] demo-felhom kernel: EXT4-fs (sdb): mounted filesystem 47a3361a-91e0-4831-a69d-27f540ed3f48 r/w with ordered data mode. Quota mode: none.
[ 4.537227] demo-felhom systemd[1]: Mounted mnt-hdd_1.mount - Felhom storage mount 47a3361a-91e0-4831-a69d-27f540ed3f48.
[ 4.545015] demo-felhom kernel: i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[ 4.569019] demo-felhom kernel: i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[ 4.581021] demo-felhom kernel: i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[ 4.610025] demo-felhom kernel: snd_hda_intel 0000:00:1f.3: enabling device (0000 -> 0002)
[ 4.610349] demo-felhom kernel: snd_hda_intel 0000:00:1f.3: bound 0000:00:02.0 (ops intel_audio_component_bind_ops [i915])
[ 4.917022] demo-felhom kernel: input: HDA Intel PCH HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:1f.3/sound/card1/input5
[ 4.917152] demo-felhom kernel: input: HDA Intel PCH HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:1f.3/sound/card1/input6
[ 4.917202] demo-felhom kernel: input: HDA Intel PCH HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:1f.3/sound/card1/input7
[ 4.917241] demo-felhom kernel: input: HDA Intel PCH HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:1f.3/sound/card1/input8
[ 4.940127] demo-felhom systemd[1]: Finished ifupdown2-pre.service - Helper to synchronize boot up for ifupdown.
[ 4.943663] demo-felhom systemd[1]: Finished systemd-udev-settle.service - Wait for udev To Complete Device Initialization.
[ 4.943871] demo-felhom systemd[1]: zfs-import-cache.service - Import ZFS pools by cache file skipped, unmet condition check ConditionFileNotEmpty=/etc/zfs/zpool.cache
[ 4.943900] demo-felhom systemd[1]: Reached target zfs-import.target - ZFS pool import target.
[ 4.945492] demo-felhom systemd[1]: Starting zfs-mount.service - Mount ZFS filesystems...
[ 4.946429] demo-felhom systemd[1]: Starting zfs-volume-wait.service - Wait for ZFS Volume (zvol) links in /dev...
[ 4.966316] demo-felhom systemd[1]: Finished zfs-mount.service - Mount ZFS filesystems.
[ 4.967027] demo-felhom systemd[1]: Finished zfs-volume-wait.service - Wait for ZFS Volume (zvol) links in /dev.
[ 4.968549] demo-felhom zvol_wait[531]: No zvols found, nothing to do.
[ 4.967458] demo-felhom systemd[1]: Reached target local-fs.target - Local File Systems.
[ 4.967518] demo-felhom systemd[1]: Reached target zfs-volumes.target - ZFS volumes are ready.
[ 4.969218] demo-felhom systemd[1]: Listening on systemd-sysext.socket - System Extension Image Management.
[ 4.970224] demo-felhom systemd[1]: Starting apparmor.service - Load AppArmor profiles...
[ 4.971086] demo-felhom systemd[1]: Starting console-setup.service - Set console font and keymap...
[ 4.972029] demo-felhom systemd[1]: Starting felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit)...
[ 4.973401] demo-felhom systemd[1]: Starting networking.service - Network initialization...
[ 4.977223] demo-felhom systemd[1]: Starting pvebanner.service - Proxmox VE Login Banner...
[ 4.978444] demo-felhom systemd[1]: Starting pvefw-logger.service - Proxmox VE firewall logger...
[ 4.979795] demo-felhom systemd[1]: Starting pvenetcommit.service - Commit Proxmox VE network changes...
[ 4.983183] demo-felhom systemd[1]: Starting systemd-binfmt.service - Set Up Additional Binary Formats...
[ 4.985507] demo-felhom systemd[1]: Starting systemd-tmpfiles-setup.service - Create System Files and Directories...
[ 4.986954] demo-felhom systemd[1]: Finished console-setup.service - Set console font and keymap.
[ 4.998578] demo-felhom systemd[1]: proc-sys-fs-binfmt_misc.automount: Got automount request for /proc/sys/fs/binfmt_misc, triggered by 547 (systemd-binfmt)
[ 4.999914] demo-felhom systemd[1]: Mounting proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System...
[ 5.000113] demo-felhom systemd-tmpfiles[550]: /etc/tmpfiles.d/felhom-privsep.conf:15: Duplicate line for path "/run/sshd", ignoring.
[ 5.000258] demo-felhom systemd-tmpfiles[550]: /usr/lib/tmpfiles.d/legacy.conf:14: Duplicate line for path "/run/lock", ignoring.
[ 5.001716] demo-felhom systemd[1]: Finished pvenetcommit.service - Commit Proxmox VE network changes.
[ 5.003282] demo-felhom apparmor.systemd[538]: Restarting AppArmor
[ 5.003282] demo-felhom apparmor.systemd[538]: Reloading AppArmor profiles
[ 5.021348] demo-felhom networking[541]: networking: Configuring network interfaces
[ 5.022383] demo-felhom systemd[1]: Mounted proc-sys-fs-binfmt_misc.mount - Arbitrary Executable File Formats File System.
[ 5.025891] demo-felhom systemd[1]: Finished systemd-binfmt.service - Set Up Additional Binary Formats.
[ 5.048314] demo-felhom pvefw-logger[573]: starting pvefw logger
[ 5.048548] demo-felhom systemd[1]: Started pvefw-logger.service - Proxmox VE firewall logger.
[ 5.049951] demo-felhom systemd[1]: Finished systemd-tmpfiles-setup.service - Create System Files and Directories.
[ 5.050310] demo-felhom systemd[1]: ldconfig.service - Rebuild Dynamic Linker Cache skipped, no trigger condition checks were met.
[ 5.055441] demo-felhom systemd[1]: Starting rpcbind.service - RPC bind portmap service...
[ 5.055478] demo-felhom systemd[1]: systemd-firstboot.service - First Boot Wizard skipped, unmet condition check ConditionFirstBoot=yes
[ 5.055507] demo-felhom systemd[1]: first-boot-complete.target - First Boot Complete skipped, unmet condition check ConditionFirstBoot=yes
[ 5.055569] demo-felhom systemd[1]: systemd-journal-catalog-update.service - Rebuild Journal Catalog skipped, unmet condition check ConditionNeedsUpdate=/var
[ 5.055622] demo-felhom systemd[1]: systemd-machine-id-commit.service - Save Transient machine-id to Disk skipped, unmet condition check ConditionPathIsMountPoint=/etc/machine-id
[ 5.055680] demo-felhom systemd[1]: systemd-update-done.service - Update is Completed skipped, no trigger condition checks were met.
[ 5.069037] demo-felhom kernel: audit: type=1400 audit(1794112523.118:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name=4D6F6E676F444220436F6D70617373 pid=572 comm="apparmor_
[ 5.069130] demo-felhom kernel: audit: type=1400 audit(1794112523.118:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="QtWebEngineProcess" pid=574 comm="apparmor_parser"
[ 5.069157] demo-felhom kernel: audit: type=1400 audit(1794112523.118:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="1password" pid=570 comm="apparmor_parser"
[ 5.069183] demo-felhom kernel: audit: type=1400 audit(1794112523.119:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="Discord" pid=571 comm="apparmor_parser"
[ 5.072026] demo-felhom kernel: audit: type=1400 audit(1794112523.123:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="brave" pid=581 comm="apparmor_parser"
[ 5.073026] demo-felhom kernel: audit: type=1400 audit(1794112523.124:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="busybox" pid=583 comm="apparmor_parser"
[ 5.074325] demo-felhom kernel: audit: type=1400 audit(1794112523.125:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="balena-etcher" pid=580 comm="apparmor_parser"
[ 5.074387] demo-felhom kernel: audit: type=1400 audit(1794112523.125:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="buildah" pid=582 comm="apparmor_parser"
[ 5.074411] demo-felhom kernel: audit: type=1400 audit(1794112523.125:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="cam" pid=584 comm="apparmor_parser"
[ 5.075909] demo-felhom systemd[1]: Started rpcbind.service - RPC bind portmap service.
[ 5.076068] demo-felhom systemd[1]: Reached target rpcbind.target - RPC Port Mapper.
[ 5.119035] demo-felhom kernel: kauditd_printk_skb: 40 callbacks suppressed
[ 5.119141] demo-felhom kernel: audit: type=1400 audit(1794112523.170:51): apparmor="STATUS" operation="profile_load" profile="unconfined" name="opera" pid=624 comm="apparmor_parser"
[ 5.120172] demo-felhom kernel: audit: type=1400 audit(1794112523.171:52): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default" pid=610 comm="apparmor_parser
[ 5.120257] demo-felhom kernel: audit: type=1400 audit(1794112523.171:53): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-cgns" pid=610 comm="apparmor_p
[ 5.120282] demo-felhom kernel: audit: type=1400 audit(1794112523.171:54): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-mounting" pid=610 comm="a
[ 5.120308] demo-felhom kernel: audit: type=1400 audit(1794112523.171:55): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxc-container-default-with-nesting" pid=610 comm="ap
[ 5.120329] demo-felhom kernel: audit: type=1400 audit(1794112523.172:56): apparmor="STATUS" operation="profile_load" profile="unconfined" name="polypane" pid=627 comm="apparmor_parser"
[ 5.121021] demo-felhom kernel: audit: type=1400 audit(1794112523.172:57): apparmor="STATUS" operation="profile_load" profile="unconfined" name="pageedit" pid=625 comm="apparmor_parser"
[ 5.124025] demo-felhom kernel: audit: type=1400 audit(1794112523.175:58): apparmor="STATUS" operation="profile_load" profile="unconfined" name="privacybrowser" pid=628 comm="apparmor_parser"
[ 5.124103] demo-felhom kernel: audit: type=1400 audit(1794112523.176:59): apparmor="STATUS" operation="profile_load" profile="unconfined" name="qcam" pid=630 comm="apparmor_parser"
[ 5.126023] demo-felhom kernel: audit: type=1400 audit(1794112523.177:60): apparmor="STATUS" operation="profile_load" profile="unconfined" name="pve-container-mounthotplug" pid=629 comm="apparmor_p
[ 5.133470] demo-felhom felhom-crash-guard[540]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10
[ 5.138735] demo-felhom systemd[1]: Mounting run-rpc_pipefs.mount - RPC Pipe File System...
[ 5.153807] demo-felhom felhom-crash-guard[665]: crash-guard: boot first=False unclean=False in-window=0 tripped=False kernel.panic=10
[ 5.162616] demo-felhom systemd[1]: Finished felhom-crash-guard.service - Felhom crash guard (restart after a kernel crash, with a limit).
[ 5.199068] demo-felhom kernel: RPC: Registered named UNIX socket transport module.
[ 5.199201] demo-felhom kernel: RPC: Registered udp transport module.
[ 5.199231] demo-felhom kernel: RPC: Registered tcp transport module.
[ 5.199252] demo-felhom kernel: RPC: Registered tcp-with-tls transport module.
[ 5.199278] demo-felhom kernel: RPC: Registered tcp NFSv4.1 backchannel transport module.
[ 5.201359] demo-felhom systemd[1]: Mounted run-rpc_pipefs.mount - RPC Pipe File System.
[ 5.201831] demo-felhom systemd[1]: Reached target rpc_pipefs.target.
[ 5.204008] demo-felhom systemd[1]: Starting nfs-blkmap.service - pNFS block layout mapping daemon...
[ 5.204272] demo-felhom systemd[1]: rpc-gssd.service - RPC security service for NFS client and server skipped, unmet condition check ConditionPathExists=/etc/krb5.keytab
[ 5.204309] demo-felhom systemd[1]: Reached target nfs-client.target - NFS client services.
[ 5.207340] demo-felhom systemd[1]: Finished apparmor.service - Load AppArmor profiles.
[ 5.207747] demo-felhom systemd[1]: Reached target sysinit.target - System Initialization.
## chrony lines per boot (all boots)
-- boot -5
-- boot -4
-- boot -3
Oct 16 05:09:25 demo-felhom systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Oct 16 05:09:25 demo-felhom chronyd[692]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Oct 16 05:09:25 demo-felhom chronyd[692]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Oct 16 05:09:25 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE
Oct 16 05:09:25 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'.
Oct 16 05:09:25 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.
-- boot -2
Oct 21 05:23:50 demo-felhom systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Oct 21 05:23:50 demo-felhom chronyd[676]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Oct 21 05:23:50 demo-felhom chronyd[676]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Oct 21 05:23:50 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE
Oct 21 05:23:50 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'.
Oct 21 05:23:50 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.
-- boot -1
Oct 29 04:48:13 demo-felhom systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Oct 29 04:48:14 demo-felhom chronyd[682]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Oct 29 04:48:14 demo-felhom chronyd[682]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Oct 29 04:48:14 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE
Oct 29 04:48:14 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'.
Oct 29 04:48:14 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.
-- boot 0
Nov 08 05:35:23 demo-felhom systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE
Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'.
Nov 08 05:35:23 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.
## sandbox env of pid1
total 0
drwxr-xr-x 21 root root 560 Nov 11 14:06 .
drwxr-xr-x 36 root root 1520 Nov 11 14:08 ..
drwxr-xr-x 2 root root 40 Nov 8 05:35 ask-password
drwx------ 2 root root 60 Nov 8 21:45 ask-password-block
drwxr-xr-x 6 root root 260 Nov 10 15:50 generator
dr-xr-xr-x 3 root root 160 Nov 8 05:35 inaccessible
drwxr-xr-x 2 root root 40 Nov 8 05:35 incoming
drwxr-xr-x 2 root root 40 Nov 9 05:33 inhibit
srw-rw-rw- 1 root root 0 Nov 8 05:35 io.systemd.Credentials
srw-rw-rw- 1 root root 0 Nov 8 05:35 io.systemd.Hostname
srw-rw-rw- 1 root root 0 Nov 8 05:35 io.systemd.ManagedOOM
srw------- 1 root root 0 Nov 8 05:35 io.systemd.sysext
drwxr-xr-x 3 root root 200 Nov 8 05:35 journal
drwxr-xr-x 2 root root 40 Nov 8 05:35 machines
dr-xr-xr-x 2 root root 40 Nov 8 05:35 mount-rootfs
drwxr-xr-x 4 systemd-network systemd-network 80 Nov 8 05:35 netif
srwxrwxrwx 1 root root 0 Nov 8 05:35 notify
srwx------ 1 root root 0 Nov 8 05:35 private
drwxr-xr-x 6 root root 120 Nov 11 14:07 propagate
drwxr-xr-x 2 root root 60 Nov 8 05:35 seats
drwxr-xr-x 2 root root 120 Nov 11 14:08 sessions
drwxr-xr-x 2 root root 40 Nov 8 05:35 shutdown
drwxr-xr-x 2 root root 40 Nov 8 05:35 system
-r--r--r-- 1 root root 0 Nov 10 15:50 systemd-units-load
drwxr-xr-x 2 root root 60 Nov 11 14:08 transient
drwxr-xr-x 2 root root 1880 Nov 11 14:09 units
drwxr-xr-x 2 root root 60 Nov 8 05:35 userdb
drwxr-xr-x 2 root root 60 Nov 11 14:08 users
## tailscale
active
## rtc
ls: cannot access '/dev/ptp*': No such file or directory
ls: cannot access '/dev/pps*': No such file or directory
/dev/rtc
/dev/rtc0
## cpu
4
model name : Intel(R) N100
@@ -0,0 +1,250 @@
#### host demo-felhom 2026-11-11T13:06:53Z local 14:06:53+0100
## versions
pve-manager/9.2.26/41a0e2e09b9e6f9d (running kernel: 7.0.14-28-pve)
ii chrony 4.8-4~bpo13+2
ii systemd 257.13-1~deb13u1
un systemd-timesyncd <none>
7.0.14-28-pve
## unit state
Type=notify
PIDFile=/run/chrony/chronyd.pid
NotifyAccess=main
Result=exit-code
NRestarts=0
ExecMainCode=1
ExecMainStatus=1
ActiveState=failed
SubState=failed
FragmentPath=/usr/lib/systemd/system/chrony.service
DropInPaths=
UnitFileState=enabled
ActiveEnterTimestamp=
InactiveEnterTimestamp=Sun 2026-11-08 05:35:23 CET
## unit cat
# /usr/lib/systemd/system/chrony.service
[Unit]
Description=chrony, an NTP client/server
Documentation=man:chronyd(8) man:chronyc(1) man:chrony.conf(5)
Conflicts=openntpd.service ntp.service ntpsec.service
ConditionVirtualization=|!container
ConditionVirtualization=|wsl
ConditionCapability=CAP_SYS_TIME
[Service]
Type=notify
PIDFile=/run/chrony/chronyd.pid
EnvironmentFile=-/etc/default/chrony
User=_chrony
# Daemon is started as root, but still sandboxed
ExecStart=!/usr/sbin/chronyd -n $DAEMON_OPTS
CapabilityBoundingSet=~CAP_AUDIT_CONTROL CAP_AUDIT_READ CAP_AUDIT_WRITE
CapabilityBoundingSet=~CAP_BLOCK_SUSPEND CAP_KILL CAP_LEASE CAP_LINUX_IMMUTABLE
CapabilityBoundingSet=~CAP_MAC_ADMIN CAP_MAC_OVERRIDE CAP_MKNOD CAP_SYS_ADMIN
CapabilityBoundingSet=~CAP_SYS_BOOT CAP_SYS_CHROOT CAP_SYS_MODULE CAP_SYS_PACCT
CapabilityBoundingSet=~CAP_SYS_PTRACE CAP_SYS_RAWIO CAP_SYS_TTY_CONFIG CAP_WAKE_ALARM
DeviceAllow=char-pps rw
DeviceAllow=char-ptp rw
DeviceAllow=char-rtc rw
DevicePolicy=closed
LockPersonality=yes
MemoryDenyWriteExecute=yes
NoNewPrivileges=yes
PrivateTmp=yes
ProcSubset=pid
ProtectControlGroups=yes
ProtectHome=yes
ProtectHostname=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectKernelTunables=yes
ProtectProc=invisible
ProtectSystem=strict
# Used for gps refclocks
ReadWritePaths=/run
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
RestrictNamespaces=yes
RestrictSUIDSGID=yes
SystemCallArchitectures=native
SystemCallFilter=~@cpu-emulation @debug @module @mount @obsolete @raw-io @reboot @swap
ConfigurationDirectory=chrony
RuntimeDirectory=chrony
RuntimeDirectoryMode=0700
# See dumpdir in chrony.conf(5)
RuntimeDirectoryPreserve=restart
StateDirectory=chrony
StateDirectoryMode=0750
LogsDirectory=chrony
LogsDirectoryMode=0750
# Adjust restrictions for /usr/sbin/sendmail (mailonchange directive)
NoNewPrivileges=no
ReadWritePaths=-/var/spool
RestrictAddressFamilies=AF_NETLINK
[Install]
Alias=chronyd.service
WantedBy=multi-user.target
## files
ls: cannot access '/etc/systemd/system/chrony.service.d': No such file or directory
ls: cannot access '/etc/systemd/system/chronyd.service.d': No such file or directory
ls: cannot access '/etc/systemd/system/chrony.service': No such file or directory
lrwxrwxrwx 1 root root 38 May 21 13:32 /etc/systemd/system/chronyd.service -> /usr/lib/systemd/system/chrony.service
-rw-r--r-- 1 root root 1929 Jul 1 22:28 /lib/systemd/system/chrony.service
-rw-r--r-- 1 root root 1929 Jul 1 22:28 /usr/lib/systemd/system/chrony.service
## dpkg verify chrony
rc=0
## conf
/etc/chrony:
total 24
drwxr-xr-x 4 root root 4096 May 21 13:32 .
drwxr-xr-x 102 root root 4096 Nov 8 05:32 ..
-rw-r--r-- 1 root root 1450 Feb 16 2026 chrony.conf
-rw-r----- 1 root _chrony 481 Feb 16 2026 chrony.keys
drwxr-xr-x 2 root root 4096 Oct 7 14:15 conf.d
drwxr-xr-x 2 root root 4096 Oct 7 14:15 sources.d
/etc/chrony/conf.d:
total 12
drwxr-xr-x 2 root root 4096 Oct 7 14:15 .
drwxr-xr-x 4 root root 4096 May 21 13:32 ..
-rw-r--r-- 1 root root 422 Feb 16 2026 README
/etc/chrony/sources.d:
total 12
drwxr-xr-x 2 root root 4096 Oct 7 14:15 .
drwxr-xr-x 4 root root 4096 May 21 13:32 ..
-rw-r--r-- 1 root root 498 Feb 16 2026 README
pool 2.debian.pool.ntp.org iburst
sourcedir /run/chrony-dhcp
sourcedir /etc/chrony/sources.d
keyfile /etc/chrony/chrony.keys
driftfile /var/lib/chrony/chrony.drift
ntsdumpdir /var/lib/chrony
logdir /var/log/chrony
maxupdateskew 100.0
rtcsync
makestep 1 3
leapseclist /usr/share/zoneinfo/leap-seconds.list
confdir /etc/chrony/conf.d
-- /etc/chrony/conf.d/README
Files found under the /etc/chrony/conf.d directory with the .conf suffix are
parsed in the lexicographical order of the file names when chronyd starts up.
This enables a fragmented configuration of chronyd.
Although those files can contain any directives listed in chrony.conf(5),
it would be wiser to add NTP sources in the /etc/chrony/sources.d
directory. Please read /etc/chrony/sources.d/README for more information.
-- /etc/chrony/sources.d/README
Only NTP sources can be specified in the /etc/chrony/sources.d directory.
Files in this directory must end with the ".sources" suffix. They can only
contain the "peer", "pool" and "server" directives and must have all
lines terminated by a trailing newline.
There is no need to restart chronyd for these time sources to be usable,
running 'chronyc reload sources' is sufficient.
Example:
## default
DAEMON_OPTS="-F 1"
## apparmor
/usr/sbin/chronyd
-rw-r--r-- 1 root root 0 May 21 13:32 /etc/apparmor.d/local/usr.sbin.chronyd
-rw-r--r-- 1 root root 2837 Jul 1 22:28 /etc/apparmor.d/usr.sbin.chronyd
/etc/apparmor.d/disable:
total 8
drwxr-xr-x 2 root root 4096 Jul 21 2025 .
drwxr-xr-x 9 root root 4096 Nov 8 05:32 ..
## apparmor profile notify lines
81: # Allow systemd Type=notify using sd_notify's $NOTIFY_SOCKET
82: @{run}/systemd/notify w,
## kernel audit lines for chronyd (last 20)
## journal chrony this boot
Nov 08 05:35:23 demo-felhom systemd[1]: Starting chrony.service - chrony, an NTP client/server...
Nov 08 05:35:23 demo-felhom chronyd[690]: chronyd version 4.8 starting (+CMDMON +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +NTS +SECHASH +IPV6 -DEBUG)
Nov 08 05:35:23 demo-felhom chronyd[690]: Loaded 0 symmetric keys
Nov 08 05:35:23 demo-felhom chronyd[690]: Using leap second list /usr/share/zoneinfo/leap-seconds.list
Nov 08 05:35:23 demo-felhom chronyd[690]: Frequency -2.666 +/- 0.016 ppm read from /var/lib/chrony/chrony.drift
Nov 08 05:35:23 demo-felhom chronyd[690]: Loaded seccomp filter (level 1)
Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Nov 08 05:35:23 demo-felhom chronyd[690]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Main process exited, code=exited, status=1/FAILURE
Nov 08 05:35:23 demo-felhom systemd[1]: chrony.service: Failed with result 'exit-code'.
Nov 08 05:35:23 demo-felhom systemd[1]: Failed to start chrony.service - chrony, an NTP client/server.
## first failure ever in the journal
Oct 16 05:09:25 demo-felhom chronyd[692]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Oct 16 05:09:25 demo-felhom chronyd[692]: Fatal error : Could not send notification to $NOTIFY_SOCKET
Oct 21 05:23:50 demo-felhom chronyd[676]: Fatal error : Could not send notification to $NOTIFY_SOCKET
count: 8
## oldest journal line
Oct 04 21:43:33 demo-felhom systemd[3282788]: Queued start job for default targe
## last good start in the journal
## boots
-5 46081f567f384ab88591d4e604555a48 Sun 2026-10-04 21:43:33 CEST Sun 2026-10-04 21:43:46 CEST
-4 e82176ae7cf644518507468fc609eaa9 Wed 2026-10-14 15:43:07 CEST Fri 2026-10-16 05:09:06 CEST
-3 ddd3d7305e244b2a9eafb5cf229476f1 Fri 2026-10-16 05:09:23 CEST Wed 2026-10-21 05:23:34 CEST
-2 56db71938b304a57bdedf30f89860e1f Wed 2026-10-21 05:23:48 CEST Thu 2026-10-29 04:47:58 CET
-1 4f6d8b388e224b8ca8ed45e06a2f96b9 Thu 2026-10-29 04:48:11 CET Sun 2026-11-08 05:35:13 CET
0 72899d2625244e069e8e8099775d1e51 Sun 2026-11-08 05:35:21 CET Wed 2026-11-11 14:06:53 CET
## what starts with it
chrony.service
● ├─-.mount
● ├─system.slice
● ├─systemd-journald.socket
● ├─systemd-remount-fs.service
● ├─systemd-tmpfiles-setup.service
● ├─tmp.mount
● ├─basic.target
● │ ├─-.mount
○ │ ├─systemd-pcrphase-sysinit.service
● │ ├─tmp.mount
● │ ├─paths.target
● │ │ ├─systemd-ask-password-console.path
● │ │ └─systemd-ask-password-wall.path
● │ ├─slices.target
● │ │ ├─-.slice
● │ │ ├─system-xfs_scrub.slice
● │ │ ├─system.slice
● │ │ └─user.slice
● │ ├─sockets.target
● │ │ ├─dbus.socket
● │ │ ├─iscsid.socket
● │ │ ├─netavark-dhcp-proxy.socket
● │ │ ├─rrdcached.socket
● │ │ ├─sshd-unix-local.socket
The time when unit became active or started is printed after the "@" character.
The time the unit took to start is printed after the "+" character.
chrony.service +256ms
└─basic.target @2.481s
└─sockets.target @2.481s
└─systemd-hostnamed.socket @2.481s
└─sysinit.target @2.478s
└─apparmor.service @2.241s +237ms
└─local-fs.target @2.238s
└─mnt-felhom\x2ddrives-hdd_1.mount @11.537s
└─mnt-felhom\x2ddrives.mount @2.549s
## other time services
inactive inactive inactive inactive
Local time: Wed 2026-11-11 14:06:55 CET
Universal time: Wed 2026-11-11 13:06:55 UTC
RTC time: Wed 2026-11-11 13:06:52
Time zone: Europe/Budapest (CET, +0100)
System clock synchronized: no
NTP service: inactive
RTC in local TZ: no
## chronyc
506 Cannot talk to daemon
506 Cannot talk to daemon
## dpkg history chrony/apparmor/systemd
2026-07-18 16:14:58 install apparmor:amd64 <none> 4.1.1-pmx1
2026-07-18 16:15:52 configure apparmor:amd64 4.1.1-pmx1 <none>
2026-10-07 14:15:40 upgrade chrony:amd64 4.6.1-3+deb13u1 4.8-4~bpo13+2
2026-10-07 14:15:46 configure chrony:amd64 4.8-4~bpo13+2 <none>
## felhom files that name chrony
## ps
## run dir
ls: cannot access '/run/chrony': No such file or directory
srwxrwxrwx 1 root root 0 Nov 8 05:35 /run/systemd/notify
## uptime
2026-11-08 05:35:18
@@ -0,0 +1,25 @@
#### demo-felhom 2026-11-11T13:16:03Z
## fresh compile of the on-disk profile to stdout (no load, cache not read, not written)
83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1 -
## loaded
dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647
## cache
83417e83b538721f6931dc14bdb61c62e4132dc28ca33a1898a707be5112e7c1 /var/cache/apparmor/ae1a4050.0/usr.sbin.chronyd
dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 /var/cache/apparmor/f6da0254.0/usr.sbin.chronyd
drwx------ 2 root root 4096 2026-10-07 14:16:03.051992948 +0200 /var/cache/apparmor/ae1a4050.0/
-rw-r--r-- 1 root root 3177 2026-07-18 18:17:02.251100923 +0200 /var/cache/apparmor/ae1a4050.0/.features
drwx------ 2 root root 4096 2026-11-08 05:32:24.065142783 +0100 /var/cache/apparmor/f6da0254.0/
-rw-r--r-- 1 root root 3219 2026-10-07 12:11:32.040268492 +0200 /var/cache/apparmor/f6da0254.0/.features
## state
failed
chrony 4.8-4~bpo13+2
7.0.14-28-pve
1794402963.132843793
## chrony fatal lines
8
## stale check, every cached profile: cache differs from a fresh compile
current dir: /var/cache/apparmor/f6da0254.0/
STALE: lxc-containers
STALE: usr.sbin.chronyd
checked=106 stale=2
@@ -0,0 +1,33 @@
#### felhom 2026-11-11T13:17:03Z
## fresh compile of the on-disk profile to stdout (no load, cache not read, not written)
dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 -
## loaded
dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647
## cache
dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 /var/cache/apparmor/ae1a4050.0/usr.sbin.chronyd
dbf67f16ac2525e039d6a94230fe0f5d5bcfb44a89a1fc441ac3700737d3b647 /var/cache/apparmor/f6da0254.0/usr.sbin.chronyd
drwx------ 2 root root 4096 2026-10-04 21:37:58.088000000 +0200 /var/cache/apparmor/ae1a4050.0/
-rw-r--r-- 1 root root 3177 2026-10-04 21:37:57.515000000 +0200 /var/cache/apparmor/ae1a4050.0/.features
drwx------ 2 root root 4096 2026-10-07 10:41:12.158575707 +0200 /var/cache/apparmor/f6da0254.0/
-rw-r--r-- 1 root root 3219 2026-10-07 10:41:11.594575707 +0200 /var/cache/apparmor/f6da0254.0/.features
## state
active
chrony 4.6.1-3+deb13u1
7.0.14-22-pve
System time : 0.000177019 seconds slow of NTP time
Leap status : Normal
1794403023.309780841
## chrony fatal lines
0
## stale check, every cached profile: cache differs from a fresh compile
current dir: /var/cache/apparmor/f6da0254.0/
STALE: lxc-containers
checked=106 stale=1
2026-10-04 19:31:38 install apparmor:amd64 <none> 4.1.1-pmx1
2026-10-04 19:32:11 install proxmox-kernel-7.0.2-6-pve-signed:amd64 <none> 7.0.2-6
2026-10-07 10:02:07 install proxmox-kernel-7.0.14-20-pve-signed:amd64 <none> 7.0.14-20
2026-10-07 17:14:55 install proxmox-kernel-7.0.14-22-pve-signed:amd64 <none> 7.0.14-22
NRestarts=0
ExecMainStartTimestamp=Tue 2026-11-10 15:43:32 CET
ActiveState=active
@@ -0,0 +1,26 @@
# READ ONLY. Part 3: the time service on a Proxmox host.
echo "#### host $(hostname) $(date -u +%FT%TZ) local $(date +%T%z)"
echo "## versions"; pveversion; dpkg -l chrony systemd systemd-timesyncd 2>/dev/null | grep '^[a-z][a-z] ' | awk '{print $1,$2,$3}'
uname -r
echo "## unit state"; systemctl show chrony -p ActiveState -p SubState -p Result -p Type -p NotifyAccess -p ExecMainStatus -p ExecMainCode -p NRestarts -p ActiveEnterTimestamp -p InactiveEnterTimestamp -p FragmentPath -p DropInPaths -p UnitFileState -p PIDFile
echo "## unit cat"; systemctl cat chrony 2>&1
echo "## files"; ls -la /etc/systemd/system/chrony.service.d /etc/systemd/system/chronyd.service.d /usr/lib/systemd/system/chrony.service /lib/systemd/system/chrony.service /etc/systemd/system/chrony.service /etc/systemd/system/chronyd.service 2>&1
echo "## dpkg verify chrony"; dpkg -V chrony 2>&1; echo "rc=$?"
echo "## conf"; ls -la /etc/chrony /etc/chrony/conf.d /etc/chrony/sources.d 2>&1; grep -v '^\s*#' /etc/chrony/chrony.conf 2>&1 | grep -v '^$'; for f in /etc/chrony/conf.d/* /etc/chrony/sources.d/*; do [ -f "$f" ] && { echo "-- $f"; grep -v '^\s*#' "$f" | grep -v '^$'; }; done
echo "## default"; cat /etc/default/chrony 2>&1 | grep -v '^#'
echo "## apparmor"; aa-status 2>/dev/null | grep -i chrony; ls -la /etc/apparmor.d/usr.sbin.chronyd /etc/apparmor.d/local/usr.sbin.chronyd /etc/apparmor.d/disable 2>&1; cat /etc/apparmor.d/local/usr.sbin.chronyd 2>/dev/null | grep -v '^#'
echo "## apparmor profile notify lines"; grep -n 'notify\|/run/systemd' /etc/apparmor.d/usr.sbin.chronyd 2>&1
echo "## kernel audit lines for chronyd (last 20)"; journalctl -k --no-pager 2>/dev/null | grep -i 'chrony' | tail -20 | cut -c1-330
echo "## journal chrony this boot"; journalctl -u chrony -b --no-pager 2>&1 | tail -40 | cut -c1-300
echo "## first failure ever in the journal"; journalctl -u chrony --no-pager 2>/dev/null | grep -m3 'NOTIFY_SOCKET' | cut -c1-200; echo "count: $(journalctl -u chrony --no-pager 2>/dev/null | grep -c 'NOTIFY_SOCKET')"
echo "## oldest journal line"; journalctl --no-pager 2>/dev/null | head -1 | cut -c1-80
echo "## last good start in the journal"; journalctl -u chrony --no-pager 2>/dev/null | grep 'Started chrony\|Selected source' | tail -3 | cut -c1-200
echo "## boots"; journalctl --list-boots --no-pager 2>/dev/null | tail -6
echo "## what starts with it"; systemctl list-dependencies --after chrony 2>/dev/null | head -25; systemd-analyze critical-chain chrony.service 2>&1 | head -12
echo "## other time services"; systemctl is-active systemd-timesyncd ntp ntpsec openntpd 2>&1 | tr '\n' ' '; echo; timedatectl 2>&1
echo "## chronyc"; chronyc tracking 2>&1 | head -8; chronyc sources 2>&1 | tail -6
echo "## dpkg history chrony/apparmor/systemd"; zgrep -h ' chrony\| apparmor:\| systemd:' /var/log/dpkg.log* 2>/dev/null | grep ' upgrade \| install \|configure ' | sort | tail -25
echo "## felhom files that name chrony"; grep -rl chrony /etc/felhom* /usr/local/sbin /usr/local/bin /etc/systemd/system 2>/dev/null | head;
echo "## ps"; ps -eo pid,user,args | grep '[c]hronyd'
echo "## run dir"; ls -la /run/chrony /run/systemd/notify 2>&1
echo "## uptime"; uptime -s
@@ -0,0 +1,14 @@
# READ ONLY
echo "#### $(hostname) $(date -u +%FT%TZ)"
sysctl net.unix.max_dgram_qlen kernel.printk_ratelimit 2>&1
echo "## cmdline"; cat /proc/cmdline
echo "## apparmor denied this boot: $(journalctl -b --no-pager 2>/dev/null | grep -c 'apparmor=\"DENIED\"')"; journalctl -b --no-pager | grep 'apparmor="DENIED"' | head -5 | cut -c1-300
echo "## audit lines with chrony, all boots"; journalctl --no-pager 2>/dev/null | grep -i 'audit.*chrony\|chrony.*audit' | head -5 | cut -c1-300
echo "## the second around the chrony start, this boot"
T=$(journalctl -u chrony -b --no-pager -o short-iso | grep -m1 'Starting chrony' | awk '{print $1}'); echo "start: $T"
journalctl -b --no-pager -o short-monotonic --since "$(date -d "$T - 1 sec" '+%F %T')" --until "$(date -d "$T + 2 sec" '+%F %T')" | cut -c1-200 | head -150
echo "## chrony lines per boot (all boots)"; for b in -5 -4 -3 -2 -1 0; do echo "-- boot $b"; journalctl -u chrony -b $b --no-pager 2>/dev/null | grep -v '^-- ' | grep 'Starting\|Started\|Fatal\|Failed\|Stopp\|exit' | head -8 | cut -c1-160; done
echo "## sandbox env of pid1"; systemctl show -p NotifySocket -p DefaultEnvironment 2>&1 | head; ls -la /run/systemd/ | head -30
echo "## tailscale"; systemctl is-active tailscaled 2>&1
echo "## rtc"; ls /dev/rtc* /dev/ptp* /dev/pps* 2>&1
echo "## cpu"; nproc; grep -m1 'model name' /proc/cpuinfo
@@ -0,0 +1,18 @@
main() {
echo "#### $(hostname) $(date -u +%FT%TZ)"
echo "## fresh compile of the on-disk profile, to stdout only (no load, no cache write)"
apparmor_parser -S -K --skip-bad-cache-rebuild -o /dev/stdout /etc/apparmor.d/usr.sbin.chronyd 2>/dev/null | sha256sum
apparmor_parser -S -K -o /dev/stdout /etc/apparmor.d/usr.sbin.chronyd 2>&1 >/dev/null | head -3
echo "## loaded raw policy sha256"; cat /sys/kernel/security/apparmor/policy/profiles/usr.sbin.chronyd*/raw_sha256; echo
echo "## cache dirs"; ls -la --time-style=full-iso /var/cache/apparmor/
for d in /var/cache/apparmor/*/; do echo "$d features-sha=$(sha256sum < $d/.features | cut -c1-16) files=$(ls $d | wc -l)"; ls -l --time-style=full-iso $d/usr.sbin.chronyd $d/.features; done
echo "## kernel features now"; cat /sys/kernel/security/apparmor/features/policy/versions/* 2>/dev/null | tr '\n' ' '; echo; aa-features-abi -x 2>/dev/null | sha256sum | cut -c1-16
echo "## newest source file the profile depends on"; for f in /etc/apparmor.d/usr.sbin.chronyd /etc/apparmor.d/local/usr.sbin.chronyd /etc/apparmor.d/tunables/global /etc/apparmor.d/abstractions/base /etc/apparmor.d/abstractions/nameservice /etc/apparmor.d/abi/*; do ls -l --time-style=full-iso $f 2>/dev/null; done | awk '{print $6,$7,$9}' | sort | tail -6
echo "## files in /etc/apparmor.d changed 2026-07-18 or 2026-08-21"; find /etc/apparmor.d -newermt 2026-07-18 ! -newermt 2026-07-19 | head; find /etc/apparmor.d -newermt 2026-08-21 ! -newermt 2026-08-22 | head
echo "## versions"; dpkg-query -W apparmor chrony; apparmor_parser --version | head -1
echo "## dpkg log"; zgrep -h -E ' (upgrade|install) (apparmor|chrony|proxmox-kernel-[0-9.]+-[0-9]+-pve-signed):' /var/log/dpkg.log* | sort | tail -14
echo "## kernels booted"; journalctl --list-boots --no-pager | tail -8; for b in -4 -3 -2 -1 0; do journalctl -b $b -k --no-pager -o cat 2>/dev/null | head -1 | cut -c1-60; done
echo "## chronyd confinement now"; ps -eZ 2>/dev/null | grep -i chrony || echo "no chronyd process"
echo "## what installed chrony 4.8"; zgrep -h -B3 -A3 'chrony' /var/log/apt/history.log* 2>/dev/null | grep -E 'Start-Date|Commandline|Requested-By|chrony' | cut -c1-260 | tail -12
}
main </dev/null
@@ -0,0 +1,12 @@
main() {
echo "#### $(hostname) $(date -u +%FT%TZ)"
echo "## fresh compile of the on-disk profile to stdout (no load, cache not read, not written)"
apparmor_parser -S -K /etc/apparmor.d/usr.sbin.chronyd 2>/dev/null | sha256sum
echo "## loaded"; cat /sys/kernel/security/apparmor/policy/profiles/usr.sbin.chronyd*/raw_sha256; echo
echo "## cache"; sha256sum /var/cache/apparmor/*/usr.sbin.chronyd; ls -ld --time-style=full-iso /var/cache/apparmor/*/ /var/cache/apparmor/*/.features
echo "## state"; systemctl is-active chrony; dpkg-query -W chrony; uname -r; chronyc -n tracking 2>&1 | grep -E 'System time|Leap' ; date -u +%s.%N
echo "## chrony fatal lines"; journalctl -u chrony --no-pager -o short-iso 2>/dev/null | grep -c 'Could not send notification'
echo "## stale check, every cached profile: cache differs from a fresh compile"; n=0; bad=0; cur=$(ls -d /var/cache/apparmor/*/ | while read d; do [ "$(sha256sum < $d/.features)" = "$(aa-features-abi -x 2>/dev/null | sha256sum)" ] && echo $d; done | head -1); echo "current dir: $cur"
for f in /etc/apparmor.d/*; do [ -f "$f" ] || continue; b=$(basename $f); [ -f "$cur$b" ] || continue; n=$((n+1)); a=$(apparmor_parser -S -K "$f" 2>/dev/null | sha256sum | cut -d' ' -f1); c=$(sha256sum < "$cur$b" | cut -d' ' -f1); [ "$a" = "$c" ] || { bad=$((bad+1)); echo "STALE: $b"; }; done; echo "checked=$n stale=$bad"
}
main </dev/null