Files
felhom.eu/documentation/audits/n100-clock-2026-11-11
admin d4c7e84c22
gates / gates (push) Successful in 3m34s
Golden 0.352.0 vouched; token fix finished; why the N100 keeps no time; certificate pin spike on demo-hp
- Golden 0.352.0: bake, publish, one fresh install, the operator's vouch (R-468).
- No clone holds a login in its address; the credential helper script is in the repository (R-925, R-1390).
- N100 clock: a stale cached AppArmor profile stops chrony (R-1301, new R-1388). Read only.
- Certificate pin spike on demo-hp (R-1380): the break, the proven hand repair (runbook), the CA-file test,
  03-host-agent.md 12a, decision D80.
- Rulings of 13:17 written down (09 decisions 297-302). Register 245 -> 248 (R-1388, R-1389, R-1390).
- Docs, evidence and one helper script. No release, no floor change, no hub deploy.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-11 15:15:20 +01:00
..

Why the N100 keeps no time — 2026-11-11

Read only. Nothing on the N100, on Tester 1 or on demo-hp was changed. Files: evidence/, tools/ (the read scripts). Documents read: architecture/11-os-updates.md, architecture/03-host-agent.md, the Clock list (audits/clock-list-2026-11-11/, entries CA03, CA04, CA09), rows R-1301, R-1384, R-1206.

The cause, in three lines

  1. The time service (chrony) tells systemd "I am ready" through the socket /run/systemd/notify.
  2. On the N100 the kernel refuses that one write, because the security profile (AppArmor) it loaded for chrony is an old copy from a cache — the copy made for the older chrony, which did not talk to that socket.
  3. chrony treats the refused write as fatal and stops. So the unit fails 2 seconds into every start.

The evidence (each line is a read, not a theory)

What N100 demo-hp Tester 1 host File
chrony package 4.8-4~bpo13+2 the same 4.6.1-3+deb13u1 (old) *-stale-proof.txt
unit, drop-ins, /etc/default/chrony, config, profile file on disk identical (profile sha256 214e509a…) identical not compared n100-host.txt, demo-hp-host.txt
unit state failed, 8 fatal lines = 4 starts active, 0 fatal lines active, 0 *-host-2.txt
profile the kernel has loaded (sha256) dbf67f16… 83417e83… dbf67f16… *-stale-proof.txt
a fresh compile of the profile file on disk 83417e83… 83417e83… dbf67f16… the same
loaded = fresh compile? no yes yes the same
the kernel's own line apparmor="DENIED" operation="sendmsg" profile="/usr/sbin/chronyd" name="/run/systemd/notify" … denied_mask="w" (2026-10-16 05:09:25) none none n100-host-2.txt:13

dbf67f16… is the profile of chrony 4.6.1 (Tester 1, which still runs 4.6.1, compiles exactly that). 83417e83… is the profile of chrony 4.8. The N100 runs chrony 4.8 under the profile of chrony 4.6.1.

chrony's source agrees with the log: main.c notify_system_manager() ends in LOG_FATAL("Could not send notification to $NOTIFY_SOCKET") on any failed send, and sends once at start and once at exit. That is the same line twice for each start.

How the old copy got there (dates from the box)

AppArmor keeps one cache folder for each kernel feature set. Both boxes have two: ae1a4050.0 (kernel 7.0.2) and f6da0254.0 (the 7.0.14 kernels).

When N100 demo-hp
the 7.0.14 folder was made, with the old chrony profile in it 2026-10-07 12:11 (a kernel test of that day booted the new kernel once, then went back) 2026-10-04 09:44
chrony went to 4.8 (apt, requested by felhom-agent, the OS-update leg) 2026-10-07 14:15, on the old kernel → the new profile went into the 7.0.2 folder only 2026-10-10 04:40, on a 7.0.14 kernel → the new profile went into the 7.0.14 folder
next start on a 7.0.14 kernel the 7.0.14 folder still holds the old profile; it is loaded the new profile is loaded

Why AppArmor did not see that the copy was old: the cached file carries the time 2026-07-18 18:15:53; the new profile file from the package carries 2026-07-01 22:28:16. The cache looks newer than its source, so it is used. (The time rule is AppArmor's documented cache check; this run did not read its source.)

A control on the method: the same check names lxc-containers as "differs" on all three boxes, the healthy ones too. That one is not a signal. usr.sbin.chronyd differs on the N100 only.

The three questions

  1. Is the cause the product's? In part. The product did not write the profile or the unit. But the product's OS-update leg installed the new chrony, and the bad order is: start a new kernel once → go back to the old kernel → a package with a changed profile is updated → start the new kernel again. On the N100 a test made that order. On a household box the kernel rollback (the crash guard goes back to the last good kernel) makes the same order. It is true for every AppArmor profile, not only chrony's.
  2. When did it start? The oldest failed start in the journal is 2026-10-16 05:09:25. The cause dates from 2026-10-07 14:15. The journal has no start between those two dates.
  3. The smallest repair. One hand step on the N100 (in STATUS.md, not done, not tested on a box): apparmor_parser -r -T -W /etc/apparmor.d/usr.sbin.chronyd && systemctl restart chrony && chronyc tracking. For a household box no path delivers it today. Proposed, not built: after an OS-update leg felhom-os-apply reloads the profiles with the cache ignored (-T -W), and the kernel rollback does the same. Filed as a row.

The fresh box of Part 1 cannot show this. It has one cache folder and no history. It showed only that a new install keeps time (chrony active, 0.000001 s off, before the Felhom installer ran).

What a wrong clock breaks (numbers from source)

  • Signed jobs (the only way an agent update arrives): valid from issue minus 2 minutes (DefaultClockSkew, felhom-agent/internal/authz/verifier.go:21) to expiry; a window over 24 hours is refused (MaxWindow, :26). A host clock more than 120 s behind refuses every job as "not yet valid" (verifier.go:153-162).
  • The N100 was 32.6 s behind this morning, after 26 days with no time service: about 1.25 s a day. At that rate it passes 120 s in about 70 days (mid-January 2027). This is an estimate from one measurement.
  • The guest has no clock of its own; it reads the host's. So the backup age, the off-site age and the proof age that the hub trusts, the night slots and the crash guard all read this clock (Clock list, class W2, 30 cells, R-1384).
  • Who notices today: nobody. The hub alarms only on a tunnel handshake from the future.