Compare commits

..

9 Commits

Author SHA1 Message Date
admin 861d32a4b4 CHANGELOG: unreleased — R-349 agent_sha256, R-25 agent half (burn-down night)
gates / gates (push) Successful in 43s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 21:16:51 +02:00
admin 769c4c3cf2 R-25 (agent half): the format answer carries the NEW filesystem's UUID, bound to the durable id
After mkfs the agent re-resolves the bound durable id, requires it to name the
device it just formatted, reads the superblock back (blkid -p, requested fstype)
and returns fs_uuid in POST /disks/format and GET /disks/format/status. Anything
unverified returns "" — never a path-resolved guess. The controller half
(mount fs_uuid instead of re-resolving the UUID from the /dev path) is owed
in felhom-controller.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 21:16:11 +02:00
admin 64f704d0f7 R-349: the host report carries the sha256 of the RUNNING agent binary
A hand-built proof binary and the published artifact share a version string
but not their bytes, so no version check could see the divergence. The agent
now reports agent_sha256 (hash of /proc/self/exe, once per process; empty =
unknown) beside agent_version, the same mechanism as host.wrapper_sha256.
The hub half (compare against the vouched agent_sha256, surface drift) is
owed in the hub repo.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 21:16:11 +02:00
admin 208fac8027 CHANGELOG/REPORT: v0.147.0 released (shas)
gates / gates (push) Successful in 56s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 18:58:37 +02:00
admin f1b9b41214 R-124 recipe root namespace as PBS spells it; R-118 no root size for an absent drive; R-269 rotated-out token rejected at once; R-317 dnsmasq install probed by its unit (burn-down round 2)
gates / gates (push) Successful in 47s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 18:56:13 +02:00
admin d83316326e R-291 retention record source, R-348 restart comment (no binary change; burn-down)
gates / gates (push) Successful in 22s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 16:56:18 +02:00
admin e06ed97fa8 agent v0.146.1 REPORT
gates / gates (push) Successful in 22s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 13:00:22 +02:00
admin e4b5cf9693 R-880: build-step-bundle.py — the transition bundle for a release whose bundle adds paths (an installed felhom-os-apply refuses unknown paths, R16)
gates / gates (push) Successful in 21s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 12:23:55 +02:00
admin faa3cad92e agent v0.146.1 CHANGELOG (released)
gates / gates (push) Successful in 19s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-05 12:14:07 +02:00
24 changed files with 809 additions and 68 deletions
+74
View File
@@ -1,3 +1,77 @@
## unreleased
- **R-349:** the host report carries `agent_sha256`, the sha256 of the running agent binary (read once from
`/proc/self/exe`; empty = unknown), so a hand-built binary under the vouched version name becomes visible. The hub
comparison is a separate hub change. Test `TestCollect_AgentSHA256IsTheRunningBinary`; red-proved.
- **R-25 (agent half):** `POST /disks/format` and `GET /disks/format/status` return `fs_uuid`, the new filesystem's UUID
read back after mkfs only when the bound durable id still resolves to the formatted device and the superblock is the
requested type (empty = not verified); `DeviceProbe` gains `FSUUID` from blkid. Tests `TestFormat_*FSUUID*`; three
red-proofs. The controller half (mount that UUID) is a controller change.
## v0.147.0 — the recovery recipe spells the root namespace the way PBS does; a removed drive no longer shows the root disk's size; a rotated-out token stops at once; the dnsmasq check looks at the right package (burn-down round 2: R-124, R-118, R-269, R-317) (2026-10-05)
Released by `scripts/release-agent.sh`: binary sha256 `642c4d196c48671c14ff653118303c5903af1670b7abb550edeaf73e701cd5b8`,
config bundle sha256 `326527d0993c9a62df2f790c7700ca645cedbf0673dcfb6dc1768d8610b8007d` (tag `v0.147.0` = `f1b9b41`).
Delivery order as for v0.146.1: signed `agent_update`, then signed `agent_config_update`.
MinAgent impact: none (the controller needs nothing new from this agent). Config bundle content unchanged from v0.146.1.
- **R-124 (operator ruling 2026-10-05: fix it):** the DR recipe's `pbs.namespace` for a box in PBS's ROOT namespace is
now `""` — PBS's own spelling — beside `namespace_state: resolved`; it used to be the word `root`, which no namespace
is named, so `--ns root` failed in a recovery. `hub.PBSRootNamespace`; `TestR124_RootNamespaceOnTheWireIsPBSSpelling`
(red-proof: back to "root" → FAIL). Runbook: `felhom.eu runbooks/ep0-datastore-copy.md` step 2 says how to read it
(and to treat a recorded `root` from older agents as empty). The hub stores the recipe raw; its fixture follows.
- **R-118:** the local API's drive list reads a drive's capacity only while its DEVICE is present — with the device gone
the bare mountpoint is a directory on the root filesystem, whose size was reported as the drive's.
`TestDisks_UnionPath_AbsentDeviceReportsNoRootCapacity` (red-proof convicts).
- **R-269:** the token store re-reads its shared file whenever it has grown, BEFORE answering — so a token rotated out by
another process stops authorizing on its next use (it used to keep working until an unrelated miss). One `stat` per
call. `TestTokenStore_RotatedOutTokenRejectedFirst` (red-proof convicts).
- **R-317:** the LAN resolver decides whether to install `dnsmasq` by its service UNIT, not by `/usr/sbin/dnsmasq` (which
the `dnsmasq-base` package also ships). Same `apt-get install` command; no sudoers change. `TestEnsureDnsmasq_*`
(red-proof convicts). Red-proofs: `felhom.eu/documentation/audits/burndown2-2026-10-05/agent-red-proofs.txt`,
`r124-red-proof.txt`.
Also in this release (no binary effect; from burn-down round 1):
- **R-291:** `scripts/retention-policy.json` names where its 10 comes from — the R-267 newest-10 prune of generic
packages, established 2026-08-10 (R-287) — instead of „observed, no located ruling"; the non-existent
`registry-retention.md` reader is dropped. `check-published-versions.py` still reads 10 (checked).
- **R-348:** `internal/backup/store.go` no longer says backups are „unaffected" by a restart: the reported backup list
reads 0 until the next backup runs; only the hub's verdict (7-day look-back) is unaffected.
## v0.146.1 — R-861 review fixes: the signed update flips a root-owned copy; no Wants=/continuations in mount units; the escrow read follows no symlink anywhere (2026-10-05)
Released by `scripts/release-agent.sh`: binary sha256 `badd6c9a2e40c8bfe856d2d1a203443b21b7eb92ecc35d6090ab44518d4d082a`,
config bundle sha256 `42333e969028867ad8142335e6c1bc4040eec231de0d8d330c2d4b2cf7bc3442`. **Supersedes v0.146.0, which
was released but never vouched or delivered to any box.** The same order applies: signed `agent_update` first, then the
signed `agent_config_update`.
**Delivery needs a STEP bundle (R-880, found while delivering).** An installed `felhom-os-apply` checks an incoming
bundle's paths against its OWN table (R16), so every box on the v0.145.0 bundle REFUSES the v0.146.1 bundle (it adds 4
paths). `scripts/build-step-bundle.py` builds the transition: the box's current bundle with ONLY `felhom-os-apply`
replaced (same paths — the old wrapper accepts it), published as bundle version `0.146.1-step1`; then the release's own
bundle. Order on a box: `agent_update` 0.146.1 → `agent_config_update` 0.146.1-step1 → `agent_config_update` 0.146.1.
Tests `StepBundle` (the R16 refusal reproduced; the step accepted; exactly one file changed). Tooling only — not in the
binary or the bundle.
A background security review of the v0.146.0 commit found three holes in the new code; each is fixed and red-proved
(`felhom.eu/documentation/audits/hub-safety-2026-10-05/partF/red-proof.txt`, S1–S3):
- **S1 — a race in the signed update.** `felhom-os-apply` hashed the agent's staged file and then let the A/B wrapper copy
it BY PATH; the agent owns that directory and could swap the file in between. Now the root step reads the file ONCE
(`read_staged_once`: O_NOFOLLOW, fstat, owner, size), hashes those bytes, writes them to a root-owned directory
(`/var/lib/felhom-os-apply/agent-update/`) and hands ONLY that copy to `felhom-selfupdate-guarded apply`, which now
refuses any other directory, a symlink, or a file not owned by root. Tests: `AgentUpdate` (+1),
`SelfupdateWrapperConfinement`.
- **S2 — an allowlist escape in `felhom-priv-apply`.** `[Unit]` accepted `Wants=`/`Requires=`/`Before=` naming any unit, so
a mount unit could start e.g. `reboot.target`. `[Unit]` now holds only `Description` and `After=local-fs-pre.target`
(what the renderers write), and any line ending in a backslash (a systemd continuation this parser would read
differently) is refused. Tests `test_U2_wants_starts_another_unit`, `test_U2_continuation_line`.
- **S3 — a path traversal in the escrow read.** `O_NOFOLLOW` guards only the last component; a symlinked DIRECTORY in the
agent's own state dir still redirected the root read. `readStagedNoFollow` now walks the path from `/` with
`openat(O_NOFOLLOW)` per component. Test `TestAttach_RefusesASymlinkedDirectory`.
## v0.146.0 — the agent's root grants narrowed: exact sudo patterns, a root content checker, fixed files from the bundle, the signed update checked as root (R-861) (2026-10-05)
Released by `scripts/release-agent.sh`: binary sha256 `b860af465076041e07f35fed1b12d64ae2b2985d8995f0ce167418d39c2b00d5`,
+7 -11
View File
@@ -1,14 +1,10 @@
# REPORT — agent v0.145.0 (2026-10-05, afternoon): the OS update repairs itself after a power cut
# REPORT — agent v0.147.0 (2026-10-05, burn-down round 2)
Brief: the 2026-10-05 catch-up brief (operator), Parts C (R-874, R-875) and D (R-876). Full session report:
`felhom.eu/REPORT-catchup-2026-10-05.md`. Architecture: `11-os-updates.md` §5.4.1, §8.4–8.5; `09` decisions 117–118.
Full session report: `felhom.eu/REPORT-burndown2-2026-10-05.md`. Baseline `d833163` (v0.146.1). Code commit `f1b9b41`
(CI job 1365 success), tag `v0.147.0`, binary sha256 `642c4d19…`, bundle sha256 `326527d0…` (verified by download).
| Row | Fix | Proof |
|---|---|---|
| R-876 | the wrapper reads `dpkg --audit` AND the update journal in ONE call, repairs on either; belt: repair + retry once when apt says "interrupted" | 3 tests + 3 red-proofs; **live (operator's go): crash mid-unpack on demo-hp → the next pass `REPAIR … journal=1` → `DONE rc=0 upgraded=12`, nobody touched the box** |
| R-874 | the restore-test's first due-check 30 min after start | 2 tests + red-proof; live on demo-felhom: passed restore-test at start + 30 min |
| R-875 | a kept report's reason is neutral ("sent late …") | test + red-proof |
Rows: R-124 (recipe root namespace = ""), R-118 (no root size for an absent drive), R-269 (rotated-out token rejected at
once), R-317 (dnsmasq install probed by its unit). Tests + red-proofs: `felhom.eu/documentation/audits/burndown2-2026-10-05/`.
`go build/vet/test ./...` green; `agent_gates.py --fast` green after the release (release-complete needs the tag).
Released by `scripts/release-agent.sh`: v0.145.0 (`894da35c…`, bundle `78c00adc…`), verified by download; signed
`agent_update` + `agent_config_update` to demo-hp, demo-felhom, tester-1 (71/71 after each bundle); vouched with
golden 0.295.0, min_agent 0.131.0. `go test ./...` rc 0, Python suites OK, `agent_gates.py` OK.
Delivery: see the session report (vouch, signed jobs per box, the hub System page afterwards).
+3 -2
View File
@@ -66,8 +66,8 @@
|---|---|---|---|---|
| `IntentStore` (`Get/SetEnrolled/SetEjected/SetDecommissioned/OnAbsent`) | internal/storage/intent.go | `OpenIntentStore(path)` | drive intent (4-state self-heal) | Keyed by durable-id only; `OnAbsent` is the ONLY ejected→enrolled path; refuses empty ids |
| `GuestBindStore` (`Record/Remove/Guests`) | internal/localapi/guestbindstore.go | `OpenGuestBindStore(path)` | per-guest enrolled binds (F9 re-assert) | Same tmp+rename 0600 pattern as IntentStore |
| `FormatJobStore` + `startFormatDetached` + `RecoverFormatJob` | internal/localapi/formatjob.go | `startFormatDetached(device, durableID, fstype, blank) <-chan error` | detached, restart-surviving mkfs (F20-BUG3) | Runs off `s.baseCtx` (60-min bound) so a request deadline can't SIGKILL mkfs; recovery re-resolves by durable id; blank jobs re-check STILL-blank |
| `TokenStore.Mint` / `Lookup` | internal/localapi/tokenstore.go | `Mint(vmid) (plaintext, error)` | per-guest local-API tokens | Only the SHA-256 hash persists (fsync'd append log); constant-time compare on lookup; plaintext returned exactly once. Lookup RELOADS the file once on a miss (v0.63.0, B3): the one-shot provisioner mints into the same file the daemon indexes — cross-process coherence without a restart; append-only size check bounds the re-read |
| `FormatJobStore` + `startFormatDetached` + `RecoverFormatJob` | internal/localapi/formatjob.go | `startFormatDetached(device, durableID, fstype, blank) (*formatJob, <-chan error)` | detached, restart-surviving mkfs (F20-BUG3) | Runs off `s.baseCtx` (60-min bound) so a request deadline can't SIGKILL mkfs; recovery re-resolves by durable id; blank jobs re-check STILL-blank; on success `job.FSUUID` = the new filesystem's UUID, read back only when the durable id still resolves to the formatted device (R-25) — read it only after `done` delivers |
| `TokenStore.Mint` / `Lookup` | internal/localapi/tokenstore.go | `Mint(vmid) (plaintext, error)` | per-guest local-API tokens | Only the SHA-256 hash persists (fsync'd append log); constant-time compare on lookup; plaintext returned exactly once. Lookup stats the file on EVERY call and reloads BEFORE answering when the append-only log grew (R-269; was reload-on-miss only, v0.63.0 B3, which let a token rotated out by another process keep authorizing as a map hit): the one-shot provisioner mints into the same file the daemon indexes — cross-process coherence both ways without a restart; unchanged size = no re-read. Pinned by `TestTokenStore_RotatedOutTokenRejectedFirst` |
| `FileNonceStore.SeenOrRecord` | internal/authz/noncestore.go | `SeenOrRecord(nonce, exp) bool` | durable anti-replay | fsync'd before returning false; prune only after exp |
| `Journal` (`Append/Latest/InFlight/AlreadyApplied`) | internal/reconcile/journal.go | `OpenJournal(path)` | op journal + idempotency + crash recovery | `Recover` consumes `InFlight()`; scratch entries special-cased |
@@ -157,6 +157,7 @@
| `storage.HostOps` | internal/storage/hostops.go | `*SudoHostOps` (prod), `NoopHostOps` (degraded) | fakes in internal/storage/observe_test.go, watchdog_test.go |
| `storage.HostReader` | internal/storage/hostread.go | `*ProcHostReader` | `fakeHostReader` internal/localapi/disks_test.go; internal/storage/role_test.go. v0.87.0: `BlockSlaves(name)` lists `/sys/block/<name>/slaves` (root-free) — backs the `SystemDisks` dm/md walk (`physicalDisksOf`/`walkSlaves`, role.go); per-branch conservatism: an unresolvable slave fails the WHOLE walk → all-system fail-safe. NEVER weaken the signature test `TestSystemDisks_WalkTopologies` (root-backing disk always in the system set). |
| `localapi.DiskOps` / `StorageGate` / `GuestAttacher` / `GuestLister` | internal/localapi/disks.go | `*storage.SudoHostOps`; `storageGateAdapter` (cmd/felhom-agent/main.go); `*GuestBinder`; `*proxmox.Client` | `fakeDiskOps`/`fakeGate`/`fakeGuestAttacher`/`fakeGuestList` internal/localapi/disks_test.go |
| `lanresolver.hostRoot` + `dnsmasqUnitPaths` (data seam, R-317) | internal/lanresolver/lanresolver.go | prod `hostRoot = "/"`; probe = the `dnsmasq` package's systemd UNIT, never `/usr/sbin/dnsmasq` (owned by `dnsmasq-base`) | internal/lanresolver/ensure_dnsmasq_test.go — fixture root tree + recording `proxmox.Runner`; the REAL `os.Stat` probe and `EnsureDnsmasq` run. `TestEnsureDnsmasq_ProductionProbeIsTheUnit` pins the production wiring |
| `localapi.GuestAPI` / `BackupService` / `BackupStore` / `TokenAuthority` | internal/localapi/server.go | `*proxmox.Client`, `*backup.BackupRunner`, `*backup.Store`, `*TokenStore` | `fakeGuests`/`fakeBackups`/`fakeStore` internal/localapi/server_test.go |
| `backup.InFlight` | internal/backup/inflight.go | `TryAcquire(what) (release, busy, ok)` / `Busy()` | THE host-wide "one heavy guest operation at a time" gate — shared by the local-API backup path and the restore-test scheduler (R-85) | A **LINK** guard, not a lock one: the scratch VMID never touches the live guest's vzdump lock, but an offsite restore PULLS multi-GB over the tunnel a backup PUSHES one. Callers **DEFER, never cancel** — a deferred restore-test costs coverage, a cancelled backup costs the backup. A nil gate is ungated (pre-R-85 callers). |
| `capability` store-grant probe (`storeGrantStatuses` / `storeGrantVerdict` / `Client.Permissions`) | cmd/felhom-agent/main.go, internal/proxmox/query.go | *"may the agent READ this backup tier?"*, one `capability.Status` per configured tier | R-185. **Never infer permission from an empty content listing** — `{"data":[]}` is what a FORBIDDEN tier and a NEWBORN tier both return, and that ambiguity hid an unreadable host tier on both demo boxes. Ask `/access/permissions` **as the agent's own token** (root always says yes). **The ungranted answer is not empty and not a 403** — it carries the privileges inherited from the box-wide `/` grant, so test for **`Datastore.AllocateSpace`** specifically; path-presence or `Datastore.Audit` reports a blinded storage healthy. Probed set comes from `BackupTiers()`, never a fixed list. Critical except the `local` fallback. Composes AROUND the sudo prober (the `poolReadStatus` precedent); `Status`'s wire shape is untouched so the hub alert is free. Unreachable PVE ⇒ degraded, never ok. |
+62
View File
@@ -667,5 +667,67 @@ class SelfupdateWrapperConfinement(unittest.TestCase):
self.assertEqual(p.returncode, 1, p.stderr)
self.assertIn("outside /var/lib/felhom-os-apply/agent-update", p.stderr)
_sb = importlib.machinery.SourceFileLoader("stepbuild", str(REPO / "scripts" / "build-step-bundle.py"))
_ss = importlib.util.spec_from_loader("stepbuild", _sb)
stepbuild = importlib.util.module_from_spec(_ss)
_sb.exec_module(stepbuild)
NEW_IN_0146 = {"/usr/local/sbin/felhom-priv-apply", "/var/lib/vz/snippets/felhom-guest-hook.sh",
"/usr/local/sbin/felhom-shared-parent.sh", "/etc/systemd/system/felhom-shared-parent.service"}
class StepBundle(unittest.TestCase):
"""R-880 (agent v0.146.1): an INSTALLED wrapper checks an incoming bundle's paths against its OWN table (R16), so a
release that adds paths needs a step bundle: the boxes' current bundle with only felhom-os-apply replaced.
RED-PROOF: deliver the full bundle to the old table → R16 (test_the_full_bundle_is_refused_by_an_old_table)."""
def old_world(self):
"""The base bundle an older wrapper (no R-861 paths) installed, and that wrapper's table."""
full = json.loads(builder.build("0.145.0"))
full["files"] = [e for e in full["files"] if e["path"] not in NEW_IN_0146]
old_wrapper = b'# the v0.145.0 wrapper stands in here\nBUNDLE_OP = "agent_config_update"\n'
for e in full["files"]:
if e["path"] == "/usr/local/sbin/felhom-os-apply":
e["content_b64"], e["sha256"] = base64.b64encode(old_wrapper).decode(), hashlib.sha256(old_wrapper).hexdigest()
base = (json.dumps(full, indent=1, sort_keys=True) + "\n").encode()
old_dests = {k: v for k, v in osapply.BUNDLE_DESTS.items() if k not in NEW_IN_0146}
return base, old_dests
def parse_with_table(self, data, dests, version):
saved = osapply.BUNDLE_DESTS
osapply.BUNDLE_DESTS = dests
try:
return osapply.Bundle(osapply.Apply(Box(b"{}", None), "")).parse(data, hashlib.sha256(data).hexdigest(), version)
finally:
osapply.BUNDLE_DESTS = saved
def test_the_full_bundle_is_refused_by_an_old_table(self):
_, old_dests = self.old_world()
full = builder.build("0.146.1")
with self.assertRaises(osapply.Refused) as cm:
self.parse_with_table(full, old_dests, "0.146.1")
self.assertEqual(cm.exception.code, "R16")
def test_the_step_bundle_is_accepted_by_the_old_table_and_changes_only_the_wrapper(self):
base, old_dests = self.old_world()
new_wrapper = (HERE / "felhom-os-apply").read_bytes()
step = stepbuild.build_step(base, "0.146.1-step1", new_wrapper)
ver, files = self.parse_with_table(step, old_dests, "0.146.1-step1")
self.assertEqual(ver, "0.146.1-step1")
b, s_ = json.loads(base), json.loads(step)
self.assertEqual(sorted(e["path"] for e in b["files"]), sorted(e["path"] for e in s_["files"]), "the paths must not change")
changed = [e["path"] for e, f in zip(sorted(b["files"], key=lambda x: x["path"]), sorted(s_["files"], key=lambda x: x["path"]))
if e != f]
self.assertEqual(changed, ["/usr/local/sbin/felhom-os-apply"], "exactly the wrapper changes")
installed = dict((d, c) for d, c, *_ in files)
self.assertEqual(installed["/usr/local/sbin/felhom-os-apply"], new_wrapper)
# and the NEW wrapper (now installed) knows every path the release's full bundle names
self.assertTrue({e["path"] for e in json.loads(builder.build("0.146.1"))["files"]} <= set(osapply.BUNDLE_DESTS))
def test_a_step_version_must_carry_a_suffix(self):
base, _ = self.old_world()
with self.assertRaises(SystemExit):
stepbuild.build_step(base, "0.146.1", b"x")
if __name__ == "__main__":
unittest.main()
+5 -1
View File
@@ -25,7 +25,11 @@ import (
// merges the two — see hub.ProvenRestoreTestReporter. This store remains the ONLY place a FAILURE is
// recorded, and that asymmetry is deliberate: a failing tier stays due and is retried, so a lost
// failure heals itself, while a lost success leaves the system quietly less tested than it believes.
// Backups are unaffected — their freshness has a ground truth on the storage (R-84).
// Backups are NOT unaffected (corrected 2026-10-05, R-348): byTarget is in memory too, so after a restart the
// reported backup LIST reads 0 until the next backup of each tier runs (daily local, weekly offsite) — measured
// 2026-08-20, two consecutive host-reports with `0 backups` while `pvesm list` showed archives on both tiers. What
// is unaffected is the hub's VERDICT: it looks back 7 days over stored reports (felhom.eu hub/internal/monitor/
// deadline.go backupEvidenceLookback) and the storage stays the ground truth (R-84).
type Store struct {
mu sync.Mutex
byTarget map[string]hub.Backup // latest backup per target id
+26 -2
View File
@@ -10,6 +10,7 @@ import (
"log/slog"
"os"
"strings"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/capability"
@@ -115,6 +116,7 @@ type Collector struct {
backupTarget func() ConfiguredBackupTarget // R-109: primary backup tier id (nil → recipe records unknown)
hostID string
agentVersion string
selfSHA func() string // R-349: sha256 of the running binary; default runningBinarySHA256
logger *slog.Logger
now func() time.Time
}
@@ -135,6 +137,7 @@ func NewCollector(px proxmoxReader, cf CloudflaredProber, storage StorageObserve
temp: SysfsTempReader{}, // slice 9: real sysfs reader by default; tests inject a fake
hostID: hostID,
agentVersion: agentVersion,
selfSHA: runningBinarySHA256,
logger: logger,
now: func() time.Time { return time.Now().UTC() },
}
@@ -337,6 +340,7 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
HostID: c.hostID,
ReportedAt: c.now().Format(time.RFC3339),
AgentVersion: c.agentVersion,
AgentSHA256: c.agentSHA256(),
Host: host,
Guests: c.collectGuests(ctx),
// storage_targets populated this slice (slice 5) via the observer; the rest stay
@@ -431,8 +435,28 @@ const pbsWrapperPath = "/usr/local/sbin/felhom-pbs-apply"
// unreadable file yields "", which the hub reads as UNKNOWN rather than as drift — a host that
// legitimately has no DR wrapper must not light up amber. The file is 0755, so no privilege is
// needed to read it.
func pbsWrapperSHA256() string {
f, err := os.Open(pbsWrapperPath)
func pbsWrapperSHA256() string { return fileSHA256(pbsWrapperPath) }
// selfExePath is the running binary as the kernel holds it. /proc/self/exe, not the installed path:
// after an A/B flip the file at /usr/local/bin/felhom-agent may already be the NEXT binary while this
// process still runs the old one, and the report must describe what runs (R-349). Test seam.
var selfExePath = "/proc/self/exe"
// runningBinarySHA256 hashes the running binary ONCE per process — the bytes cannot change under a
// running process, and re-hashing ~20 MB every report cycle buys nothing. A failed read is cached as
// "" (UNKNOWN); it never fails the report.
var runningBinarySHA256 = sync.OnceValue(func() string { return fileSHA256(selfExePath) })
func (c *Collector) agentSHA256() string {
if c.selfSHA == nil {
return ""
}
return c.selfSHA()
}
// fileSHA256 is the hex sha256 of a file's bytes, or "" when it cannot be read.
func fileSHA256(path string) string {
f, err := os.Open(path)
if err != nil {
return ""
}
+64
View File
@@ -0,0 +1,64 @@
package hub
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
)
// R-349: the report carries the sha256 of the binary that is RUNNING, so the hub can tell a
// hand-built proof binary from the vouched artifact of the same version string. The consequence
// asserted: the wire field equals the hash of this very test binary's bytes (read independently via
// os.Executable, a different channel from /proc/self/exe), and it is on the wire as agent_sha256.
func TestCollect_AgentSHA256IsTheRunningBinary(t *testing.T) {
exe, err := os.Executable()
if err != nil {
t.Skipf("os.Executable: %v", err)
}
raw, err := os.ReadFile(exe)
if err != nil {
t.Fatalf("read own binary: %v", err)
}
sum := sha256.Sum256(raw)
want := hex.EncodeToString(sum[:])
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "running"}, nil, nil, nil, nil, "h", "0.3.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
}
if r.AgentSHA256 != want {
t.Fatalf("agent_sha256 = %q, want the running binary's %q", r.AgentSHA256, want)
}
b, err := json.Marshal(r)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(b), `"agent_sha256":"`+want+`"`) {
t.Fatalf("agent_sha256 not on the wire: %s", b)
}
}
// An unreadable binary is UNKNOWN (empty, omitted) — never a made-up hash, never a failed report.
func TestFileSHA256_UnreadableIsEmpty(t *testing.T) {
if got := fileSHA256(filepath.Join(t.TempDir(), "absent")); got != "" {
t.Fatalf("absent file hashed to %q, want empty", got)
}
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "running"}, nil, nil, nil, nil, "h", "0.3.0", quietLogger())
c.selfSHA = func() string { return "" }
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect must not fail on an unreadable binary: %v", err)
}
b, _ := json.Marshal(r)
if strings.Contains(string(b), "agent_sha256") {
t.Fatalf("empty agent_sha256 must be omitted: %s", b)
}
}
+9 -7
View File
@@ -54,11 +54,13 @@ const (
DRReasonNoPBSStorage = "no_pbs_storage_observed"
)
// PBSRootNamespace is how the recipe spells PBS's root namespace. The PBS API spells it as the EMPTY
// string (and `pct restore --ns root` would name a namespace that does not exist) — "root" is a display
// convention this wire has always used, kept here so the field's meaning did not change under R-106.
// Only a box with no `namespace` line in its pbs storage.cfg stanza ever emits it.
const PBSRootNamespace = "root"
// PBSRootNamespace is how the recipe spells PBS's root namespace: the EMPTY string, PBS's own spelling (R-124,
// agent v0.147.0). It used to be the display word "root", which no PBS namespace is named — an operator pasting it
// into `proxmox-backup-client … --ns root` during a real recovery got a failure. An empty namespace is ambiguous on
// its own, so READ IT WITH namespace_state: resolved + "" = the root namespace (pass no --ns, or --ns ""); unknown +
// "" = the agent could not tell. Only a box with no `namespace` line in its pbs storage.cfg stanza emits it.
// Pinned by TestDRRecipe_PBSNamespaceRootIsResolvedNotUnknown and TestR124_RootNamespaceOnTheWireIsPBSSpelling.
const PBSRootNamespace = ""
// DRRecipeHostHalf is the agent-emitted half (guest/drive/storage/PBS scaffolding). Derived entirely
// from facts the report already collects — no new privileged reads.
@@ -123,8 +125,8 @@ type DRPBSCoord struct {
RepoID string `json:"repo_id"` // the PVE pbs storage id (e.g. "felhom-pbs") — not a token
// Namespace is the PBS namespace the restore targets, resolved from the pbs storage's storage.cfg
// stanza — the same field `vzdump --storage <pbs>` makes PVE read, so the recipe cannot disagree
// with the backup that produced the snapshot. PBSRootNamespace when the box has no namespace
// configured; "" when NamespaceState is unknown.
// with the backup that produced the snapshot. PBSRootNamespace ("", PBS's spelling, R-124) when the box has no
// namespace configured; also "" when NamespaceState is unknown — consult NamespaceState.
//
// R-106: this used to come from the listed snapshot's own `ns`, which PBS does not echo per item once
// the request is already namespace-scoped via `?ns=` (internal/pbs/client.go). The field was
+39 -3
View File
@@ -63,8 +63,8 @@ func TestBuildDRRecipeHostHalf(t *testing.T) {
t.Error("felhom-flash (local-dir user-data drive) missing from drives")
}
// pbs: latest snapshot's coords + the pbs storage id as repo_id.
if h.PBS == nil || h.PBS.RepoID != "felhom-pbs" || h.PBS.Namespace != "root" || h.PBS.LatestSnapshotID != "9201" {
t.Errorf("pbs coord = %+v, want repo felhom-pbs/root/9201", h.PBS)
if h.PBS == nil || h.PBS.RepoID != "felhom-pbs" || h.PBS.Namespace != PBSRootNamespace || h.PBS.LatestSnapshotID != "9201" {
t.Errorf("pbs coord = %+v, want repo felhom-pbs, the root namespace (\"\", R-124), snapshot 9201", h.PBS)
}
}
@@ -252,7 +252,7 @@ func TestDRRecipe_PBSNamespaceIsThePerCustomerOne(t *testing.T) {
}
// TestDRRecipe_PBSNamespaceRootIsResolvedNotUnknown: a box with a pbs storage and NO namespace line is
// genuinely in the root namespace. That is an answer, not a gap — it must read resolved/"root", so the
// genuinely in the root namespace. That is an answer, not a gap — it must read resolved/"" (PBS's spelling, R-124), so the
// honest root case is never confused with "I could not tell".
func TestDRRecipe_PBSNamespaceRootIsResolvedNotUnknown(t *testing.T) {
h := BuildDRRecipeHostHalf(nil,
@@ -453,3 +453,39 @@ func assertNoSecretKeys(t *testing.T, jsonBytes []byte) {
}
walk("<root>", v)
}
// R-124: on the WIRE the root namespace is PBS's own spelling — an empty string, present (not omitted), beside
// namespace_state "resolved". The display word "root" names no PBS namespace, and `--ns root` fails in a recovery.
// RED-PROOF: set PBSRootNamespace back to "root" → this test fails.
func TestR124_RootNamespaceOnTheWireIsPBSSpelling(t *testing.T) {
h := BuildDRRecipeHostHalf(nil,
[]StorageTarget{{Name: "felhom-pbs", Type: StorageTypePBS, Content: "backup", PBSNamespace: ""}},
capturedDemoFelhomSnapshots(),
ConfiguredBackupTarget{StorageID: "felhom-pbs", Known: true})
b, err := json.Marshal(h.PBS)
if err != nil {
t.Fatal(err)
}
var m map[string]any
if err := json.Unmarshal(b, &m); err != nil {
t.Fatal(err)
}
ns, present := m["namespace"]
if !present {
t.Fatalf("namespace key missing from %s — an omitted key reads as 'unknown', not 'root'", b)
}
if ns != "" {
t.Fatalf("root namespace on the wire = %q, want \"\" (PBS's spelling; no namespace is named %q)", ns, ns)
}
if m["namespace_state"] != DRStateResolved {
t.Fatalf("namespace_state = %v, want %q beside the empty root namespace", m["namespace_state"], DRStateResolved)
}
// A configured namespace still passes through unchanged.
h2 := BuildDRRecipeHostHalf(nil,
[]StorageTarget{{Name: "felhom-pbs", Type: StorageTypePBS, Content: "backup", PBSNamespace: "demo-felhom"}},
capturedDemoFelhomSnapshots(),
ConfiguredBackupTarget{StorageID: "felhom-pbs", Known: true})
if h2.PBS.Namespace != "demo-felhom" {
t.Fatalf("configured namespace = %q, want demo-felhom", h2.PBS.Namespace)
}
}
+10
View File
@@ -18,6 +18,16 @@ type HostReport struct {
HostID string `json:"host_id"` // echoes config.Hub.HostID
ReportedAt string `json:"reported_at"` // RFC3339, agent clock
AgentVersion string `json:"agent_version"`
// AgentSHA256 is the sha256 of the binary this process is RUNNING (read through /proc/self/exe,
// once per process), R-349. The version string cannot tell a hand-built proof binary from the
// published, vouched artifact of the same version — same source, different bytes (`-trimpath
// -buildvcs=false` in release-agent.sh) — so self-update sees "already installed" and never
// corrects it. Reporting the bytes lets the hub compare against the vouched agent_sha256, the
// same mechanism host.wrapper_sha256 is for the PBS wrapper (R-50b(a)).
//
// Empty = unreadable, which the hub must treat as UNKNOWN, never as drift. Pinned by
// TestCollect_AgentSHA256IsTheRunningBinary.
AgentSHA256 string `json:"agent_sha256,omitempty"`
Host HostMetrics `json:"host"`
Guests []Guest `json:"guests"`
+122
View File
@@ -0,0 +1,122 @@
package lanresolver
import (
"context"
"io"
"log/slog"
"os"
"path/filepath"
"strings"
"sync"
"testing"
)
// recRunner records every privileged command EnsureDnsmasq would run and succeeds — nothing reaches
// apt, systemctl or the root checker.
type recRunner struct {
mu sync.Mutex
calls []string
}
func (r *recRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.mu.Lock()
defer r.mu.Unlock()
r.calls = append(r.calls, strings.Join(append([]string{name}, args...), " "))
return nil, nil, nil
}
func (r *recRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
return r.Run(ctx, name, args...)
}
func (r *recRunner) installed() bool {
for _, c := range r.calls {
if strings.HasPrefix(c, "apt-get install") && strings.HasSuffix(c, " dnsmasq") {
return true
}
}
return false
}
// fixtureRoot builds a fake host root holding exactly the given relative files and points the REAL
// probe at it for the test's duration.
func fixtureRoot(t *testing.T, files ...string) {
t.Helper()
root := t.TempDir()
for _, f := range files {
p := filepath.Join(root, f)
if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(p, nil, 0o644); err != nil {
t.Fatal(err)
}
}
prev := hostRoot
hostRoot = root
t.Cleanup(func() { hostRoot = prev })
}
func ensure(t *testing.T) *recRunner {
t.Helper()
r := &recRunner{}
m := NewManager(r, "192.0.2.10", []string{"1.1.1.1"}, slog.New(slog.NewTextHandler(io.Discard, nil)))
if err := m.EnsureDnsmasq(context.Background()); err != nil {
t.Fatalf("EnsureDnsmasq: %v", err)
}
return r
}
// R-317: a host with `dnsmasq-base` (the /usr/sbin/dnsmasq binary) but WITHOUT the `dnsmasq` package
// (the service unit) must get the package installed — else the following `systemctl enable --now
// dnsmasq` hits a unit that does not exist and LAN name resolution silently never comes up.
//
// RED-PROOF: probe "usr/sbin/dnsmasq" instead of the unit paths in dnsmasqUnitInstalled → this fails
// with "install was skipped".
func TestEnsureDnsmasq_BinaryWithoutUnitInstalls(t *testing.T) {
fixtureRoot(t, "usr/sbin/dnsmasq")
r := ensure(t)
if !r.installed() {
t.Fatalf("install was skipped on a dnsmasq-base-only host (binary present, unit absent) — "+
"the enable that follows targets a missing unit (R-317). calls: %q", r.calls)
}
}
func TestEnsureDnsmasq_UnitPresentSkipsInstall(t *testing.T) {
for _, unit := range []string{"usr/lib/systemd/system/dnsmasq.service", "lib/systemd/system/dnsmasq.service"} {
t.Run(unit, func(t *testing.T) {
fixtureRoot(t, "usr/sbin/dnsmasq", unit)
if r := ensure(t); r.installed() {
t.Fatalf("apt-get install ran although the dnsmasq unit is present at %s: %q", unit, r.calls)
}
})
}
}
func TestEnsureDnsmasq_NothingPresentInstalls(t *testing.T) {
fixtureRoot(t)
if r := ensure(t); !r.installed() {
t.Fatalf("install skipped on a host with no dnsmasq at all: %q", r.calls)
}
}
// Production wiring for the hostRoot seam: the shipped probe resolves against the real root and asks
// about the unit the `dnsmasq` package owns — never the dnsmasq-base binary.
func TestEnsureDnsmasq_ProductionProbeIsTheUnit(t *testing.T) {
if hostRoot != "/" {
t.Fatalf("hostRoot default = %q, want \"/\" — the production probe would look in the wrong tree", hostRoot)
}
var sawUsrLib bool
for _, p := range dnsmasqUnitPaths {
full := filepath.Join(hostRoot, p)
if strings.HasSuffix(full, "/sbin/dnsmasq") || strings.HasSuffix(full, "/bin/dnsmasq") {
t.Errorf("probe path %s is the dnsmasq-base binary, not the dnsmasq unit (R-317)", full)
}
if full == "/usr/lib/systemd/system/dnsmasq.service" {
sawUsrLib = true
}
}
if !sawUsrLib {
t.Errorf("probe paths %q miss /usr/lib/systemd/system/dnsmasq.service (dpkg -S: owned by dnsmasq)", dnsmasqUnitPaths)
}
}
+26 -2
View File
@@ -101,11 +101,35 @@ func NewManager(runner proxmox.Runner, hostIP string, upstreams []string, logger
}
}
// hostRoot is the filesystem root the install probe resolves against: "/" in production; a test
// points it at a fixture tree so the REAL probe runs against files it controls.
var hostRoot = "/"
// dnsmasqUnitPaths are where the `dnsmasq` package ships its systemd unit (Debian; /lib is the
// pre-usrmerge spelling). R-317: probe the UNIT, never /usr/sbin/dnsmasq — that binary belongs to
// `dnsmasq-base`, so a host carrying dnsmasq-base without dnsmasq used to skip the install and then
// `systemctl enable --now dnsmasq` failed against a unit that is not there (resolver never up).
// Pinned by TestEnsureDnsmasq_BinaryWithoutUnitInstalls.
var dnsmasqUnitPaths = []string{
"usr/lib/systemd/system/dnsmasq.service",
"lib/systemd/system/dnsmasq.service",
}
// dnsmasqUnitInstalled reports whether the dnsmasq service unit (the `dnsmasq` package) is present.
func dnsmasqUnitInstalled() bool {
for _, p := range dnsmasqUnitPaths {
if _, err := os.Stat(filepath.Join(hostRoot, p)); err == nil {
return true
}
}
return false
}
// EnsureDnsmasq makes dnsmasq present + enabled and writes the host base config. Idempotent: it
// installs the package only when absent, and writes the base drop-in only when its content changes.
func (m *Manager) EnsureDnsmasq(ctx context.Context) error {
if _, err := os.Stat("/usr/sbin/dnsmasq"); err != nil { // metadata read, no privilege needed
m.logger.Info("lanresolver: dnsmasq absent — installing")
if !dnsmasqUnitInstalled() { // metadata read, no privilege needed
m.logger.Info("lanresolver: dnsmasq service unit absent — installing")
if out, errOut, ierr := m.runner.Run(ctx, "apt-get", "install", "-y", "-q", "dnsmasq"); ierr != nil {
return fmt.Errorf("install dnsmasq: %s: %w", strings.TrimSpace(string(errOut))+string(out), ierr)
}
+20 -8
View File
@@ -398,9 +398,16 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
di.Smart = &sm
}
}
if total, used, okc := statfsCapacity(d.MountPath); okc {
di.TotalBytes, di.UsedBytes = total, used
di.UsedFraction = float64(used) / float64(total)
// R-118: statfs ONLY while the drive's device is present. With the device gone the raw
// mountpoint reverts to a bare directory on the ROOT filesystem, and statfs would report
// pve-root's size as this drive's (measured: a 4 GB drive advertising 46 GiB). Same trap
// observe.go guards on the Observe path. Absent → capacity left zero (unknown), never root's.
// Pinned by TestDisks_UnionPath_AbsentDeviceReportsNoRootCapacity.
if s.devicePresent(d.MountPath) {
if total, used, okc := statfsCapacity(d.MountPath); okc {
di.TotalBytes, di.UsedBytes = total, used
di.UsedFraction = float64(used) / float64(total)
}
}
out = append(out, di)
}
@@ -761,6 +768,11 @@ type FormatResponse struct {
// signature — the customer authorizes the wipe of their own data drive.
NeedsConfirmation bool `json:"needs_confirmation,omitempty"`
DurableID string `json:"durable_id,omitempty"` // the durable id to confirm against (user-data)
// FSUUID (on Formatted) is the UUID of the filesystem the agent just made, verified against the
// bound durable id after mkfs (R-25). The caller mounts THIS — re-resolving a UUID from the /dev
// path later can name another disk if /dev re-enumerated. "" = not verified: the caller must not
// substitute a path-resolved guess silently.
FSUUID string `json:"fs_uuid,omitempty"`
// PendingOp is set on a SYSTEM/BACKUP data-bearing refusal — the exact op the operator must sign.
PendingOp *PendingOp `json:"pending_op,omitempty"`
}
@@ -808,7 +820,7 @@ func (s *Server) handleDiskFormatStatus(w http.ResponseWriter, r *http.Request,
writeOK(w, map[string]any{
"vmid": vmid, "phase": job.Phase, "device": job.Device, "fstype": job.FSType,
"durable_id": job.DurableID, "error": job.Error, "started_at": job.StartedAt, "updated_at": job.UpdatedAt,
"job_id": job.JobID,
"job_id": job.JobID, "fs_uuid": job.FSUUID, // R-25: "" until done + verified
})
}
@@ -871,7 +883,7 @@ func (s *Server) handleDiskFormat(w http.ResponseWriter, r *http.Request, vmid i
"format refused (device may have changed since inspection): "+rerr.Error())
return
}
done := s.startFormatDetached(device, blankDurable, req.FSType, true)
job, done := s.startFormatDetached(device, blankDurable, req.FSType, true)
if err := s.awaitFormat(r.Context(), done, vmid, device); err != nil {
if err == errFormatClientGone {
return // client gone; mkfs continues detached + the job record records the outcome
@@ -880,7 +892,7 @@ func (s *Server) handleDiskFormat(w http.ResponseWriter, r *http.Request, vmid i
writeErr(w, http.StatusBadGateway, "format failed: "+err.Error())
return
}
writeOK(w, FormatResponse{VMID: vmid, Device: device, Formatted: true, DataBearing: false, DurableID: blankDurable, Reason: "blank device formatted " + req.FSType})
writeOK(w, FormatResponse{VMID: vmid, Device: device, Formatted: true, DataBearing: false, DurableID: blankDurable, FSUUID: job.FSUUID, Reason: "blank device formatted " + req.FSType})
return
}
@@ -917,7 +929,7 @@ func (s *Server) handleDiskFormat(w http.ResponseWriter, r *http.Request, vmid i
// F20-BUG3: run the destructive mkfs DETACHED off s.baseCtx (bound durable id recorded for
// restart-recovery), so a request/client deadline can never SIGKILL it mid-write and corrupt the
// disk. We still wait to return the synchronous result (backward-compatible with the controller).
done := s.startFormatDetached(device, deviceDurable, req.FSType, false)
job, done := s.startFormatDetached(device, deviceDurable, req.FSType, false)
if err := s.awaitFormat(r.Context(), done, vmid, device); err != nil {
if err == errFormatClientGone {
return // client gone; the wipe continues detached + survives a restart via the job record
@@ -929,7 +941,7 @@ func (s *Server) handleDiskFormat(w http.ResponseWriter, r *http.Request, vmid i
s.logger.Warn("local-api: USER-DATA data-bearing format — CUSTOMER CONFIRMED (no operator signature)",
"vmid", vmid, "device", device, "durable_id", deviceDurable, "fstype", req.FSType, "why", probe.Reason())
writeOK(w, FormatResponse{VMID: vmid, Device: device, Formatted: true, DataBearing: true,
Role: string(role), DurableID: deviceDurable, Reason: "customer-confirmed wipe (" + probe.Reason() + ")"})
Role: string(role), DurableID: deviceDurable, FSUUID: job.FSUUID, Reason: "customer-confirmed wipe (" + probe.Reason() + ")"})
return
}
@@ -5,6 +5,7 @@ import (
"encoding/json"
"io"
"log/slog"
"runtime"
"strings"
"testing"
@@ -184,3 +185,39 @@ func TestDisks_DevicePresence_WireFieldIsFalseOnDeviceLoss(t *testing.T) {
t.Fatalf("the drive never reached the wire: %s", body)
}
}
// ── R-118 — an absent drive must not advertise the ROOT filesystem's capacity ───────────────────
// TestDisks_UnionPath_AbsentDeviceReportsNoRootCapacity drives the REAL statfsCapacity (no capacity
// seam): the registry drive's mount path is a real, bare temp directory — exactly what /mnt/<name>
// becomes once its device is gone (a plain directory on the host's filesystem). With the device absent
// the row must carry NO capacity; before R-118 the union path statfs'd that bare directory and reported
// the host filesystem's size and usage as the drive's (46 GiB at 9.2 % for a 4 GB drive, measured).
// The present half proves the test is not hollow: the same directory DOES yield capacity when the
// device is there, so a zero on the absent half is the guard's doing, not a statfs failure.
//
// RED-PROOF: drop the `if s.devicePresent(d.MountPath)` guard around statfsCapacity in disks.go → the
// absent subtest fails with "advertises ... bytes".
func TestDisks_UnionPath_AbsentDeviceReportsNoRootCapacity(t *testing.T) {
if runtime.GOOS != "linux" {
t.Skip("statfsCapacity is linux-only; production target is linux")
}
bare := t.TempDir()
known := []storage.KnownTarget{
{Name: "cel", Type: hub.StorageTypeUSB, MountPath: bare, DurableID: "uuid:4242", UUID: "4242"},
}
t.Run("absent", func(t *testing.T) {
di := diskByMount(t, presenceServer(t, nil, known, true, false), bare)
if di.TotalBytes != 0 || di.UsedBytes != 0 || di.UsedFraction != 0 {
t.Errorf("absent drive advertises total=%d used=%d frac=%.3f — that is the filesystem UNDER "+
"the bare mountpoint, not the drive (R-118)", di.TotalBytes, di.UsedBytes, di.UsedFraction)
}
})
t.Run("present", func(t *testing.T) {
di := diskByMount(t, presenceServer(t, nil, known, true, true), bare)
if di.TotalBytes <= 0 {
t.Errorf("present drive reports no capacity (total=%d) — the guard over-corrected and the "+
"size bar is gone for every healthy registry drive", di.TotalBytes)
}
})
}
+6
View File
@@ -28,6 +28,7 @@ type fakeDiskOps struct {
unmountCalls []string
candidates []storage.CandidateDisk // returned by ListCandidateDisks
candErr error
afterFormat *storage.DeviceProbe // R-25: when set, InspectDevice returns it once a format ran
}
func (f *fakeDiskOps) ListCandidateDisks(_ context.Context) ([]storage.CandidateDisk, error) {
@@ -35,7 +36,12 @@ func (f *fakeDiskOps) ListCandidateDisks(_ context.Context) ([]storage.Candidate
}
func (f *fakeDiskOps) InspectDevice(_ context.Context, device string) (storage.DeviceProbe, error) {
f.mu.Lock()
p := f.probe
if f.afterFormat != nil && len(f.formatCalls) > 0 {
p = *f.afterFormat
}
f.mu.Unlock()
p.Device = device
return p, f.inspectErr
}
+96
View File
@@ -0,0 +1,96 @@
package localapi
import (
"context"
"encoding/json"
"net/http"
"sync"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/storage"
)
// R-25: the format answer carries the UUID of the filesystem the agent JUST made, verified against the
// bound durable id after mkfs, so the controller mounts that filesystem rather than whatever the /dev
// path resolves to a few requests later. The consequence asserted: the UUID on the wire (and in the
// polled job record) is the new superblock's — and is EMPTY whenever the binding cannot be re-proved.
const newFSUUID = "0fc63daf-8483-4772-8e79-3d69d8477de4"
func confirmedFormat(t *testing.T, d *fakeDiskOps, srv *Server, fj *FormatJobStore) (string, *formatJob) {
t.Helper()
w := do(t, srv.Handler(), "POST", "/disks/format", "A", `{"device":"/dev/sdb1","fstype":"ext4","confirmed":true,"durable_id":"byid:wwn-/dev/sdb1"}`)
if w.Code != http.StatusOK {
t.Fatalf("confirmed format: %d (%s)", w.Code, w.Body.String())
}
var resp struct {
Data FormatResponse `json:"data"`
}
if err := json.Unmarshal(w.Body.Bytes(), &resp); err != nil {
t.Fatalf("decode: %v (%s)", err, w.Body.String())
}
if !resp.Data.Formatted {
t.Fatalf("not formatted: %s", w.Body.String())
}
return resp.Data.FSUUID, waitFormatPhase(t, fj, formatPhaseDone)
}
func confirmedGate() *fakeGate {
return &fakeGate{decision: WipeDecision{Allowed: true, Tier: "customer_confirmable", Reason: "customer_confirmed"}}
}
func TestFormat_ReportsNewFilesystemUUID(t *testing.T) {
d := &fakeDiskOps{probe: deviceProbeDataBearing(),
afterFormat: &storage.DeviceProbe{Probed: true, HasFilesystem: true, FSType: "ext4", FSUUID: newFSUUID}}
fj := tempFormatStore(t)
srv := formatServer(t, d, confirmedGate(), fj)
got, job := confirmedFormat(t, d, srv, fj)
if got != newFSUUID {
t.Fatalf("response fs_uuid = %q, want the new filesystem's %q", got, newFSUUID)
}
if job.FSUUID != newFSUUID {
t.Fatalf("job record fs_uuid = %q, want %q (the polled status path)", job.FSUUID, newFSUUID)
}
}
// The node moved between mkfs and the read-back: the bound durable id now resolves elsewhere. The
// UUID must NOT be reported — reading it would name the other disk's filesystem.
func TestFormat_FSUUIDWithheldWhenDurableIDMoved(t *testing.T) {
d := &fakeDiskOps{probe: deviceProbeDataBearing(),
afterFormat: &storage.DeviceProbe{Probed: true, HasFilesystem: true, FSType: "ext4", FSUUID: newFSUUID}}
fj := tempFormatStore(t)
srv := formatServer(t, d, confirmedGate(), fj)
var mu sync.Mutex
calls := 0
srv.reresolveWipe = func(_ context.Context, _ string) (string, error) {
mu.Lock()
defer mu.Unlock()
calls++
if calls == 1 {
return "/dev/sdb", nil // the pre-mkfs anti-retarget re-resolve
}
return "/dev/sdc", nil // after mkfs: the durable id now names another node
}
got, job := confirmedFormat(t, d, srv, fj)
if got != "" || job.FSUUID != "" {
t.Fatalf("fs_uuid reported after the durable id moved (response %q, job %q) — must be empty", got, job.FSUUID)
}
if calls < 2 {
t.Fatalf("the post-mkfs re-resolve never ran (calls=%d)", calls)
}
}
// The superblock did not read back as the requested filesystem → not verified → empty.
func TestFormat_FSUUIDWithheldOnFSTypeMismatch(t *testing.T) {
d := &fakeDiskOps{probe: deviceProbeDataBearing(),
afterFormat: &storage.DeviceProbe{Probed: true, HasFilesystem: true, FSType: "xfs", FSUUID: newFSUUID}}
fj := tempFormatStore(t)
srv := formatServer(t, d, confirmedGate(), fj)
got, job := confirmedFormat(t, d, srv, fj)
if got != "" || job.FSUUID != "" {
t.Fatalf("fs_uuid reported for a superblock of the wrong type (response %q, job %q)", got, job.FSUUID)
}
}
+39 -3
View File
@@ -22,6 +22,10 @@ type formatJob struct {
Blank bool `json:"blank,omitempty"` // audit D3: blank (benign) format — recovery re-checks STILL-blank, not data-bearing
Phase string `json:"phase"` // running | done | failed
Error string `json:"error,omitempty"`
// FSUUID is the filesystem UUID of the NEW filesystem, read by the agent right after mkfs on the
// device the bound durable id still resolves to (R-25). "" = not verified (the controller must not
// read that as a UUID). Set only on phase done.
FSUUID string `json:"fs_uuid,omitempty"`
StartedAt string `json:"started_at"`
UpdatedAt string `json:"updated_at"`
}
@@ -96,7 +100,10 @@ func (s *FormatJobStore) save(j *formatJob) error {
// runs to completion and records the outcome. device is the ALREADY anti-retarget-resolved device; the
// record carries durableID so a restart can re-resolve + re-run. blank marks a benign (blank-device)
// format, so restart recovery re-checks STILL-blank rather than data-bearing (audit D3).
func (s *Server) startFormatDetached(device, durableID, fstype string, blank bool) <-chan error {
//
// The returned job may be read (FSUUID) only AFTER a value arrives on done — the goroutine writes it
// before the send, which is the happens-before edge.
func (s *Server) startFormatDetached(device, durableID, fstype string, blank bool) (*formatJob, <-chan error) {
base := s.baseCtx
if base == nil {
base = context.Background()
@@ -116,10 +123,39 @@ func (s *Server) startFormatDetached(device, durableID, fstype string, blank boo
ctx, cancel := context.WithTimeout(base, 60*time.Minute)
defer cancel()
err := s.disks.Format(ctx, device, fstype)
if err == nil {
job.FSUUID = s.formattedFSUUID(ctx, device, durableID, fstype)
}
s.finishFormatJob(job, err)
done <- err
}()
return done
return job, done
}
// formattedFSUUID reads the UUID of the filesystem the agent has JUST made (R-25). The caller used to
// re-resolve the UUID from the mutable /dev path afterwards, over separate requests — a re-enumeration
// in that window could hand it ANOTHER disk's filesystem to mount. Here the bound durable id must still
// resolve to the very device that was formatted (and re-derive to the same id), and the superblock must
// carry the fstype that was asked for; anything else returns "" (not verified), never a guess.
func (s *Server) formattedFSUUID(ctx context.Context, device, durableID, fstype string) string {
if durableID == "" || s.reresolveWipe == nil {
return ""
}
// The device now holds a filesystem, so the data-bearing anti-retarget re-resolve is the right one.
now, err := s.reresolveWipe(ctx, durableID)
if err != nil || now != device {
s.logger.Warn("format: new filesystem UUID NOT reported — bound durable id no longer resolves to the formatted device",
"device", device, "durable_id", durableID, "resolves_to", now, "err", err)
return ""
}
probe, err := s.disks.InspectDevice(ctx, device)
if err != nil || !probe.Probed || probe.FSType != fstype || probe.FSUUID == "" {
s.logger.Warn("format: new filesystem UUID NOT reported — superblock did not read back as the requested filesystem",
"device", device, "want_fstype", fstype, "got_fstype", probe.FSType, "has_uuid", probe.FSUUID != "", "err", err)
return ""
}
s.logger.Info("format: new filesystem bound to its durable id", "device", device, "durable_id", durableID, "fs_uuid", probe.FSUUID)
return probe.FSUUID
}
// finishFormatJob updates the persisted record to done/failed.
@@ -172,7 +208,7 @@ func (s *Server) RecoverFormatJob(ctx context.Context) {
return
}
s.logger.Warn("format-job recover: re-running interrupted format detached", "durable_id", job.DurableID, "device", device, "fstype", job.FSType, "blank", job.Blank)
_ = s.startFormatDetached(device, job.DurableID, job.FSType, job.Blank) // detached; updates the record on completion
_, _ = s.startFormatDetached(device, job.DurableID, job.FSType, job.Blank) // detached; updates the record on completion
}
// nowFn returns the server clock (testable), defaulting to time.Now.
+31 -17
View File
@@ -154,12 +154,15 @@ func (s *TokenStore) Mint(vmid int) (string, error) {
// looks it up; the per-candidate comparison is constant-time to avoid a timing oracle on the
// stored hash. ok is false for an unknown/empty token.
//
// Reload-on-miss (B3): the store FILE is shared across processes — the one-shot provisioner
// (`--selftest=provision`) Mints into it while the long-lived daemon serves Lookup from an index
// built at open. On a miss, re-read the file ONCE and re-check, so a token minted after this
// process started authorizes without a daemon restart (the drill's fresh-install 401). The
// append-only log makes an unchanged file size proof of no new records, so a genuinely unknown
// token costs at most one stat once the index is current — never a reload loop.
// Reload-on-change (B3, R-269): the store FILE is shared across processes — the one-shot
// provisioner (`--selftest=provision`) Mints into it while the long-lived daemon serves Lookup from
// an index built at open. Every Lookup stats the file first and re-reads it when the append-only
// log has grown, BEFORE answering — so a token minted elsewhere authorizes without a restart AND a
// token rotated out elsewhere stops authorizing on its very next presentation. (Before R-269 the
// re-read ran only on a MISS, so a superseded token was a direct map hit and kept authorizing until
// some unrelated miss forced the reload.) An unchanged size is proof of no new records, so the
// steady state costs one stat per call and never a reload loop. Pinned by
// TestTokenStore_RotatedOutTokenRejectedFirst.
func (s *TokenStore) Lookup(token string) (int, bool) {
if token == "" {
return 0, false
@@ -167,23 +170,34 @@ func (s *TokenStore) Lookup(token string) (int, bool) {
want := hashToken(token)
s.mu.Lock()
defer s.mu.Unlock()
// Direct map hit is the common path; the constant-time compare guards against a timing
// side-channel by re-checking the matched key (map lookup itself is not the secret-bearing
// comparison — the hash of a random 256-bit token is not feasibly guessable regardless).
if vmid, ok := s.byHash[want]; ok {
if subtle.ConstantTimeCompare([]byte(want), []byte(s.byVMID[vmid])) == 1 {
return vmid, true
st, statErr := os.Stat(s.path)
if statErr == nil && st.Size() != s.loadedSize {
// The log changed under us (another process minted/rotated): converge first, then answer.
s.reloads++
if err := s.reloadLocked(); err != nil {
return 0, false // unreadable store: fail closed, never crash the auth path
}
return s.matchLocked(want)
}
// Miss: skip the re-read when the append-only log has not grown (nothing new to see).
// A stat error falls through to the reload, which handles a missing file as empty.
if st, err := os.Stat(s.path); err == nil && st.Size() == s.loadedSize {
return 0, false
if vmid, ok := s.matchLocked(want); ok {
return vmid, true
}
if statErr == nil {
return 0, false // file unchanged since the last (re)load: genuinely unknown
}
// Stat failed (e.g. the file vanished): reload, which treats a missing file as empty.
s.reloads++
if err := s.reloadLocked(); err != nil {
return 0, false // unreadable store: fail closed, never crash the auth path
return 0, false
}
return s.matchLocked(want)
}
// matchLocked answers from the in-memory index. Direct map hit is the common path; the
// constant-time compare re-checks the matched key against the guest's CURRENT hash (map lookup
// itself is not the secret-bearing comparison — the hash of a random 256-bit token is not
// feasibly guessable regardless). Caller holds the mutex.
func (s *TokenStore) matchLocked(want string) (int, bool) {
if vmid, ok := s.byHash[want]; ok {
if subtle.ConstantTimeCompare([]byte(want), []byte(s.byVMID[vmid])) == 1 {
return vmid, true
+45
View File
@@ -210,6 +210,51 @@ func TestTokenStore_ReloadOnMiss_RemintCoherence(t *testing.T) {
}
}
// R-269: a token rotated out by ANOTHER process must stop authorizing on its very next
// presentation — with NO intervening lookup of the new token. This is the order the operator hits
// after rotating a leaked token: the leaked one is presented first. RemintCoherence above looks the
// NEW token up first, and that miss is what used to evict the old hash, so it passed while the leaked
// token kept returning HTTP 200 on hardware (2026-08-09) until something unrelated forced a reload.
//
// RED-PROOF: restore the reload-on-MISS-only Lookup (answer a map hit before stat-ing the file) and
// this fails with "rotated-out token still authorizes".
func TestTokenStore_RotatedOutTokenRejectedFirst(t *testing.T) {
path := filepath.Join(t.TempDir(), "tokens.log")
daemon, err := OpenTokenStore(path)
if err != nil {
t.Fatalf("open daemon store: %v", err)
}
defer daemon.Close()
minter, err := OpenTokenStore(path)
if err != nil {
t.Fatalf("open minter store: %v", err)
}
defer minter.Close()
old, err := minter.Mint(130)
if err != nil {
t.Fatalf("mint old: %v", err)
}
if vmid, ok := daemon.Lookup(old); !ok || vmid != 130 { // the daemon has learned the old token
t.Fatalf("old token before rotation: (%d,%v), want (130,true)", vmid, ok)
}
fresh, err := minter.Mint(130) // rotation, written by another process
if err != nil {
t.Fatalf("mint fresh: %v", err)
}
if vmid, ok := daemon.Lookup(old); ok { // the leaked token FIRST
t.Fatalf("rotated-out token still authorizes vmid %d on its first presentation after rotation — "+
"Mint's 'any previous token for this guest is revoked' is false across processes (R-269)", vmid)
}
if vmid, ok := daemon.Lookup(fresh); !ok || vmid != 130 {
t.Fatalf("fresh token after rotation: (%d,%v), want (130,true)", vmid, ok)
}
if vmid, ok := daemon.Lookup(old); ok {
t.Fatalf("rotated-out token authorizes vmid %d after the fresh one was seen", vmid)
}
}
// §8 edge: the store file deleted between open and a miss — reload treats it as empty; Lookup
// fails closed, no crash.
func TestTokenStore_ReloadOnMiss_MissingFile(t *testing.T) {
+2 -2
View File
@@ -60,8 +60,8 @@ func TestLiveReporter_CoordPresentWithoutPriorVerify(t *testing.T) {
if h.PBS == nil {
t.Fatal("pbs coord absent despite a reachable PBS — the gap this fixes")
}
if h.PBS.RepoID != "felhom-pbs" || h.PBS.Namespace != "root" || h.PBS.LatestSnapshotID != "9201" {
t.Errorf("pbs coord = %+v, want felhom-pbs/root/9201", h.PBS)
if h.PBS.RepoID != "felhom-pbs" || h.PBS.Namespace != hub.PBSRootNamespace || h.PBS.LatestSnapshotID != "9201" {
t.Errorf("pbs coord = %+v, want felhom-pbs, the root namespace (\"\", R-124), 9201", h.PBS)
}
// COMPANION (pre-fix): the bare SnapshotStore (no live read) with an empty store omits pbs.
+6
View File
@@ -57,6 +57,10 @@ type DeviceProbe struct {
HasPartitions bool `json:"has_partitions"` // child partitions present (lsblk)
Mounted bool `json:"mounted"` // currently mounted somewhere
FSType string `json:"fstype,omitempty"`
// FSUUID is the filesystem UUID blkid read from the on-disk superblock (`blkid -p`, no cache),
// "" when there is none. R-25: the format path reports it so the caller mounts the filesystem the
// agent just made, not whatever a /dev path resolves to later.
FSUUID string `json:"fs_uuid,omitempty"`
}
// DataBearing is the conservative verdict: any signature / partition table / partition / mount —
@@ -423,6 +427,8 @@ func (h *SudoHostOps) InspectDevice(ctx context.Context, device string) (DeviceP
probe.FSType = v
case "PTTYPE":
probe.HasPartitionTable = true
case "UUID":
probe.FSUUID = v
case "USAGE":
if v != "" {
probe.HasFilesystem = true // filesystem/raid/crypto member = data-bearing
+14
View File
@@ -208,3 +208,17 @@ func TestFormat_RejectsBadArgs(t *testing.T) {
t.Fatalf("mkfs ran despite invalid input: %v", r.calls)
}
}
// R-25: the probe carries the superblock's filesystem UUID so the format path can report the new one.
func TestInspect_ReadsFilesystemUUID(t *testing.T) {
r := &scriptedRunner{
outputs: map[string][]byte{
"blkid": []byte("DEVNAME=/dev/sdb\nUUID=0fc63daf-8483-4772-8e79-3d69d8477de4\nTYPE=ext4\nUSAGE=filesystem\n"),
"lsblk": []byte(`{"blockdevices":[{"name":"sdb","fstype":"ext4","pttype":null,"mountpoint":null}]}`),
},
}
p, _ := newSudo(r).InspectDevice(context.Background(), "/dev/sdb")
if p.FSUUID != "0fc63daf-8483-4772-8e79-3d69d8477de4" {
t.Fatalf("FSUUID = %q, want the blkid UUID", p.FSUUID)
}
}
+59
View File
@@ -0,0 +1,59 @@
#!/usr/bin/env python3
"""build-step-bundle.py — the TRANSITION bundle for a release whose bundle ADDS a path (R-880, `11` §5.4.2).
Usage: python3 scripts/build-step-bundle.py <base-bundle.json> <step-version> <out.json> (prints the sha256)
WHY. A box's INSTALLED felhom-os-apply checks every path of an incoming bundle against ITS OWN table (rule R16) — so a
bundle that adds a path (v0.146.1 adds felhom-priv-apply, the guest hook and the shared-parent files, R-861) is refused
by every box still running an older wrapper. The fix is a step: first a bundle the old wrapper accepts that brings ONLY
the new felhom-os-apply (the new table), then the release's own bundle, which the new wrapper accepts.
WHAT IT BUILDS. <base-bundle.json> is the bundle the boxes run now (download it from the package registry, e.g.
felhom-agent/0.145.0/felhom-config-bundle.json, and check its sha against the hub's record). The step bundle is that
bundle with EXACTLY ONE change: the /usr/local/sbin/felhom-os-apply entry's content is replaced by configs/felhom-os-apply
(this tree). Same paths, same modes, same checks, every other byte identical; agent_version is <step-version> (e.g.
0.146.1-step1). The old wrapper verifies it like any bundle (signature, sha, R16, content checks, self-check of the new
wrapper) — nothing about the trust route changes.
Pinned by configs/test_felhom_config_bundle.py (StepBundle): same paths as the base, only the wrapper differs, the
new wrapper's table is a superset of the base's paths.
"""
import base64
import hashlib
import json
import pathlib
import re
import sys
REPO = pathlib.Path(__file__).resolve().parent.parent
OSAPPLY_DEST = "/usr/local/sbin/felhom-os-apply"
def build_step(base_bytes, version, new_osapply_bytes):
if not re.match(r"^[0-9]+\.[0-9]+\.[0-9]+-[0-9A-Za-z.]+$", version):
raise SystemExit(f"build-step-bundle: {version!r} must be a semver with a step suffix, e.g. 0.146.1-step1")
base = json.loads(base_bytes)
files = base.get("files")
if base.get("format") != 1 or not isinstance(files, list):
raise SystemExit("build-step-bundle: the base is not a format-1 bundle")
hit = [e for e in files if e.get("path") == OSAPPLY_DEST]
if len(hit) != 1:
raise SystemExit(f"build-step-bundle: the base has {len(hit)} {OSAPPLY_DEST} entries, want exactly 1")
hit[0]["content_b64"] = base64.b64encode(new_osapply_bytes).decode()
hit[0]["sha256"] = hashlib.sha256(new_osapply_bytes).hexdigest()
base["agent_version"] = version
return (json.dumps(base, indent=1, sort_keys=True) + "\n").encode()
def main(argv):
if len(argv) != 4:
print(__doc__, file=sys.stderr)
return 2
data = build_step(pathlib.Path(argv[1]).read_bytes(), argv[2], (REPO / "configs" / "felhom-os-apply").read_bytes())
pathlib.Path(argv[3]).write_bytes(data)
print(hashlib.sha256(data).hexdigest())
return 0
if __name__ == "__main__":
sys.exit(main(sys.argv))
+7 -10
View File
@@ -10,13 +10,11 @@
"CI went red at a commit whose own run had been green the day before, on a true finding that",
"no one could act on. The red will return at the next publish unless the two read one number.",
"",
"HOW THE NUMBER WAS ARRIVED AT — stated honestly, because it is weaker than it looks.",
"generic_versions_kept is 10 because that is what the registry demonstrably holds today",
"(felhom-agent 0.121.0..0.128.0 = 10 versions, queried 2026-08-09). It is an OBSERVED state,",
"NOT a ruling anyone has been able to locate: no register row records a package prune, R-210",
"is WAITING-ON-OPERATOR and says 'Nothing was deleted; this is a list, not an action', and it",
"concerns local Docker images rather than this registry. Container packages currently hold 19",
"each, so there is no uniform ten-per-package cap visible either. See R-287.",
"HOW THE NUMBER WAS ARRIVED AT. generic_versions_kept is 10 because that is what the registry",
"keeps: the deleter was ESTABLISHED on 2026-08-10 (R-287) — the newest-10 prune of generic packages",
"run under R-267 (felhom-agent and felhom-golden generic held exactly 10 afterwards). Until then this",
"file called the 10 an observed state with no located ruling; that is superseded (corrected",
"2026-10-05, R-291). Container packages are not pruned by that rule.",
"",
"SO THIS FILE IS A FLOOR, NOT A LICENCE. It says: CI may assume nothing older than the newest",
"N generic versions is still downloadable. It does NOT authorise deleting anything, and the",
@@ -36,9 +34,8 @@
],
"generic_versions_kept": 10,
"readers": [
"scripts/check-published-versions.py — bounds its assertion to the newest N versions",
"documentation/runbooks/registry-retention.md (felhom.eu) — the prune procedure"
"scripts/check-published-versions.py — bounds its assertion to the newest N versions"
],
"recorded": "2026-08-09",
"recorded_by": "CC, from the registry's observed state; NOT from a located operator ruling"
"recorded_by": "CC 2026-08-09; the number's source (the R-267 newest-10 prune, established 2026-08-10 by R-287) recorded 2026-10-05 (R-291)"
}