687fedd8ee87a90e83530935eb7ea080e6d45754
462 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 admin
|
0a9bd3829d |
D5 SHIPPED: Tier-1/2 restore no longer depends on the whole-guest tier
Records controller v0.188.0 across the four coupled artifacts. 07-backup-architecture.md is the owning doc: - new 7.4 = the recovery chain AFTER D5 (7.1 leg 1 superseded; leg 2, the living-app dependency, explicitly unchanged so this is not read as more than it is) - 7.3 collapsed to history, with the correction that the target as written (data_key-only) was tested in Part 0 and rejected - 3 records that the two-lane split is now real, not just intended - matrix rows 3 / 3c (new) / 13; 10.1 D5 itself shipped Also: new capability-map row, D5 collapsed in ROADMAP + OPEN-ITEMS, and R-127 filed in both (data_key flag unreliable; O4 can regenerate a DB password that no longer matches the restored data directory). The audit is named D5-drive-alone-restore rather than "...secrets..." because .gitignore blocks *secret* -- a guard worth respecting, not forcing past. |
||
|
admin
|
d42d90fed7 |
R-108 CLOSED — D5's precondition is met (controller v0.187.0)
Four-artifact update per the coupling rule, plus the audit. 07-backup-architecture.md: §10.1 retitled CLOSED with the ruling and the D5 sentence; the FileBrowser network-share row flipped YES->NO, closed at the PLACEMENT rather than at the bind; the exposure chain annotated with the fifth surface (decommission-with-migrate guarded only its source) and the correction that the boundary is the deploy POST, not the dropdown; §7.3 retitled UNBLOCKED; register row collapsed; open question F answered. 00-capability-map.md: new §D row PROVEN-LIVE, with the un-exercised legs named — the deploy-POST and decommission refusals are unit-tested, not live-fired. OPEN-ITEMS.md: R-108 dispositioned; D5 given its OWN row as READY/UNBLOCKED (it had existed only inside other rows' prose — the R-123 thread-loss pattern); R-126 registered. ROADMAP.md: R-108 collapsed to a shipped one-liner; R-126 added. R-126 filed not fixed: a .fab bundle (plaintext secrets, optional password) can be exported ONTO a NAS. Split out of R-108 rather than folded in — it is an explicit customer-chosen export destination, not a browsing surface reaching a backup tree, so it was never part of D5's precondition. Live evidence: same-box before/after on demo-felhom through the real authenticated endpoint, the network-specific refusal on demo-hp, non-effect verified in the registry, and R-67's share-root bind diffed byte-identical across the deploy. |
||
|
admin
|
70f84941d4 |
R-106/R-109 audit + registers: shipped at agent 0.118.1, plus R-125
Adds the full audit: Part 0's three answers, the pre/post recipe for both boxes, the on-disk proof that `local` froze at the 2026-07-28 target move while felhom-backup kept running, all seven red-proofs, and the three publish observables. R-125 filed: v0.118.0's R-106 half shipped INERT. Two tests ran the real Collector.Collect() but both injected a fakeObserver, and the break was one layer below in mergeConfig, which dropped the pbs namespace. The recipe still said "root" — now with namespace_state "resolved" beside it, confident and wrong. Caught by live validation, not by the green suite. Fixed in 0.118.1; filed for the doctrine point that a production-path claim must name the seam it injects at. |
||
|
admin
|
acfc2b7e95 |
R-109 + R-122: the recipe assembly stops dropping sections (hub v0.83.0)
AssembleDRRecipe's hostHalfShape/appHalfShape are ALLOW-LISTS, not the forward-compat their comment advertised: a section an emitter adds is silently discarded until it is named in both the shape struct and AssembledRecipe. No error, no log, no failing test. R-122 (found this session): that already happened and shipped. The controller has emitted offsite_restic since fork-4 — the offsite recovery LOCATION — the hub stored it for all three real customers, and appHalfShape never listed the key, so no delivered recipe has ever contained it. It stayed green because the fixture drAppHalf is hand-written and omits the field. R-109: the agent's new backup_target is a new top-level host-half section and would have been dropped identically, making the fix read as shipped while changing nothing an operator can see. 3 tests built on halves read verbatim out of the live dr_recipe table, plus 2 red-proofs (each mutation asserted to have landed). vet rc=0, suite rc=0, 17 ok. Registers: R-106 + R-109 dispositioned; R-105/R-106 were READY in ROADMAP with no OPEN-ITEMS row (→ R-123, registered); R-124 filed on the "root" spelling. |
||
|
admin
|
3d504d58c8 |
docs(R-117): CLOSED — proven live on demo-hp; R-121 filed for agent-on-box drift
R-117 row → SHIPPED + PROVEN-LIVE (agent v0.117.0), with the full validation in audits/R117-v0117-2026-07-30.md. Both dead states detected on real hardware through the shipped predicate: RETURN raw 8:32 /dev/sdc | bind 8:16 shutdown → stale-device, usable false IN-PLACE both 252:11 emergency_ro, raw unit active → filesystem-aborted, usable false healthy → live 340-497us per call. No block I/O proven by strace (only /proc/self/mountinfo, 0 statfs) — the Part 1 CLAUDE.md fence applied to its own first consumer. No regression through the real pipeline: the live backup-target drive reads bound_under_parent=True via GET /disks with the controller's own credential, with 32 gate lines in 3 min as the positive observable and zero spurious transitions. The ruling asked for in §2.2 is recorded in full and flagged for overrule: Aborted must NOT self-heal. A re-bind lands on the same dead superblock and the call site runs every 20s, so repairing would be an infinite silent retry that masks the state. It surfaces instead. No operator decision was taken quietly — the reasoning is that it routes an already-broken state into the existing gate, event types and Hungarian copy, so no new concept reaches the customer. R-121 filed: a box's installed agent can sit releases behind the vouched one and nothing notices. demo-hp ran 0.113.0 against a vouched 0.116.0 through the whole R-116/R-117 arc. Confirmed at source that R-120's gate cannot catch it — it compares goldenVer against NewestReportedControllerVersion(), i.e. golden-artifact vs fleet-CONTROLLER. MinAgent is protective, not an alarm, and 0.113.0 equalled the floor. Fourth instance of the drift family. Also filed: R-117g (an aborted filesystem is never cleared automatically by design, so it alarms until a human acts, with no guided recovery) and R-117h (StablePathForRaw hardcodes the parent, so the repair path cannot be exercised on hardware without writing into a live customer guest's namespace). |
||
|
admin
|
37515cda7c |
docs(R-117 Part 1): a health check issues no block I/O — and narrow one R-116 claim
Two record items, banked before any Go file is opened. 1. CLAUDE.md gains a standing rule beside the seam-wiring rule: a health check issues no block I/O. A probe that touches a wedged device enters uninterruptible sleep, survives SIGKILL, and cannot be recovered until the device returns or the host reboots — so `systemctl restart` hangs too. A timeout protects the caller's control flow and nothing else. Liveness is decided from /proc and kernel state. Measured in the R-117 spike §6.3: D state 3m50s after kill -9; a buffered write with no fsync blocked too (O_CREAT needs journal access); statfs and getdents returned HEALTHY on a namespace that EIOs every byte. Repeated as a one-line pointer in felhom-agent/CLAUDE.md, because health checks are written in that repo and felhom.eu/CLAUDE.md does not load in an agent-only session — a standing rule that does not load where it binds is the inert-seam shape applied to a rule. 2. The R-116 row gains the clause the spike recommended but did not apply. Its verdict stands and every input to the pairing fix is configuration-derived. But the over-correction window's degraded:false was read off a drive whose bind was dead, so it evidences "the gate did not over-fire", not "the drive was healthy". The two RETURNED lines remain a genuine positive observable, so rule 3 is still satisfied. Nothing else about the row changed. |
||
|
admin
|
e70b5feebe |
docs(R-117): the hang case measured — an I/O probe turns a wedged drive into an unkillable agent
Completes the spike once the venue came back. Q4's hang case and teardown are now measurements, not plans. Against a dmsetup-suspended device (I/O queues instead of returning EIO): - P1 (devno compare) and P2 (ext4 abort flags) completed in 364us / 206us. They read /proc, so no block device is involved. - statfs and getdents completed and reported HEALTHY — on a wedged device they do not even hang. R-117b confirmed in a second failure mode. - EVERY probe that touches the device blocked, including a buffered write with no fsync: the O_CREAT metadata path needs journal access (wchan=do_get_write_access). There is no cheap-and-safe write probe. - The blocked process survived SIGTERM AND SIGKILL (stat=D, wchan=folio_wait_bit_common, still alive 3m50s after kill -9) and died only when the device was resumed. So `systemctl restart felhom-agent` would hang, leaving the agent unrecoverable until the device returns or the host reboots. The thread count does not reveal the leak (5->5, 5->6). Filed as R-117f. A timeout protects the caller's control flow and nothing else, so "the fix must issue no block I/O" is now a fence rather than a preference — the thread-leak hypothesis the probes were built to test turned out to be the weaker half of the result. Teardown done, all three layers: guest 9301 destroyed, r117scratch removed, both dm and both loop devices gone, scsi_debug unloaded, local back to 37.02% against a 37.00% session start. Fences re-verified AFTER teardown: 9201 running, drill-r50 stopped, local-lvm 38.84% byte-identical, felhom-backup content unchanged, live /mnt/felhom-drives intact with both submounts, agent active. Layer 3 genuinely empty — 9301 had no NIC and ran no controller. Trap recorded: a suspended dm device must be resumed BEFORE any umount, or the teardown blocks on the same uninterruptible sleep. |
||
|
admin
|
c949389c95 |
docs(R-117): spike — the mechanism, a recipe, and a steady-state half nobody had looked for
Both halves of the R-113 conjunction are path-presence tests: GuestSeesMount (intermediary.go:276) and isHostMountpoint (:394) compare field 5 of a mountinfo line and never read field 3, so neither can see that the bind and the raw mount name different devices. Measured BoundUnderParent=TRUE over a namespace that EIOs on every read and write. Reproduced 3/3 on a purpose-built scratch LXC on demo-hp; predicates evaluated by a throwaway probe calling the real localapi code from d4eb259. Three results that change the shape of the fix: - Q7: a bind can die in STEADY STATE with no detach/return cycle. The gate produces no action and nothing is emitted on any channel. A Return-branch fix cannot reach this half, and a devno comparison does not detect it. - Q6/R-117d: AttachDrive's normalize leg already performs the repair, and three call sites already invoke it - including the controller's Return branch before it restarts apps. All defeated by one early return at :235. Unblock the existing path; do not add a new one. - Q1: the device-node change is a CONSEQUENCE, not a precondition. The stale bind pins the dead superblock, forcing the returning device onto a new number. Control test: released, the letter is reused. Not established: the hang case. Venue and probes built, run lost to a site internet outage; the thread-leak hypothesis is not claimed as a result. Teardown of the spike venue is owed - commands in the findings doc; nothing fenced was touched and no hub-side record was created. |
||
|
admin
|
29bcfeb214 |
docs(R-120): CLOSED on both halves — golden current, and the class has a gate that refuses
Half 1, the artifact: golden 0.186.0 baked, published, vouched, and proven on a REAL day-0 on demo-hp (not the fixture, per the rule committed in Part 1). With the target detached, the fresh box's endpoint returned the TargetAbsent copy -- "A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod..." -- with offer_path absent entirely. The day-old read on the 0.185.1 golden had returned the false system-disk message plus an offer of the other drive. That is the customer-visible defect closed. Half 2, the mechanism: operator ruled REFUSE, shipped as hub v0.82.0 and DEPLOYED. Proven live by re-attempting the original mistake -- vouching the stale 0.185.1 golden now yields HTTP 303 flash=golden_behind_fleet plus [WARN] artifact vouch REFUSED, and the manifest reads back unchanged at 0.186.0. Refused AND unwritten, against the real fleet signal rather than a unit fixture. Recorded on R-29's audit list as the first ENFORCED gate beside its three orphans, so the contrast is kept rather than lost. The orphans are unchanged -- this proves the pattern is available, not that the backlog moved. Teardown all three layers: VM 9402 purged, r120-images removed with the space measured back, hub layer gate-blocked on ONLINE with the command recorded. Last session's sess-e was deleted this run, discharging its recorded layer 3. |
||
|
admin
|
1a68b53b06 |
hub v0.82.0 (R-120): the vouch path REFUSES a golden the fleet has already outrun
The golden's version IS the controller it bakes (build-golden.sh:345 defaults GOLDEN_VERSION to the controller tag), so a golden behind the newest deployed controller means every FRESH install lands on stale application code. On the R-120 occurrence that stale code shipped a customer-facing falsehood: a box from the 0.185.1 golden told a customer whose backup drive had fallen out that the backup was on the same disk as the system -- false, the drive was gone -- and offered a different drive as the remedy. WHY A GATE, NOT A REMINDER. The gap has opened three times: R-111 (golden's agent 17 releases behind), R-115 (agent built and deployed, never published), R-120 (this). The first two were closed by re-baking and remembering; remembering then failed again. R-29 is the standing proof that a check nobody runs is worse than none because it reads as coverage -- hostinstall_gates.py sat RED and uninvoked across three version bumps and hub_confirm_gate.py has never run at all. So the property that matters is not whether a check exists but whether it BLOCKS. - Wired into handleSetArtifacts (internal/web/configs.go), immediately before the only write, on the sole UI path to SetArtifactManifest -- it runs on every vouch without anyone choosing to. A script in scripts/ would have been a fourth orphan. - It REFUSES (operator ruling, 2026-07-30), with a flash naming the remedy. - Signal: store.NewestReportedControllerVersion() over reports.controller_version, SEMVER-compared in Go -- MAX() in SQL ranks 0.99.0 above 0.186.0, a pair this fleet has shipped. No outbound call, no new credential. - Fail-open in exactly two deliberate cases: an empty golden field (clearing the manifest is legitimate) and an unknown fleet version (a new hub must vouch its first golden). NEAR-MISS RECORDED: the first draft read guests.controller_version, a column that exists in the schema and that NOTHING writes -- it would always have seen "" and failed open, i.e. inert, this gate's own failure shape. Caught by grepping for a writer before trusting the column. Blind spot stated rather than papered over: a controller no box has ever run is invisible to this signal. Not the failure that has bitten -- all three instances were deployed-newer-than-baked. 4 tests through the PRODUCTION handler over httptest, never an injected seam. The refusal asserts both the flash and that the manifest was NOT written, because a gate that redirects and saves anyway reads as enforcement while providing none. Red-proof: deleting the block makes the stale golden vouchable and both assertions fail. ROADMAP R-29's audit list now records this as the FIRST enforced gate, so the contrast with its three orphans is kept rather than lost. The orphans are unchanged. Suite rc=0 read separately from this commit. |
||
|
admin
|
49b627684c |
docs(R-120): golden rebaked to 0.186.0, published, vouched, proven on a real day-0
The golden baked controller 0.185.1 -- confirmed from the golden's OWN record (drill/bake-0.185.1.log:1 and :330) and from build-golden.sh:345, which derives GOLDEN_VERSION from the controller tag. 0.185.1 predates R-114 + R-112, so every freshly installed box told a customer whose backup drive had fallen out that the backup was on the same disk as the system (false) and offered a different drive as the remedy. Baked golden 0.186.0 from main's controller in the DooPlex bake fixture: overlay2 OK, 3 mounts included, FATAL 0, exclusions 0, 618 MB, upload HTTP 201, GOLDEN_SHA256 b760ac6a33e70700..., token-leak grep 0, GL-1 teardown with drill.qcow2 back to virgin. Three observables, quoted as returned: PUBLISHED (anonymous GET -- what the installer does -- 200 / 648930639 bytes / sha identical to the bake); VOUCHED (manifest read BACK, not the 303); RESOLVED BY A CONSUMER (Artifact manifest served for customer sess-f, golden=0.186.0). Floor NOT touched per publish-train rule 2 -- it is a separate form and min_controller_version still reads 0.156.0. MinAgent left 0.113.0 because 0.186.0 declares it unchanged. Proven on a REAL day-0 on demo-hp, not the fixture, per the rule committed in Part 1: VM 9402 from the v1.25.0 ISO -> Controller elindult (0.186.0), box confirms felhom-controller:0.186.0 + agent 0.116.0. A fresh box now runs 0.186.0 where it ran 0.185.1. The procedure was NOT unwritten: RUNBOOK-manual-build.md:101-115 documents it and build-golden.sh carries its own usage and publishes to Gitea itself. One documentation-integrity finding: that runbook says to use the RECORDED qemu line and not reconstruct, while the line it cites is itself labelled reconstructed, the canonical one never having been saved. NOT done and not claimed: the TargetAbsent/empty-offer_path endpoint capture (the claim gate runs before auth with no Bearer escape -- R-119's fourth instance), and the Part 3 mechanism, which awaits the operator ruling. Recommendation and exact wiring recorded in the audit rather than built. VM 9402 + r120-images + customer sess-f retained pending that read, with teardown commands recorded. Previous session's sess-e layer-3 is now DISCHARGED -- it aged to STALE and the cascade completed, full residue purge logged. |
||
|
admin
|
376365bb12 |
docs(target-selection): a fixture may prove a mechanism; only a fresh box may prove a path
The page said which machine is safe to break but not when reusing a test box is legitimate. That distinction is exactly what surfaced R-120: R-116's closing run deliberately did a real day-0 from the ISO instead of reusing the standing fixture, and the fresh box installed the golden's controller -- a release behind -- and showed the customer the wrong absent-target message. A fixture would have shown a controller nobody installs. Adds to the Tier 1 section: a reusable snapshot-reset fixture is the right default for MECHANISM work (payload capture, fix cycles, claims about code behaviour), while a fresh day-0 from the ISO is REQUIRED for any claim about the install path, the golden image, agent publish/vouch or first-boot state -- naming the drift family it exists to catch (R-111, R-115, R-120). Also: a fixture must record its provenance (which golden, agent and controller, and when), because a fixture whose versions drift silently is R-120's mechanism turned into a permanent installation -- worse than no fixture, since it produces confident wrong results quickly. Part 1 of the R-120 task, committed alone and before the bake. Docs only. |
||
|
admin
|
772956d214 |
docs(R-116): CLOSED — proven live; capability row F to PROVEN-LIVE; R-120 filed
The events leg the previous commit reported as not-reached is now done. The operator relayed the claim code (the only route: bcrypt-hashed hub-side, emailed only), the two storage paths were registered through the real POST /api/storage/register, and the cycle ran on the fresh box: 07:20:04 backup_target_absent (error) Cel meghajto <- TARGET, specific 07:22:34 backup_target_restored (info) Cel meghajto <- its matching pair 07:24:04 storage_disconnected (error) Adat meghajto <- NON-target, generic 07:25:34 storage_reconnected (info) Adat meghajto All four at the hub; gate fired in 3 s. Two matched pairs, correctly discriminated -- and discrimination is proven NON-trivially for the first time, since both prior runs had the target itself emit the generic event. Over-correction passes on a positive observable, with two RETURNED lines proving the gate was ticking. 00-capability-map row F: PARTIAL -> PROVEN-LIVE with the evidence and the caveat. R-120 filed: the golden bakes controller 0.185.1, which PREDATES R-114 + R-112, so a freshly installed box shows the customer the WRONG absent-target message -- observed live on the drill box: the generic "the backup is on the same disk as the system" copy (false; the target is a drive that vanished) plus an offer of the other drive as the remedy. That is E2D 5.3's exact payload, still reachable on any new install. R-115's class one layer up -- R-111 closed by re-baking the golden, 0.186.0 then shipped, the golden did not move, and the gap reopened silently; this time the stale artifact carries a customer-facing falsehood in exactly the state R-116 now alarms about correctly. Teardown recorded for all three layers, hub layer gate-blocked with the command. |
||
|
admin
|
315c469fc8 |
docs(R-116): v0.116.0 proven live at the payload layer; events leg blocked on an emailed claim code
audits/R116-v0116-2026-07-30.md + the R-116 register row. WHAT PASSED, on real hardware. Agent 0.116.0 published (independent registry GET verified the bytes), vouched, and installed UNAIDED by a fresh box -- "Artifact manifest served for customer sess-e (agent=0.116.0 golden=0.185.1)", host sess-e-5d4427 ... 0.116.0 ONLINE. Real day-0 on a nested PVE on demo-hp (per runbooks/target-selection.md, which sent this run there rather than to the DooPlex fixture the previous run used), both drives enrolled through the real endpoints, device loss a real hot-detach. Captured live, absent state: the target is now ONE row carrying backup_target:true AND guest_path:/mnt/felhom-drives/cel with mount_path:"", so isTarget[/mnt/felhom-drives/cel] = TRUE -- it was false through v0.115.0. RETURNED gives true as well, so the pair matches. All three guards pass from the same payload: R-114 preserved (no row combines the flag with a non-empty mount_path), no over-correction (bound_under_parent:false), and discrimination at the payload layer (the non-target carries the flag on no row) -- the thing neither prior run could show. WHAT DID NOT HAPPEN, and is not claimed. No backup_target_absent or backup_target_restored event was observed on the wire. planDriveGates iterates registered StoragePaths and the drill controller has none ([WARN] Storage paths: no storage paths registered); every storage route answers 401 "dashboard not yet claimed". The claim code is bcrypt-hashed and emailed-only, and handleSelfBindLinkSend (selfbind_mint.go:139-161) renders a flash and never the token, so no operator-side route exists. A gen-2 code was re-sent; the drill VM, its storage and customer sess-e are DELIBERATELY RETAINED with teardown commands recorded, so the leg finishes without a rebuild. Reported as not-reached rather than as a third trivial pass. R-119 filed: the claim gate makes drive-gate legs unreachable to CC by design, and has now stopped three sessions at the same wall -- needs a ruling (operator-scoped test affordance, or a documented prerequisite step), not a fix. R-117 reproduced on real hardware with a read/write probe (EIO both directions while /disks reports attached + bound_under_parent:true) and §5 records how it colours the reattach leg. R-118's symptom vanishes incidentally on this one row; R-118 is NOT fixed. sess-c and sess-d verified GONE (404, absent from both tables) -- cleared by the operator using the previously recorded commands, not by this session. |
||
|
admin
|
1aa1bd17c2 |
docs(template): §13 gains a teardown step, §15 gains its evidence line
Three drills, three orphaned hub customers -- drill-r50, sess-c, sess-d -- because §13 covered the clean-tree gate, build/deploy, live validation and the STOP point and said nothing about teardown at all. Layers 1 and 2 (the VM and its volumes; the host's reclaimed space) get remembered because they are visible on the box. Layer 3, the hub-side customer or appliance record, is invisible from there and has been missed every time -- sess-c was not even recorded by its own report, so the record claimed a clean teardown that had not happened. §13: a Teardown subsection at the end, before §14. All three layers, with the hub layer requiring an EXPLICIT disposition -- deleted, retained as a fixture with the reason, or gate-blocked with the command recorded -- because silence is how drill-r50 became simultaneously a blocked customer and the only drift fixture. Cites runbooks/target-selection.md for which machine to provision on rather than restating it. §15: deliverable 8 demands the evidence for all three layers and names the failure it prevents; the former 8 (Observations) becomes 9. No section renumbered, §13/§15 not restructured, author checklist untouched. Part 1 of the R-116 join task, committed alone and before any Go file is opened -- the code half ends in a live run and live runs have stalled twice, while the record work is unconditional. |
||
|
admin
|
e6b5fa1e63 |
docs: retract the expired NVMe fence, drop component versions from the inventory, fence acts in the template
Follow-up acting on the observations filed with runbooks/target-selection.md. operations/nodes.md - The demo-hp NVMe was documented "PRESENT AND UNENROLLED -- do not touch" and listed under "What is NOT enrolled here (deliberately)". Both are FALSE and had been for eight days: it was enrolled 2026-07-22 through the normal Tarhely flow and is now /mnt/nvme-1tb -- the enrolled user-data drive AND the felhom-backup target (verified live 2026-07-30: nvme0n1 -> /mnt/nvme-1tb, and dir: felhom-backup / path /mnt/nvme-1tb / is_mountpoint 1). The fence's own condition (join via Tarhely, not the installer, not by hand) was SATISFIED, so the prohibition expired with it -- while still contradicting the task specs that correctly sent drill-VM disks there. Retracted with its reason recorded, and the caution that IS still live kept (dir storage at the mountpoint ROOT, else exactMount fails and the storage reads disconnected forever). - Component versions REMOVED and a note explains why: agent/controller/hub versions change several times a day, so a number written in an inventory is wrong within hours and then read as fact -- and the fleet is not uniform (on 2026-07-30 the two boxes ran different agent AND different controller versions). Points at the authorities instead: hub /hosts + /configs, felhom-agent --version, docker ps. - Site addresses now say re-check rather than asserting one (the N100 read .162, not the recorded .147); records that LAN literals are unreachable from DooPlex while the boxes are away. Adds the target-selection pointer: this page is what the hardware IS, that page is what may be done to it. PROMPT-TEMPLATE.md -- the upstream generator of the defect - Section 12's "Do NOT touch [the untouchable]" asked the spec author to name a THING. Now asks for the forbidden ACT plus its REASON, with the demo-hp case as the worked example of how a bare object-fence over-reads. - Section 13 gains the positive counterpart, which was the actual gap: if a task needs a machine to break, NAME IT. Listing only what is off-limits leaves the most valuable unfenced machine as the residual choice. runbooks/workspace-CLAUDE.md (+ the untracked root copy re-synced, verified identical) - Host table gains a Blast radius column and the missing demo-hp row, notes felhotest as Connection refused, and points at target-selection.md. This is the file that loads FIRST every session, so leaving it with the old table would have undercut the whole fix. No code, no build, no deploy, no host reconfigured or renamed. |
||
|
admin
|
699790b12d |
docs: write down which boxes are disposable (target selection by blast radius)
Nothing in the repo said which machines are safe to break. The host table gave access and role and stopped there, so a session needing a victim had to guess -- and the guessing inverted: the two boxes that exist to be broken were treated as sacred, and DooPlex (the recovery chain) got used because it was the only box no spec had fenced. New documentation/runbooks/target-selection.md -- one page, three tiers, and per machine what is freely permitted / needs care / forbidden, each carrying its REASON so a rule can be correctly narrowed later instead of ossifying. States the selection rule positively (start at Tier 0; a Tier 2 box only when a task says so explicitly; an absent fence is not permission) and that fences name ACTS, not machines -- demo-hp's over-subscribed local-lvm is one dangerous storage, not a dangerous box. CLAUDE.md: host table gains a Blast radius column, gains the missing demo-hp row (it was where the drill VMs ran and it was not in the table at all), and a pointer line to the new runbook. CORRECTION to the spec's problem statement: the designation was not missing. The 2026-07-25 operator ruling naming the t740 as drill+build VM host -- explicitly "moved off DooPlex" -- already existed in operations/nodes.md. It sat where no session reads at start, while the prohibitions were repeated in every task spec. The defect is reachability of the ruling, not its absence, and the R-116 drill on DooPlex contradicted a written ruling rather than filling a vacuum. CORRECTION to the R-116 record, same commit: the baseline claimed controller 0.186.0 on both demo boxes. Only felhom-pve was sampled and generalised; demo-hp re-checked directly runs 0.185.1, so the fleet is split and R-114's TargetAbsent branch is absent from demo-hp. Fixed in the audit table and REPORT-r116-diag. Docs only -- no code, no build, no deploy, no host reconfigured, no host renamed. |
||
|
admin
|
d56e395a2a |
docs(R-116): isolate the mechanism from the real /disks payload; file R-117 + R-118
The absent-state /disks payload was captured on a genuine device loss, after a present-drive control run proved the query works (Part 5's three attempts failed on token extraction, and its control returned 0 rows). The answer is theory #1 -- "the registry-union row writes false" -- which was raised, declared wrong and retracted. The retraction was the error. Absent state returns 4 rows, not 3. The drive appears twice and the two facts the controller needs sit on different rows: the Observe row has backup_target:true but mount_path:"" and guest_path:"", so it contributes no key to driveTargetByPath; the registry-union row owns /mnt/felhom-drives/<name> and omits BackupTarget from its struct literal (disks.go:301-306) => false. The union row is not deduped because seen is keyed on MountPath (:290-295), the one field the absent state empties, and its own MountPath comes from the systemd .mount unit FILE (registry_known.go:40-75), which never reads the mount table. Theory #2 (the basis of the shipped v0.115.0) is false on both halves; #3 is false too. v0.115.0 is provably inert: StablePathForRaw("") returns "". Also files the read path verbatim -- the token plaintext lives only in bootstrap.json on the Proxmox host; the agent's store keeps hashes only. New: R-117 (READY M, outranks R-116) -- a returned drive's guest bind is a DEAD mount (EIO both ways) while /disks reports attached + bound_under_parent:true, so the gate restarts the customer's apps onto it and reports healthy with no alarm. R-118 (READY XS) -- an absent drive's union row advertises the root filesystem's capacity as its own. Docs only. No code written, nothing built or published; v0.115.0 untouched. Both demo boxes read-only; drill fixture restored to virgin. |
||
|
admin
|
c3ce4c7b20 |
R-116 Part 5 FAILED: the fix shipped, C5 still fails, mechanism NOT isolated
A fresh box running the fully shipped stack -- agent 0.115.0 from the Day-0 manifest plus controller 0.185.1 from the vouched golden, no hand-deploy -- still fired the GENERIC storage_disconnected on detach and the SPECIFIC backup_target_restored on return. backup_target_absent count 0. Identical to Session C. The v0.115.0 fix changed nothing observable. Part 4's three positive observables were all obtained before the run (registry newest 0.115.0, hub vouches 0.115.0, felhom-pve running 0.115.0 clean), so the publish step forgotten twice was not forgotten a third time, and the box demonstrably installed the fix under test. Discrimination FAILS: the target itself produced the generic event, so the two cannot be told apart regardless of the non-target leg -- which was therefore not staged. Reported as a fail, not as Session C's trivial pass. Over-correction guard PASSES: 0 ABSENT lines with the drive present, target degraded:false. THE HONEST PART. The fix targets a shape that does not occur live, and which shape does occur is NOT ISOLATED. With the drive detached PVE reports the storage inactive with zeroed fields -- a shape the unit fixture did not model. Three attempts to read the real /disks payload failed on token extraction across the ssh -> guest -> container layers, and a present-drive CONTROL query also returned 0 rows, proving the query was broken rather than the payload. Without that control this run would have recorded a third false mechanism, after "the union row writes false" (wrong, corrected yesterday) and "no row carries the guest path" (unverified). The leading hypothesis -- an inactive storage reaching Observe with an empty MountPath, so StablePathForRaw returns "" -- is consistent with the pvesm output but is NOT evidence and is recorded as such. Next session's first job is a working /disks read, with a present-drive control run FIRST, before any further code. agent v0.115.0 is published, vouched and INERT. Not reverted: reverting is itself a change, the runbook forbids fixing mid-run, and the code is tested and harmless. Capability-map row F stays PARTIAL, now citing the re-test. Teardown clean: pvesm status after == before (local-lvm 38.83%), guest 9201 and drill-r50 untouched. Customer sess-d pending the usual ONLINE-ages-to-DOWN gate. |
||
|
admin
|
e87d6b26bb |
Correct the Session C audit: the union row is DEDUPED AWAY, not written false
The audit said the union row "writes false" for the guest-path key. That is wrong, and the next reader would have inherited the error. Isolated during R-116's Phase 0: RoleForStorage returns RoleSystem whenever backingDevice == "" (felhom-agent internal/storage/role.go:180-181). When the device vanishes the target row's role flips to system and it loses its guest path, but KEEPS its MountPath -- and the union loop skips any drive whose MountPath is already seen, so the registry row is never emitted at all. /disks therefore carries NO row with that guest path: isTarget[guestPath] is a MISSING KEY, not a false value. The practical difference is decisive -- the obvious fix (set BackupTarget on the union row) could not have worked, because that row does not exist in the state where the alarm is needed. The section's own "not isolated" caveat is replaced by the isolated answer. |
||
|
admin
|
952ebf4862 |
Record work, banked first: shrink the E-2d row, create the missing capability-map rows
Unconditional and three sessions overdue, so it commits before any code is
touched — E-2d itself stopped at Phase 0 and banked nothing.
E-2d row: 822 words -> 121, and the contradiction resolved. Its State read
CLOSED — PARTIALLY PROVEN while the cell's final sentence read "This row stays
OPEN only for the residue"; a reader could not tell which. It is CLOSED, with
R-116 the single named open leg.
Nothing unique was binned. Three facts existed ONLY in that cell and are moved
into audits/E2D-fresh-vm-2026-07-29.md as a new §1a: the local-lvm fence figures
with the 888 GB nvme alternative, the exactMount subdirectory caveat and why the
subdirectory is nonetheless the safe placement (no durable_id collision), and
the ISO/PAIRING -> DIRECT fall-through derived at source with its line
citations. drill-r50's blocked status was already in both audits.
Capability map: it had ZERO rows for the backup-target work — grep gives 0 hits
for backup_target and one for "E-2" that is a campaign date string. Three
scenario rows added, at today's honest status, not the value hoped for later:
C. Protection & recovery — installer Case A/B, DEGRADED recorded not hidden
PROVEN-LIVE, cites E2D-fresh-vm C1+C2
D. Storage & devices — the offer, and that registration confers no role
PROVEN-LIVE, cites SESSION-C C4 + the decline path
F. Notifications & monitoring — the absent-target alarm and its pairing
PARTIAL, cites SESSION-C C5, leg named, -> R-116
Row F is PARTIAL today per the doc's own strict enum (a leg not exercised live
is PARTIAL with the leg named, never PROVEN-LIVE). A later session may flip it;
this commit must not.
|
||
|
admin
|
06d7788392 |
Session C: R-113/R-114/R-112 PROVEN LIVE; C5 fails on a new defect (R-116)
Full ISO/PAIRING run on a fresh nested box. Agent 0.114.0 came from the Day-0 manifest -- the SHIPPED binary -- so C5 tested the real artifact. Controller 0.186.0 hand-deployed after install per the §3.1 ruling; the vouched golden bakes 0.185.1, so C3/C4 prove the code not the shipped golden, and that lag is filed against R-115 rather than a new ID. R-113 PROVEN: detach 18:43:50, gate fired 18:43:54 -- four seconds, where E-2d measured zero over 4.5 minutes -- and SetDisconnected was reached. It fired on exactly the shape that defeated it: raw /mnt/mentes NOT mounted while the bind /mnt/felhom-drives/mentes still read /dev/sdb[/felhom-data]. R-114 PROVEN: with the target absent the page rendered the absent copy, the system-disk copy 0 and the offer block 0. Both of E-2d's falsehoods are gone. R-112 PROVEN: the banner reached a customer's page for the first time. Healthy renders nothing, proven POSITIVELY -- idle delta 0 /backup/tiers calls, page load delta +1, single caller, so the seam ran and chose silence. C5 FAILED on a fourth, separate defect. The alarm fires but as the GENERIC storage_disconnected, while the recovery is the SPECIFIC backup_target_restored -- a pair an operator cannot match, which is what notifyDriveReturned's own comment forbids. backup_target_absent count 0 across the run. Root cause: the drive is TWO /disks rows and BackupTarget and GuestPath sit on different ones; absent they separate, on return they rejoin. v0.184.1 fixed the keying, not this. Only reachable because R-113 made the gate fire at all. Filed as R-116. Mirror + over-correction guard PASS: non-target drive -> storage_disconnected, backup_target_absent 0; both drives present -> 0 ABSENT lines and the target stayed healthy. Caveat recorded: the mirror passes trivially because the target also produced the generic event. E-2 and E-2d CLOSED as partially proven with R-116 the one named open leg, per the runbook's §9 rule decided in advance rather than mid-run. Capability map NOT touched: it has no E-2 rows at all, so nothing could move to PROVEN-LIVE. Creating them is a design act, not a validation act. Teardown clean: pvesm status after == before (local-lvm 38.78%), guest 9201 and drill-r50 untouched. Customer delete attempted and correctly refused while the host still reads ONLINE; command recorded for once it ages to DOWN. |
||
|
admin
|
af518ba151 |
R-114 + R-112 code shipped (controller v0.186.0) — seam proven live, copy not
R-114: new BackupTargetState.TargetAbsent separates configured-and-gone from never-configured. Degraded keeps its meaning so the wire contract is unchanged; TargetAbsent answers which problem, because the remedies are opposite. Copy is verbatim the hub's backup_target_absent email. The offer is suppressed on the branch itself, not left to firstOfferableDrive's Disconnected skip -- that flag comes from R-113 in another repo and this state must be right without it. R-112: the state finally has a consumer. Server-rendered on /backups via backupsHandler -> backupTargetView -> backups.html, not a 19th JS fetch. The view is nil for healthy and unknown so those render nothing at all. SEAM PROVEN LIVE by a DIFFERENTIAL positive observable rather than by an absent banner: idle 8s produced 0 new /backup/tiers agent calls; each /backups load produced exactly +1, and that call has a single caller. The demo box is healthy and correctly rendered nothing, which matches its real state but is a negative and so proves nothing about wiring on its own. MinAgent unchanged at 0.113.0 -- R-114 reads BackupTarget/MountPath/GuestPath/ Role, none of which R-113 altered. demo-hp is not held. Session C scope unchanged: neither fix touches the agent, so the leg awaiting proof is still device loss -> gate Stop -> SetDisconnected -> backup_target_absent on the wire. One rebuild validates all three. |
||
|
admin
|
338b2ccf86 |
agent 0.114.0 published + vouched; R-115 files the recurring publish gap
PART 1 — Session C unblocked. Agent 0.114.0 (the R-113 fix) was built, pushed and deployed but never published, so a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix. Published from the clean tree at b58d7bc via scripts/publish-agent.sh; sha 5e4c15ebee2d7583d57301d1f7c9cc7d4276262966bf738b05e34653bfd18c31, verified by an INDEPENDENT round-trip GET (http=200, sha match, binary self-reports 0.114.0), and the hub manifest read back after the write. Deliberately NOT done, each with a reason: - No golden bake. The golden bakes the CONTROLLER, not the agent, and host-install fetches them as separate generic packages (:1945 / :2573). Golden 0.185.1 is current, so there is no new-agent-against-old-golden risk. - min_agent NOT raised, stays 0.113.0. It expresses what the CONTROLLER requires of the agent, and controller v0.185.0 declares MinAgent 0.113.0 — which 0.114.0 already satisfies. Raising it to 0.114.0 would have been a false claim AND would have held demo-hp and drill-r50. No box is held; no §3 STOP fired. - Global controller floor NOT raised (v0.156.0), per R-111's reasoning. - wrapper_sha256 preserved verbatim; re-checked against configs/felhom-pbs-apply before and after — no drift both times. demo-hp RULING: left on 0.113.0. The R-113 fix is not live-validated, so putting it on a second box widens exposure for no proof, and Session C's nested box takes its agent from the manifest, not from demo-hp's host agent. Move the fleet once, after Session C. PART 2 — R-115 opened (WAITING-ON-OPERATOR). The finding is the RECURRENCE, not either instance: publishing is a remembered step, and it was forgotten within eight hours of R-111 documenting it as forgettable. Filed as a new ID with a back-pointer rather than reopening R-111, because R-111's finding (the channel WAS stale) is closed and verified end-to-end, while the process defect that caused it is a distinct problem with a distinct fix and owner. Class cross-linked to R-29 (a control that exists and is never walked) WITHOUT minting a second ID for it. Options are stated as the operator's decision, with mechanisms (build-step, deploy gate) separated from reminders (checklist, manual) — R-29's whole finding being that reminders do not hold. No code written, by design. R-111 gains a deferred-leg-recurred line; its shipped evidence is untouched and it is NOT reopened. R-113 records that Session C is now unblocked. |
||
|
admin
|
ca4c8b3afc |
R-113 code shipped (agent v0.114.0) — NOT live-validated, awaiting Session C
BoundUnderParent is now a CONJUNCTION: bound under the parent AND the drive's raw host mount still mounted. The raw mount is the device-bound systemd unit that dies with the device; the agent's own bind is not, which is why the bind outlived the device and the gate could never fire. Conjunction deliberately, not replacement: the device half alone would regress boot ordering (raw mounts early, bind lands ~18s later — that window must keep reading absent), so existing behaviour is byte-identical and only the unreachable case is closed. Unknown is never absent. Controller UNCHANGED, no MinAgent bump — BoundUnderParent has exactly one functional consumer (planDriveGates:226). A new DevicePresent bool was rejected: absent-from-JSON decodes to false, so every drive on an older agent would have read ABSENT and stopped its apps. +6 tests (208->214), 4 red-proofs run and reverted. Deployed to demo-felhom and the over-correction guard verified in production: raw mount present, drive still reads present, 10/10 apps untouched, no gate action, no false alarm. demo-hp deliberately left on 0.113.0 (the spec scoped deploy to felhom-pve). SESSION C BLOCKER recorded on the row: the hub Day-0 manifest vouches agent 0.113.0, so a fresh drill box would install WITHOUT this fix and validate nothing. Publish + vouch 0.114.0 first — R-111's trap in the same shape. |
||
|
admin
|
d839ddcb60 |
E-2d teardown complete: drill customer + host removed from the hub
The delete was correctly refused at four successive gates while the host still read ONLINE (acknowledgements -> typed confirm_id -> expect_hosts stale-preview -> "host is ONLINE"). Rather than force it, the run waited for the destroyed host to age to DOWN; delete-impact then reported deletable:true and the documented cascade ran: host deleted (escrow demoted to retained custody), tenantsync deprovisioned, PBS tenancy deprovisioned, claim reset to unclaimed, residue purged (reports=5 app_telemetry=5 notif_prefs=1 appliance_registrations=1) Verified after: 0 occurrences of "e2d" anywhere on the hosts page; demo-felhom and demo-hp ONLINE on agent 0.113.0; drill-r50 and peti-felhom unchanged; demo-hp carries only guest 9201 and VM 300. Scoping checked rather than assumed: the single purged appliance_registration was this run's own appliance (810d10c5, bound to e2d-fresh). The unrelated stale 2026-07-25 appliance (206c8838 / QWA-WJE) was NOT touched by the cascade — the operator removed it separately. - OPEN-ITEMS.md: the drill-cleanup WATCHING row is removed (done, not open). - audits/E2D-fresh-vm-2026-07-29.md §8 + REPORT-e2d.md: teardown recorded as complete, with the cascade output and the appliance-scoping note. |
||
|
admin
|
f3975cf5bc |
E-2d executed on a fresh box: C1/C2 proven, C3/C4 partial, C5 FAILS — R-112/113/114
Full ISO/PAIRING route on a nested PVE VM on demo-hp, after R-111 was fixed earlier in the session. Bind -> running controller in 3m35s. The install fetched the artifacts published an hour before and restored the golden baked 20 minutes before, so the publish train is proven end to end on a real install. C1 PROVEN: "felhom-host-install v1.22.0", "Day-0 provision SUCCESS", guest 9201 running, bootstrap unit wrote its done-flag and self-disabled. This retires E-2's "installer-logic-tested, not install-tested". C2 PROVEN: both DEGRADED warning lines verbatim, backup.local_backup_target=local, no felhom-backup storage created, and the install did not abort. C3/C4 PARTIAL and C5 FAILED — three findings, none fixed: R-112 (P1): E-2's degraded banner and offer have NO UI CONSUMER. The endpoint returns byte-exact copy; grep 'backup-target' across every html/js/css is 0 hits and no page handler injects the state. Templates fetch 18 distinct /api/storage/* endpoints; these two are the only ones with zero references. v0.185.1 fixed the router mount and stopped one layer short of the render. Fifth instance of seam-built-but-never-wired. R-113 (P1): the drive-absent gate CANNOT FIRE on device loss. planDriveGates reads presence from BoundUnderParent = "is this path in the guest's mountinfo". The raw mount is a device-bound systemd unit and dies with the device; the agent's own bind is not device-bound and outlives it, so the gate sees "present" forever. Live: agent reported the drive absent every 20s for 4.5 minutes, the controller logged 0 [gate] lines, the hub received zero events -- neither backup_target_absent nor the generic storage_disconnected. Sixth instance of the class: E-2b wired the seam to a condition that cannot occur. R-114: on target-drive loss the message claims the backup is on the system disk (false) and offers the drive that just vanished. Invisible only because of R-112, so it must be fixed BEFORE R-112 is wired. Also filed as a second instance under R-110 rather than a new ID: host-install fetches nine files from raw/branch/main and the hub vouches a sha for one; E-2a's wrapper is installed 0755 to /usr/local/sbin, root-fenced in sudoers, validated only by bash -n. C4 is fully proven at API level: decline path (registration confers no role), restart_required:true, agent did NOT self-restart (in-flight check performed and recorded first), E-2a wrapper created the storage at the drive's own mountpoint, and healthy renders nothing. Teardown: VM destroyed, scratch storage removed, pvesm status after == before (local-lvm 38.77%), guest 9201 and drill-r50 untouched. Hub records for e2d-fresh remain -- delete correctly refused at four gates, finally "host is ONLINE"; deletable once it ages to DOWN. Command recorded in OPEN-ITEMS.md. capability-map NOT touched: the customer-facing legs are broken rather than proven, and the map has no E-2 rows at all. |
||
|
admin
|
3dff3573f7 |
R-111 SHIPPED: the Day-0 artifact channel now serves agent 0.113.0 + golden 0.185.1
Found and fixed the same day. The channel was 17 agent releases stale — a box installed today would have received agent 0.96.0 and controller 0.161.0. - agent 0.113.0 built from the clean tree @ 58b598b and published via scripts/publish-agent.sh; sha 5f3247f756cb658e…, round-trip GET verified. - golden 0.185.1 baked on the nested drill VM embedding controller 0.185.1; sha dba00f3e845c415e…. Bake clean: Result=success, overlay2, all 3 mounts included (rootfs+mp0+mp1), 0 FATAL/exclusions, HTTP 201, token-leak grep 0. GL-1 teardown: guest 9100 purged, secrets shredded, drill disk restored to the virgin snapshot. Log saved to drill/bake-0.185.1.log. - Hub Day-0 manifest: agent and golden moved TOGETHER in one POST so the manifest never vouched a new agent against an old golden. min_agent 0.93.0 -> 0.113.0, which is what controller v0.185.0 declares. Zero fleet impact, verified: all three enrolled hosts already run agent 0.113.0. wrapper_sha256 preserved verbatim (re-checked, no drift). - The global controller floor was deliberately NOT raised: the golden now bakes 0.185.1, so a fresh box needs no self-update. This unblocks E-2d C3/C4/C5, which the Phase 0 gate had blocked. |
||
|
admin
|
f3f0d58844 |
E-2d: Phase 0 STOP — the Day-0 artifact channel cannot deliver the code under test
No VM created, no install run, no box touched. The run stopped at the Phase 0 gate per runbook §3, before provisioning. felhom-host-install.sh does not install what is on main. resolve_artifacts() (:423-436) reads the hub-vouched manifest and fetches Gitea GENERIC PACKAGES (agent :1945, golden :2573). Gitea's newest are agent 0.96.0 and golden 0.161.0; the hub manifest selects exactly those; the global floor v0.156.0 is below the golden's 0.161.0 so nothing self-updates. A fresh box therefore lands on agent 0.96.0 + controller 0.161.0 against main's 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by direct deploy and is not in the channel at all. Claim impact, each pinned to its introducing commit: - C1 (real rc=0 1.22.0 install) and C2 (Case B natural) — ACHIEVABLE, not run; both are installer-side and host-install is served at 1.22.0. - C3 — BLOCKED: banner + GET /api/storage/backup-target are controller v0.185.1 (cdaeb36), copy v0.185.0 (3f7cf2a). Unblocks cheaply by raising the hub floor to >=0.185.0; measured fleet impact nil (both demo boxes already 0.185.1). - C4 — BLOCKED: needs controller v0.185.1 + agent v0.113.0 (58b598b). - C5 — BLOCKED: needs controller v0.184.0 (c1a63de) + agent v0.112.0. Filed R-111 (P1): 17 unpublished agent releases (v0.97.0-v0.113.0) strand the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT, so a new customer's box installs without them. Mirror of R-110, not a duplicate. - audits/E2D-fresh-vm-2026-07-29.md — all four Phase 0 answers recorded so a resumed run does not re-derive them (cadence 30s; hot-detach available; ISO present; local-lvm fence re-measured at 38.77%, unchanged). - OPEN-ITEMS.md — R-111 opened; E-2d re-stated, NOT closed. - ROADMAP.md — R-111 under P1. - capability map NOT touched: nothing was proven live. The §5.1a operator STOP is retired — HUB_PW is in ~/.config/credentials and hub auth was verified, so CC can bind on a resumed run. |
||
|
admin
|
de5a3e5765 |
docs: retire the last two false gate-enforcement claims; scope the ranking heading
Closes the record-hygiene rider. Part 3 of the spec (documenting a ROADMAP/OPEN-ITEMS state convention) is deliberately NOT done — its stated evidence is false; see REPORT-record-correction-2026-07-29.md. - CONTEXT.md:540 — "scripts/hub_confirm_gate.py enforces" was present tense about a gate invoked by nothing. Now says it asserts but is not enforced (R-29). Third instance of the class after :564 and configs.go:27. - REUSE.md:62 — same claim, "enforces zero". The RULE stays (never native confirm()/prompt() is correct guidance and this is a reuse-reference row); only the enforcement claim changes, and it now says the rule holds only as long as you keep it. - OPEN-ITEMS.md:4 — root REPORT.md is the overwritten per-session file; REPORT-<topic>.md is the non-clobbering sibling form (CLAUDE.md:82-87), of which 14 exist. The prohibition on durable content living only there stays. - OPEN-ITEMS.md:55 — "Why the READY rows rank this way" promised a complete ordering and listed 5 of ~15 open rows. Scoped to TOP, with a half-sentence saying it is deliberately not a full ordering. No row added to the list. hub/internal/web/configs.go:27 left alone (R-94 leg (b), needs a hub build). No gate wired, run or fixed. Documentation only, no version bump, no CHANGELOG. |
||
|
admin
|
7383400a23 |
docs: file R-29 to the register; attach the gate-orphan instance to its class
|
||
|
admin
|
d4c07873ca |
docs: correct the installer-channel record — R-94 retracted and re-scoped, R-110 opened
The 2026-07-29 R-94/E-2d finding was written from an unverified claim and was false. `felhom-bootstrap.sh:96` fetches the installer from the WEBSITE, not the hub; the website git-syncs /scripts/ from main on a 30s period; every install since 1.22.0 hit main this morning already runs 1.22.0. Confirmed by live fetch. - OPEN-ITEMS.md: merge the two duplicate R-94 rows into one, retract the false framing, re-scope to what it actually is (a drifting hand-synced constant plus two pieces of dead safety equipment), unblock it from E-2d. - OPEN-ITEMS.md: de-rank R-94 in the ranked list — the "high-consequence" reason was the false claim in its most load-bearing form. - OPEN-ITEMS.md: E-2d — the ISO is the STRONGER proof route, not an obstacle. Phase 0 question answered at source: PAIRING falls through to run_direct in the same invocation (:495-499), so it reaches the identical installer call. - ROADMAP.md:149: same retraction; the original diagnosis (a hand-synced constant in a second repo drifts every time the first ships) survives. - ROADMAP.md + OPEN-ITEMS.md: new R-110 — main is the installer's publish channel and there is no staging, tag, pinned path or rollback, for the one artifact that runs as root on a virgin box. Operator ruling, not a defect. - day0-install.md C.1: one sentence recording the same about the fetch URL. Documentation only. No version bump, no CHANGELOG entry, no code, no box touched. |
||
|
admin
|
36d635a4cd |
E-2d: file the fresh-VM proof plan; R-94 blocked on it, with the ISO finding
Space checked on the t740 -- NOT a blocker, with one constraint: the VM disk must not go on local-lvm. That thin pool is over-subscribed (144G allocated against a 54G pool) on a box running a live customer guest, and a full thin pool corrupts every guest on it. local has 23.7G on pve-root. Use /mnt/nvme-1tb (888G free). Confirmed the ISO does NOT bake felhom-host-install.sh -- it ships felhom-bootstrap.sh, which fetches the installer FROM THE HUB. Since the hub serves 1.19.0, a fresh ISO install today would run the pre-E-2 installer and exercise neither Case A nor Case B. So R-94 must be bumped only AFTER a real 1.22.0 run, not before -- which is the ordering already decided. drill-r50 stays blocked and was restored to its r50pre state: the agent upgrade, the added disk and the moved backup target from this session are all reverted. |
||
|
admin
|
bcbe2707d6 |
E-2 complete: wrapper, installer Case A/B, offer flow, degraded banner
Live: hub 0.81.0, agent 0.113.0, controller 0.185.1 on both demo boxes; host-install 1.22.0 (script; no reinstall performed). E-2a wrapper proven live as root on demo-hp: F-1 subdirectory refused, F-2 unmounted path refused, root device refused, idempotent re-apply is a no-op, repointing refused -- 0 stray storages. The agent PVE role was NOT widened. Scenario E proven live on BOTH boxes: healthy renders nothing, no message key. Records three defects I introduced and caught: unreachable routes (mounted outside /api/storage/, caught by the first live call), a hollow test exposed by its own red-proof, and another gofmt-realignment no-op. Not live-proven: the degraded banner and offer acceptance (both boxes healthy), backup_target_absent end-to-end, Case A/B on a real install, drive-loss recovery. |
||
|
admin
|
3696188636 |
E-2 increment 1: report + close E-2b/E-2c as shipped and proven live
hub 0.81.0, agent 0.112.0, controller 0.184.1 live on BOTH demo boxes. E-2c: eject/decommission of the backup-target drive refused 409 on both boxes, drives unmoved. E-2b: the never-called disconnect seam is wired, with the target case raising the specific backup_target_absent. Records the keying bug caught before deploy (a.Path is the GUEST path, so the target branch was unreachable -- 0.184.0 superseded, never deployed) and states plainly that backup_target_absent is NOT proven end-to-end live: proving it needs a live enrolled drive to go absent. Parts 2/3/4 and E-2a remain open; Peti risk stays parked. |
||
|
admin
|
0fd0ca1373 |
PROMPT-TEMPLATE: defer is graceful-exit-only, not crash-safety
The crash-safety checklist listed "guaranteed cleanup via defer" alongside real crash-safety primitives, which reads as though defer covers a crash. It does not: a defer does not run on SIGKILL. Campaign 8 fault 10 proved this on live hardware. Crash-safe cleanup needs an on-disk marker plus a startup Recover(); defer covers graceful exits only. Both are useful, they are not substitutes. |
||
|
admin
|
2508788d38 |
E-2: file the remaining work, three Phase 0 findings, and the parked Peti risk
E-2 is partially shipped (hub v0.81.0 + controller Part 1). Filing the rest so a
foundation with no UI cannot quietly become a sixth seam-built-but-never-wired.
E-2 remaining: installer Case A/B, the offer + agent-side move, the degraded
banner, the controller half of the signal, red-proofs E/F, live validation.
Phase 0 INVERTED the emphasis: the installer has no drive-enrollment step,
so the common case at install is system-drive-only and Part 3 (drive added
later) is the PRIMARY path, not Case A.
E-2a the move needs a root-fenced wrapper -- the agent holds neither
Datastore.Allocate at /storage nor Permissions.Modify, and its sudoers has
no pvesm and no pveum. Use the guarded-wrapper pattern; do NOT widen the
agent's PVE role.
E-2b NotifyStorageDisconnected/Reconnected are defined and called NOWHERE, so a
drive going absent emits no event at all. Hub side is already plumbed, so
wiring needs no hub change.
E-2c E-1 put the whole-guest backups on a drive POST /disks/eject will eject
(RoleForStorage returns user-data for a local-dir on a non-system device).
Guard the eject specifically -- reclassifying the drive RoleBackup would
block legitimate ejects, since it is also the enrolled user-data drive.
PETI peti-felhom deliberately NOT migrated; drive failure there is offsite-only
recovery. Accepted until the operator's reinstall; re-evaluate if that
slips past ~2026-09-01.
|
||
|
admin
|
b5a73e050b |
Move the local whole-guest backup off the guest's own device (demo-hp + demo-felhom)
Supervised operational run. No code, no version bump, nothing deleted.
Primary backup tier on both demo boxes moved from `local` (a dir storage on
/var/lib/vz -- the SAME physical device as the guest) to `felhom-backup`, a dir
storage on each box's secondary drive:
demo-hp /mnt/nvme-1tb uuid:91d2dc2d-... archive 2,256,044,492 B
demo-felhom /mnt/hdd_1 uuid:47a3361a-... archive 5,957,878,962 B
Both proven end to end via the real UI path: archive lands on the secondary
drive (df delta matches the archive byte-for-byte), restore-test auto-selects it
and passes with mount_parity: ok, and freshness survives an agent restart with
an empty in-memory store -- so the age can only have come from the new storage.
Phase 0: the target is CONFIGURATION, not converged (the sole writer of
agent.json touches only escrow.pbs_storage_id and preserves unknown keys), so
the runbook's STOP did not fire. No consumer hardcodes "local" on the backup path.
Findings:
- F-1 the storage path must BE the mountpoint; a subdirectory fails exactMount
and the target reports disconnected permanently (observe.go:321)
- F-2 --is_mountpoint 1 is load-bearing; proven live, an unguarded storage on a
non-mounted path reports active with the ROOT filesystem's free space and
had already created dump/ on pve-root -- a silent retarget onto the very
device this change escapes
- F-3 FelhomAgentStore is granted per storage path; without it every backup
403s. felhom-host-install.sh must issue it for new installs
- R-109 (new) the DR recipe records no backup target, and each box now carries
two content=backup dir storages, one live and one frozen
- R-105 narrowed and TRACED: dr_recipe drives was [] fleet-wide because the
enrolled drives were never PVE storages, so isUserDataDrive never saw
them. Both boxes now populate drives; SMART on the backup drives too
Absent-drive behaviour today is fail-loudly with no silent retarget (PVE half
live-proven; agent half source-traced). That is NOT the intended fall-back-and-
alarm design -- filed as E-2 with the honest single-drive label.
Reported in full in the record: the agent was restarted with a felhom-pbs backup
in flight, producing a spurious tier failure. The backup had in fact succeeded
(PVE task OK, 6,264,034,053 B snapshot) and the spurious failure reached no
channel -- R-84 ground truth superseded it.
Outstanding: full drive-loss recovery (needs physical access) and the agent half
of the absent-drive behaviour.
|
||
|
admin
|
070b0ce2dc |
07-backup-architecture: rewrite as the recovery model (D1-D6); D5 BLOCKED by R-108
Replaces the 2026-07-14 DRAFT entirely - it was verified against controller v0.132.0, 51 versions stale, while being cited as authoritative. Now verified against controller 0.183.0 / agent 0.110.0 / hub 0.80.0 / catalog 4252121. NOT ratified. Every statement is labelled [DESIGN] (a decision from the architecture discussion) or [FACT] (file:line, live output, or a citation to _recovery-inventory-2026-07-28.md). Phase 0 verified D5's premise and it FAILED. Ten read surfaces were checked; nine are clean (SMB under both namespace shapes, FileBrowser for local drives, .fab import and download, /api/debug/*, all three ServeFile sites, registering the backup dir as a drive). One is not: FileBrowser binds a NETWORK share at its ROOT while local drives are userdata-scoped, and no IsNetwork() filter guards the deploy dropdown, the per-app migrate targets, or handleStorageMigrateApp. So an app namespace - and under D5 its plaintext secrets - can land somewhere a browsing surface serves with download:true. Not a leak today (the unit's app.yaml is secret-stripped); it becomes one under D5. D5 is therefore written as a BLOCKED target, not as settled. -> R-108 Core artifact is the failure -> recovery matrix (15 rows). RTO cells carry only measured durations; 12 are blank and each blank is listed explicitly as a finding. RPO cells carry the configured cadence that bounds RPO, labelled as such - no RPO has ever been measured from an incident and none is invented. The 7/53 vs 9/43/1 Tier-2 coverage counts are both recorded, unresolved; neither adopted. Coupling (S-1): capability map section C now cites the matrix instead of restating routes; ROADMAP gains R-102..R-108; OPEN-ITEMS gains only the in-flight R-108 plus C9-F1b/F3/F4 -> R-n cross-references; CONTEXT records D1-D6 as standing ruling S-3. No code, no version bump, no deploy, no box touched. |
||
|
admin
|
0d1cf0f732 |
Recovery baseline inventory (read-only): every restore path, tier coverage, key custody, host/hub state, failure domains, measured RTOs
Data document, no design. Parts A-G plus Unknowns/Contradictions/Observations. Headlines: - A: 12 restore paths. 7 are customer-unassisted (all app-level); every whole-guest and host-level path is operator-SSH-only. - B: independent catalog enumeration gives Tier-2 file-leg coverage 7/53 and Tier-3 mandatory coverage 4/53; does not reproduce the shipped C9-F1 figure of 9/53. Tier-3 captures named-volume tars on every run and no offsite action unpacks them. - C: three circular dependencies, incl. app-tier restore being conditioned on the whole-guest tier (encryption.key), and break-glass credentials reachable only through the hub that DooPlex hosts. - D1: host state is in no backup of any kind; no host has ever been rebuilt as itself. - D2: hosts.dr_record_json, host_escrow.directive_json and dr_recipe drives[] are all empty on the live fleet, and dr_recipe records the PBS namespace as 'root'. - E: local vzdump shares its physical disk with the guest it backs up (both hosts); hub.felhom.eu resolves straight to DooPlex's own public IP, not a CF tunnel. - F: every measured RTO extracted; the unmeasured ones listed, none estimated. Nothing was changed, deployed or restarted; peti-felhom untouched. |
||
|
admin
|
ff050cf409 |
C9-F1 (honesty half) + C9-F2 shipped in controller v0.183.0; C9-F1b and C9-F4 filed
Phase 0 sized C9-F1 properly before anything was designed: 43 of the 53 catalog apps have NO subtree the Tier-2 restore can read (not 2), 9 are covered only for their file legs and never their database or volumes, 1 is stateless. The asymmetry is Tier-2's alone — Tier-1 and offsite both restore the unit and replay volume dumps, so BookStack always had a working restore and only this button lied. Shipped: the restore refuses BEFORE stopping the app and names the action that does work; a run that proceeds claims only what it EXAMINED and discloses that the database and volumes are not covered. C9-F2 alarms after a 5-minute sustained-restarting threshold, set above the 120s deploy timeout, Mealie's 60s start_period and R-97b's 180s grace; StateRestarting is deliberately NOT added to IsDownState. Live: silent through ten 30s samples then app_start_failed at 5m25s, heartbeat now reads "1 currently down" where Campaign 9 recorded 0; a real deploy stayed silent; bookstack refused with its uptime unbroken; paperless re-restored 43/43 byte-identical, 16/16 docs clean. Filed, not fixed: C9-F1b (route to the Tier-1 restore — its own task because it puts a destructive operation behind a non-destructive button) and C9-F4 (nothing reads the Tier-2 copy's recovery-unit/ mirror, so the second local copy that exists for drive loss is unreachable by any customer action — potentially larger than C9-F1). |
||
|
admin
|
955083c0fc |
Campaign 9: the Tier-2 restore paths are PROVEN; 3 defects filed, none fixed
Phase A is the headline and it passed on live hardware, through the real endpoints the UI
posts to: a customer who deletes files — or their entire app data directory — gets everything
back byte-identical, and the app works afterwards (paperless served the restored bytes over
its own API at the exact pre-deletion sha256). A1's two non-destruction promises both hold.
Three defects, recorded not fixed:
C9-F1 (HIGH) the Tier-2 restore button is offered for apps it can never restore, takes a
real outage, and reports "nothing was missing" — indistinguishable from a
genuine result, while 156 MB of that app's data sits unread in the same copy.
C9-F2 (HIGH) an app in a crash loop never alarms on any channel; StateRestarting is in no
down-set, so F-OBS's own heartbeat printed "0 currently down" for 9 minutes.
C9-F3 (MEDIUM) an interrupted offsite run leaves a lock the self-heal cannot reach; the tier
is dead until a human unlocks and the operator is told "unknown reason".
This answers Phase C item 8.
Two candidates were deliberately NOT filed: a recovery-unit poisoning the catalog sync healed
in ~3 min, and a snapshot_id that is documented as logging-only. Reporting either would have
been reporting an artifact.
Stopped at the end of Phase B (plus D10), then full recovery — both boxes healthy, real
cadences, offsite tier proven working again, no leaked scratch guests, peti untouched.
D11's approved staleAfter compression turned out not to be a knob; reported, not worked around.
|
||
|
admin
|
f47b0a61d7 | R-101 + F-DIAG closed, F-OPS documented (manual-restore runbook) | ||
|
admin
|
6b7d516ce5 | REPORT + OPEN-ITEMS: R-100 closed and proven live; R-101 filed | ||
|
admin
|
b505ee9125 |
R-100: offsite staleness counts from the last SUCCESS (hub v0.80.0)
isStale counted from last_run, written unconditionally on failure, so a nightly-failing tier read as fresh forever. Now anchored on last_success with an explicit legacy degrade (logged once) and the never-ran branch untouched. emitStale states the real reason. |
||
|
admin
|
5ffdcecb2d | F-LEAK: record all three attempts; the ACL fix is consumed by use, the fenced destroy is the fix | ||
|
admin
|
8c793955b1 |
Campaign 8: F-REBOOT/F-LEAK/F-OBS closed; R-100 filed (offsite staleness reset by failures)
F-LEAK's first fix (pool adoption) was refuted live and is recorded as such. R-100 is F-CRIT-2's defect class on the hub for the restic tier: isStale reads only LastRun, which the controller writes unconditionally on failure. Investigated, not fixed. |
||
|
admin
|
e168600148 |
docs: F-CRIT-1 + F-A1 shipped (controller v0.179.0); invariant rule
Both marked SHIPPED + PROVEN-LIVE in OPEN-ITEMS and the campaign doc. All three of Campaign 8's alarm findings are now closed (F-CRIT-1, F-CRIT-2, F-A1). Adds the standing rule earned by this arc to the versioned workspace CLAUDE.md: a comment asserting an invariant needs a test pinning it, or it is a wish — with all six shipped-false-guarantee instances catalogued, and the corollary that a test should assert the CONSEQUENCE (does the alarm fire?) not the MECHANISM (does suppression expire?). |
||
|
admin
|
d0cec9d3f3 |
docs: F-CRIT-2 shipped (agent v0.106.0) — OPEN-ITEMS, campaign doc, report
F-CRIT-2 marked SHIPPED + PROVEN-LIVE: NewestArchiveTime now counts only plausibly-complete entries (measured 1 MiB floor), verified by replaying campaign fault 2 on demo-hp in both directions. Files two follow-ups: - R-99: prune never REMOVES phantoms. It does NOT count them toward keep-last (dry-run kept 2 real + the phantom), so there is no retention/data-loss bug — but they accumulate one per aborted upload. - F-CRIT-1: filed READY-HIGHEST. Campaign 8's other HIGH finding, untouched by this fix. |
||
|
admin
|
33a7cc51d3 |
docs: CAMPAIGN 8 — GC run 2, fault 13 complete, final fleet state
GC run 2 (04:44Z, past the atime boundary) reclaimed 0 B, and the cause is the campaign itself: the datastore is relatime, so the night's three full verifies, ~14 backups, two GC runs and several restores kept refreshing chunk atimes. Sampled atimes confirm no chunk was ever 24h5m cold. Not a defect - GC made a correct, deliberate decision both times - but a real operational property: on a relatime datastore, activity defers reclamation. The spec's intended experiment therefore could not complete inside the window; recorded as such. Fault 13 now complete: backup-vs-restore-test single-flight holds, and a full two-tier backup ran successfully DURING GC run 2 (both artefacts landed, GC TASK OK, zero .bad chunks). True three-way overlap remains untested. Final fleet verification clean on all three hosts. Campaign-caused outage (F-REBOOT, 9m47s on demo-hp) stated plainly in the final-state section. |