8ef92a3fa7d7ed946529628a7ff77ba446abef98
194 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 admin
|
ad28699761 |
docs: R-157 A / R-170 / R-171 closed — boot recovery finished
gates / gates (push) Successful in 7s
Controller v0.190.0. Docs only here; no hub change, no hub version bump. - audits/DIAG-bootrecon-drive-absent-2026-08-02.md — NEW. The Part 0 diagnosis, including the run that produced a FALSE NEGATIVE and the mechanism behind it (the agent re-binds an unmounted drive within ~60s, so the drive gate's startup reconcile restarted the apps one second before the sweep looked). Records that the write hazard was blocked only by an ACCIDENTAL filesystem permission that no code owns and no test pins. - architecture/02 §0a — the boot-recovery contract (S-1): both gates read desired state; the sweep observes a SETTLED fleet and each sample must refresh first; nothing is started without asking, fail-safe. Plus the durable warning: Manager.StartStack has no gate of its own. - 00-capability-map — the boot-recovery row, with the repeat count cited per N.5 (6 of 6 hard resets) rather than a bare PROVEN-LIVE. - OPEN-ITEMS / ROADMAP — R-157 CLOSED (both mechanisms), R-170 CLOSED, R-171 NEW and closed the same session, marked a regression from v0.189.0. - STATUS.md — the power-cut line moved from "What's broken" to "What works right now" with its repeat count; one dated bullet in the change log. - CONTEXT.md S-13 — the lessons worth carrying: "it didn't happen this time" is not a disproof; widening a window makes previously-unreachable overlaps reachable; and a settle detector is only as good as the freshness of what it samples — the fix's own defect, found live rather than by review. |
||
|
admin
|
c718aad1bc |
docs: R-168 SHIPPED, R-29 CLOSED on the demonstrated alarm, R-169 minted
gates / gates (push) Successful in 7s
SPIKE-ci-runner-2026-08-02.md: all six probes with method, measurement and ruling; none STOPped. P2 (stock image has git but no python3) and P6 (a runner that loses its state re-registers and orphans the old record) changed the design; P5 (a failed run signals NOTHING) is why the alarm exists at all. R-168 SHIPPED with its evidence. R-29 CLOSED — on the demonstrated alarm and not on a green run, as required: the class it opened is answered at both ends, the hook refusing locally and CI catching a --no-verify bypass and emailing. R-161 noted: its automatic half now exists for the STATIC gate, while its original scope, the runtime gate, is deliberately still not automatic and should stay that way. NEW R-169 (grep established R-168 was the highest in use): CI can only report, because there is no gate in the road. Making it blocking needs branch protection plus a PR workflow, both of which change how the operator works — so it is theirs to decide, and the row states the cost honestly rather than recommending it. CONTEXT gains S-8 (CI detects, does not block, and why that is structural), S-9 (a detector that tells no one is not finished, plus the curl and Cloudflare-1010 traps), S-10 (the runner is unprivileged because DooPlex is Tier 2), S-11 (CI reproduces the sibling layout). CLAUDE.md gains the rule earned by red-proofing: a go test -run pattern that matches no test prints ok and exits 0, and an instrument that can silently drop results is not a measurement. |
||
|
admin
|
482af37b7d |
Campaign 10 closeout Part 2: teardown complete — five layers, each verified gone
Evidence-survival check FIRST: HEAD == origin/main ==
|
||
|
admin
|
7efb7a53d3 |
Campaign 10 closeout Part 1: Q1 lowers R-158's rank; Q2 clears ValidateDump and kills C2's gate
Q1 - what the customer sees when a backup refuses for lack of space. The failure
IS customer-visible: /backups renders "Adatmentés sikertelen" with a cross mark.
It is absent from the dashboard, the launcher, the app detail page, and - the one
worth fixing - from /backups/apps, the per-app page where you would naturally ask
whether a given app is backed up.
Point 5 measured across three runs: it retries, stays failed while constrained
(marker persists, unit mtime unchanged at 07:30:50), and clears on recovery with a
fresh unit at 07:37:38. /backups/apps reading "Utolsó: 3 perce" tracks the unit's
REAL mtime, not the failed run, so it is honest about the age of the last good
unit rather than claiming a fresh one. Explicitly NOT the R-156 family.
So R-158 is a NOTIFICATION GAP, not a silent-failure defect, and ranks BELOW
R-157 - whose mechanism B leaves a deployed app not running while deadapp reports
"0 currently down", silent on every channel.
Q2 - ValidateDump was right and no bad dumps are shipping. The live DB genuinely
had zero accounts (only _prisma_migrations 129, cc_proof 82, instance_settings 1).
An empty table proves nothing, so an account was SEEDED as the task required: the
warning then stopped entirely and the dump provably contained the rows (c10acct 1,
c10user 2; 102766 -> 103029 bytes).
But that kills C2's proposed ordering. A fresh appliance legitimately has zero
accounts, so gating on "accounts has rows" would block the backups of every new
customer until someone registers. The validator's fact is right; its inference
("may predate the customer's data") is wrong - there was no data to predate. The
chain is therefore longer: a sound predicate first (compare the dump against the
LIVE db, per-table counts, not an absolute expectation), then warn->gate, then the
tar-drop. Until then the DB volume tar stays load-bearing - not because dumps are
bad, but because nothing can yet prove one is good.
No new R-n; register grepped. Nothing fixed. Part 2 (teardown) follows.
|
||
|
admin
|
0afadbdeff |
SPIKE: recovery-unit space — the ceiling is real on mp1, overflow is clean, but silent (R-158)
Three headline answers.
1. The ceiling is REAL and on mp1 (/mnt/sys_drive), but its shape is a MISMATCH
rather than a single number. A1: docker's data-root is a SEPARATE 50G volume
(mp0) and every app volume resolves there, so app DBs are NOT on sys_drive -
build-golden.sh:68's "like the Docker-data" reading is correct. A2: the
recovery units ARE on sys_drive, which the golden ships at 20G. So a box
permits 50 GB of live app data while capping local backup at 20 GB, and
crossing that line is invisible until a backup fails.
A3 rules out the lab-default explanation: --sysdata-grow defaults to 0
(main.go:178) and is not computed from the drive. demo-hp's REAL guest 9201
runs a bare --config ExecStart and shows mp0 50G / mp1 20G; agent.json has no
sizing keys at all. A4, measured not read: restore extracts IN PLACE on the
docker volume - sys_drive avail was 799.2M before and after a restore run under
constraint - so the constrained mount is written only during backup.
2. Overflow behaves WELL. With sys_drive ballasted to 799 MB, backup refused
per-app ("No space left on device"), other apps continued, status reported
success=false, and the "last good dump preserved" claim VERIFIED byte-for-byte:
size and md5 unchanged, tar valid end-to-end, no .tmp residue. Restoring that
preserved unit under the same constraint returned correct data and claimed
success honestly. Explicitly NOT the R-156 family.
3. But it is SILENT - R-158, filed. Zero events reached the hub.
NotifyBackupFailed exists and the hub allowlists backup_failed, but the only
production caller is the off-box/NAS leg (main.go:659); the backup manager has
tier2/offbox/offbox-enlarge notify seams and none for the local recovery-unit
capture. This is R-97's shipped defect exactly one tier over, and the fifth
instance of "seam built but never wired" - a pattern the codebase names in its
own R-97 wiring test.
Sizing rule corrected: unit ~= volume-tar bytes + logical dump bytes, not a
constant 1.90x. Measured C1: file-only apps are 1.00x (homebox 2305->2305 MB, no
db-dumps dir at all), and the SAME DB app with an empty DB is also 1.00x. So a 20G
sys_drive holds ~19 GB file-only or ~10 GB DB-backed. That is the bound on D5's
Lane-1 independence.
C2: both representations are used for a reason stated in code (F17 - the dump is
authoritative and WINS over the tar; R-47 - replayed with only the DB service up).
The dump is single-database pg_dump --no-owner, so a fresh initdb plus the dump is
logically sufficient and the tar is a PHYSICAL FALLBACK. Dropping it would halve
DB-app units and also close the D5/R-127(b) password trap (restored PGDATA makes
postgres skip initdb and ignore POSTGRES_PASSWORD) - but only after ValidateDump
is promoted from a warning to a gate, since it currently WARNS on a dump whose
accounts table has no rows. In its present form the tar is load-bearing.
No production code, no template change. Teardown still owed and itemised.
|
||
|
admin
|
5f35aa0346 |
Campaign 10: M-band RTO measured — RTO ~= 40s + 26.9s/GB, and a capacity ceiling that matters more
The S figures (66 MB -> 42.0s, two passes agreeing to 0.6s) had a spread tight enough to prove fixed work dominates, which is exactly why they said nothing about M. Second point taken 327x larger, same app, same method: clock from restore request to the app serving the correct discriminator. rallly's postgres volume grown 66 MB -> 21.1 GB (200k rows, STORAGE EXTERNAL so TOAST cannot compress it into a fake number). Two reps: rep 1 backup 406.4s unit 41149 MB RTO 624.5s discriminator correct rep 2 backup 387.2s unit 41133 MB RTO 591.8s discriminator correct 327x the data cost 14.5x the time - strongly sub-linear: RTO ~= 40s + 26.9 s/GB backup ~= 29s + 17.4 s/GB 10 GB -> 5.2 min 20 GB -> 9.6 min (measured 10.1) 100 GB -> 46 min The fixed ~40s dominates below ~1.5 GB, which IS the S band and explains its tight clustering. The more consequential result is capacity. A DB-backed app's recovery unit is 1.90x its data (volume tar PLUS SQL dump): 21.1 GB produced a 40.2 GB unit. The default appliance ships /mnt/sys_drive at 20 GB, so the largest app that can hold a local Tier-1/2 recovery unit on a default box is about 10 GB - and that fills the volume. The M band does not fit on a default box at all; this test only reached 21 GB because sys_drive was first grown 20G -> 70G with the same operation the product performs via SysDataGrowGB. A tier-sizing decision, not a defect, but it is invisible until an app crosses it. Caveats stated in the doc: two points define a line but do not test linearity; the 1.90x is DB-app-specific and a file-only app should be nearer 1.0x (inferred, not measured); synthetic incompressible data; one app, one box. |
||
|
admin
|
7ba7c2a271 |
Campaign 10: final results — 39 cycles, full atom set, R-156 + R-157, no leaks
Phase B completed in three passes: run 1 (27 cycles, 6 atom families, 0 violations), run 2a (10 cycles, stopped deliberately - two violations were harness defects), run 2b (39 cycles, 12 of the brief's ~13 atom families). 1461 invariant checks. Depth reached 39 consecutive cycles, past the brief's "drift at the thirty-eighth", with c34-c39 clean on every invariant. I7 headline: 66 restores across both passes, 66 correct discriminators - never stale, never empty. I2/I3/I4/I5/I6/I10/I11 zero violations in either pass. I1-under-load 5/5: the target pulled WHILE a backup ran still produced backup_target_absent and a clean recovery. R-117's Q7 case holds - a filesystem aborted in place surfaces and the gate stops the app on the dead namespace. RTO Tier-1 rallly 66MB: run 1 median 42.0s, run 2b median 41.4s over 38 restores - two independent passes agreeing to 0.6s. S band's lower end only; nothing extrapolates to M or L. RPO not measured. Monotonic growth, 9457 samples of 19 metrics over 13.5h: NO leak. Controller and agent RSS flat, fds flat, no orphaned volumes/images/containers despite dozens of redeploys, kills, reboots and hard resets. Only curve with real slope is the agent journal at ~20MB/h, bounded by journald. Findings: R-156 (papra's data neither persisted nor backed up, reports healthy) and R-157 (bootrecon's start-once sweep, two mechanisms - the zero-container one is silent on every channel). Four suspicions investigated and DISPROVED, each recorded with what settled it. |
||
|
admin
|
3d4c5365c1 |
Campaign 10: correct R-157 — the failure is INTERMITTENT (3 of 6), not deterministic
The first write-up said R-157 reproduced "at the same cycle in both runs - deterministic, not a coincidence". Wrong. The cycle numbers matched only because the runner's RNG is seeded so both runs drew the same permutation. The failure itself is a coin flip: run 2b's four hard resets went PASS(c2), FAIL(c10), PASS(c18), FAIL(c26); run 2a went PASS(c2), FAIL(c10). Three failures in six. The correction matters because it changes what kind of bug this is, and it strengthens rather than weakens the root cause: intermittency is exactly what a race against container-state settling predicts, whereas a wrong predicate would fail every time. Signature is identical on all three occurrences: rallly Exited 255 with rallly-postgres healthy, bootrecon reporting "no boot-orphaned apps" about 5s after controller start, and the container count still churning after the sweep (third occurrence 01:05: refresh 8, bootrecon 01:05:13, then 8 -> 7 -> 8). |
||
|
admin
|
7f6b00375b |
Campaign 10: R-157 — bootrecon's start-once sweep misses the boot orphan it exists to recover
Reproduced twice, two independent runs, same cycle (the runner's RNG is seeded so both drew the same permutation - deterministic, not coincidence). A hard reset mid-backup brought everything back except the app half of the DB-backed stack: rallly left Exited 255, oom=false, restarts=0, its own log ending "Ready" - it died healthy - while rallly-postgres returned healthy. 20:28:13 Status refresh: 8 containers across 55 stacks <-- docker ps -a shows NINE 20:28:18 [bootrecon] Boot reconciliation: no boot-orphaned apps 20:28:25 Status refresh: 7 ... 8 containers <-- still churning AFTER the sweep 20:39:14 [deadapp] 20 scans, 5 deployed evaluated, 1 currently down The predicate is sound: once settled the controller reports rallly state=degraded containers=2, and IsDownState includes StateDegraded, so len>0 && IsDownState holds. The SNAPSHOT was wrong. bootrecon fires as a goroutine ~5s after start while docker is still restoring containers, and is start-once by design, so it never re-checks. Consequence: the app stays down indefinitely. Detection is perfect and recovery never happens - R-52's original shape, an alarm with no recovery. Not fixed. Distinguished from this campaign's two earlier HARNESS defects: both drives bound, every other app returned incl. the drive-backed one, only the app half of a two-container stack missing while its DB is healthy, and it surfaced through the fixed check written for exactly this. |
||
|
admin
|
69f896d3cd |
Campaign 10 Phase B: 27 cycles, 586 invariant checks, 0 violations
Ran the soak on the Phase A rig. Ended on its own deadline — no watchdog halt, no atom exception, no I11 breach. I1 28+28 pairs, I2 28+28 pairs, I3 56, I4 56, I5/I6 28 each, I7 28, I10 135, I11 28. Zero violations. The row counts are themselves the no-silent-skip check: I3/I4 twice per cycle (both drives), I10 = 5 secret-class fields x 27, REBOOT on cycles 7/14/21 only. I7 is the headline: 28 restores, 28 correct discriminators — never stale, never empty. RTO (Tier 1, rallly, 66 MB): min 38.8s, median 42.0s, p90 42.5s, max 44.3s. That is the S band's lower end ONLY; the 5.5s spread over 28 runs says fixed work dominates, so nothing extrapolates to M or L. RPO not measured. Every atom and invariant was proven BY HAND before automation — the runner asserts nothing that was not first observed live. Caught a Phase A gap before starting: no app had HDD_PATH, so all data sat on the system disk and I3 could never have fired. Deployed calibre-web onto adatok first; otherwise the run would have produced 27 green cycles that tested nothing cross-drive. Investigated and DISPROVED a suspected defect (audit 5.2): /api/disks reports state=attached for a physically absent drive, and intermediary.go:230 really does compute presence from State=="attached". It is inert — planDriveGates only gates paths under /mnt/felhom-drives/ and uses BoundUnderParent there, which was correctly false. The gate fired; the storage page showed "Meghajtó leválasztva". No R-n minted. Honest gaps: 6 of ~12 atom families ran. Not run — Tier 3 (structurally un-isolatable), abort-fs-in-place, kill-agent-mid-backup, hard-reset-mid-write, reboot-VM, both concurrency atoms, fill-drive-near-full. I8 not checked, I9 not automated (cited from the tester-gate run, not re-claimed). kill_controller is NOT mid-backup and reboot_guest never interleaved with a detach. 27 cycles does not answer the brief's question about drift at the thirty-eighth. Teardown still OWED, including hub customer c10-soak (disposition: DELETE). |
||
|
admin
|
4691aa1a35 |
Campaign 10: Phase A complete + gated; Phase B not run; R-156 filed
Phase A passed every gate on a fresh box built from the PUBLISHED ISO 1.26.1: install, claim, two drives enrolled through the real endpoints with the backup target healthy, four apps spanning both sides of D5's secret split, and a working discriminator across all four. Isolation gate: both denials captured, each with a positive control. The PBS control FAILED first — four clean-looking 403s were worthless because the token was denied on its own datastore too (PBS token privilege separation). Fixed and re-run; the denials stand. R-156 (new, register grepped): papra's data is neither persisted nor backed up, and it reports healthy. The template mounts papra_data:/app/data; the app writes /app/app-data/db/db.sqlite. Volume empty and root-owned against a -rootless image, real DB in the container writable layer, healthcheck only probes the HTTP port. Its Tier-1/2 backup is real, verifiable and contains nothing. Not fixed. Tier 3 could not be isolated so it was not run: offsite hard-requires the DR tier (configs.go:1300) and the DR tier only provisions on ep0 (per-endpoint allocation deferred, hub/README.md:260). Both are recorded deliberate positions, so no R-n minted. The campaign touched neither ep0 nor the Storage Box. Phase B did not start. Phase A was budgeted at ~1h and took ~5.5h (1.26.1 is a public release image with no auto-install path, so the install was a blind screendump+sendkey walk). That left the runner — which judges eleven invariants and fires destructive atoms unattended — to be written at 04:00 with ~3h of night left. Stopped on the brief's own fence: a rig producing false negatives is worse than no rig. The rig is built and idle; teardown is OWED and itemised, including hub customer c10-soak (disposition: DELETE). |
||
|
admin
|
61e9b55737 |
SPIKE 4: a .deb in the ISO DOES deliver on an interactive install
Findings only — no script, profile or build file changed; no release ISO built, nothing published. documentation/audits/SPIKE-universal-iso-4-2026-07-31.md MEASURED, with a control, and the negative control is in the SAME box. One ISO (15 GRUB entries), a trivial probe .deb injected into /proxmox/packages/, two qm-created VMs on demo-hp (400 interactive / 401 automated control) on a scratch dir storage at the /mnt/nvme-1tb mount ROOT. Interactive (Terminal UI) install: - package installed (ii felhom-spike4-probe 0.0.1) - postinst RAN (marker + content intact) - it enabled a systemd unit, and that unit FIRED ON FIRST BOOT (uptime 7.98s, pid1=systemd) - while on the same machine proxmox-first-boot is NOT installed and /var/lib/proxmox-first-boot does not exist — Spike 3's negative reproduced, not assumed. Postinst environment (identical both paths): pid1=unconfigured.sh, NO running systemd, but 'systemctl enable' SUCCEEDS; /proc+/sys mounted; network+DNS happened to be up (inherited from the installer's DHCP — must NOT be relied on). Constraints: never systemctl start/daemon-reload, never require network, never fail, do the real work in the unit at first boot. Repack preserves it, but a naive 'xorriso -boot_image any replay' fails with 'Overlapping MBR partition entries' — iso-repack.sh:270-292 already documents that exact failure and its fix. R-153 RETRACTED into R-94 leg (b): OPEN-ITEMS.md:15 carries it verbatim at READY (XS), and R-29 says explicitly 'do not mint a new ID for a new instance'. Spike 3's further claim that the drift leaves the generator 'three minor versions stale' was FALSE and is corrected — R-94 retracts that exact reading; the served script is always main, so 1.22.0 is what every install already gets. No new R-rows opened. |
||
|
admin
|
bb29186d62 |
SPIKE 3: [first-boot] does NOT fire on an interactive install
Findings only — no script, profile or build file changed; no release ISO built, nothing published. documentation/audits/SPIKE-universal-iso-3-2026-07-31.md MEASURED with a control from the SAME image (one ISO, 15 GRUB entries): - Automated entry -> hook fires: ttyS0 marker, marker file, /var/lib/proxmox-first-boot/proxmox-first-boot (0700), activation symlink, unit active. - Terminal-UI entry, normal manual install -> ALL absent, and the proxmox-first-boot PACKAGE is not installed at all. A whole-filesystem grep for the marker returns nothing. Mechanism cited: Config.pm:118 defaults first_boot.enabled=0 and set_first_boot_opt is never called in the Perl tree; Install.pm:746 returns early without it; Install.pm:1360 skips the package. proxinstall (graphical) has ZERO occurrences of first-boot. [first-boot] is an automated-installer feature, unavailable on every interactive path by construction. R-154. A delivery mechanism DOES exist and is UNTESTED: Install.pm:1343-1372 unpacks every .deb in the ISO's /proxmox/packages/ into the target on every path (fixed skip-list), then dpkg --configure -a runs postinsts (:1378) — how PVE ships first-boot itself. Read from source, not measured. Q5: the public image should carry NO answer.toml at all — that removes the baked root hash, the disk profile and the whole Spike 1-2 problem space, and makes it a one-line release gate. But iso-repack.sh:100-106 refuses an ISO without auto-installer-mode.toml. R-155. Incidental R-153: hub hostInstallVersion=1.19.0 vs SCRIPT_VERSION=1.22.0; hostinstall_gates.py detects it and exits 1 — the gate works, nothing runs it. Q3 (real stub at before-network) was NOT reached and is recorded as not reached. |
||
|
admin
|
19c932a693 |
SPIKE 2 complete: locked root closes the PVE web UI; before-network gives a measured zero window
Findings only — no script, profile or build file changed; no ISO built, nothing published.
documentation/audits/SPIKE-universal-iso-2-2026-07-31.md
Both Tier 0 boxes went offline mid-session (provider cable fault; four routes tried, no Tier 2
fallback used) and returned. All three scenarios then ran to completion on real PVE, each signalled
by reboot-mode='power-off' rather than a disk hash.
- A LOCKED ROOT CLOSES THE PVE WEB INTERFACE. Measured at the exact endpoint the UI uses
(POST /api2/json/access/ticket, root@pam) WITH A WORKING CONTROL: known-password install returns
HTTP 200 + ticket; locked install returns 401 for every password and none can exist.
passwd -S root = L, shadow = literal-asterisk, PVE uses the stock PAM stack.
- GRUB recovery mode is also closed ('the root account is locked') — but init=/bin/bash still gives
an unauthenticated root@(none):/#. A locked box is recoverable, operator-only, at the console.
The installed GRUB has NO password, so locking root is not a physical-security measure. R-152.
- before-network MEASURED (A/B, same image): the hook RUNS (marker, uptime 6.58s) with entropy 256,
writable /etc, all binaries and openssl_rand_len=32, while ip_global is EMPTY and
listen_22_8006 = 0. fully-up is the converse: sshd+pveproxy active, 3 listening. Zero window.
- R-148: answer.toml.tmpl:27 justifies fully-up with a pvesh/pct dependency the stub does not have
(grep rc=1) — it blocked the ordering now measured as the fix.
- R-149 three ordering values; R-150 Condition-guarded hooks skip silently; R-151 demo-felhom built
from an uncommitted profile.
Three probes failed and are recorded as failed: a container probe that ran as uid 0, a GRUB probe
that missed the 1-second menu timeout, and a kernel-line edit one line off (caught by a pre-typing
verification screendump). The interim 'Layer 1 teardown INCOMPLETE' is corrected — the fixture had
never landed, because the staging mkdir was in the SSH call that timed out.
|
||
|
admin
|
5bdd8372f8 |
SPIKE 2: before-network gives a zero window by construction; locked root closes sulogin
Findings only — no script, profile or build file changed; no ISO built, nothing published. documentation/audits/SPIKE-universal-iso-2-2026-07-31.md BOTH Tier 0 boxes went offline mid-session (remote site, 12:28 CEST; four routes tried, our tailscale pod healthy). Q1/Q2/Q3 each keep a part needing a nested VM: those are BLOCKED, not answered. DooPlex was NOT used as a fallback — Tier 2, and this task did not authorise it. Established without them: - STRUCTURAL: ordering='before-network' maps to proxmox-first-boot-network-pre.service (Before=network-pre.target, Type=oneshot) — it completes before ANY interface is configured, so a rotation there has a zero-length window BY CONSTRUCTION, not by being fast. - R-148: the stub does not need 'fully-up'. stub-first-boot.sh has no pvesh/pct/pveum/qm call (grep rc=1); that usage is in felhom-bootstrap.sh under its own After=network-online unit. answer.toml.tmpl:27 justifies the current ordering with a dependency that does not exist. - R-149: the ordering enum has THREE values (before-network, network-online, fully-up), not two. - MECHANISM (container, not PVE): locked root closes sulogin — 'the root account is locked' for both '*' and '!', with a working control. So 'discard' and 'lock' are the SAME outcome for recovery, making the escrow decision binary. - R-150: all four proxmox-first-boot-* units are Condition-guarded; a failed condition is a SKIP, so a hook that never ran looks identical to one that succeeded. - R-151: demo-felhom was installed from an UNCOMMITTED profile — a Tier 0 reference box is not reproducible from main. - Q4: four gates in iso-repack.sh enforce the single-entry menu; default/timeout already settable. The first mechanism probe was invalid (uid 0 bypassed pam_unix; sulogin had no tty) and a teardown error (shredding the control plaintext) are both recorded as failures, not massaged. demo-hp teardown is INCOMPLETE and named as such; the command is recorded, not claimed done. |
||
|
admin
|
ea00976403 |
SPIKE: a universal ISO needs a different disk strategy and a locked root
Findings only — no script, profile or build file changed; no ISO built, nothing published. documentation/audits/SPIKE-universal-iso-2026-07-31.md - R-139 (HIGH): a disk filter matching >1 device does NOT fail safe. Observed in a nested VM — the installer silently picked one of two matching disks and wiped it; validate-answer accepts such an answer. The 'filter did not match any devices' guard covers the ZERO-match case only. - No udev property distinguishes an internal system disk from external media. Measured on demo-felhom with its 1TB external attached: ID_BUS='ata' for BOTH, lsblk RM=0 for both, and device-info exposes no removability property. demo-hp's NVMe carries no ID_BUS/ID_TYPE at all. - R-141 (HIGH): the answer schema makes a root credential mandatory, but root-password-hashed='*' validates AND installs to completion. [first-boot].ordering accepts 'before-network', the only ordering that closes the exposure window structurally. - Q3: prepare-iso leaves grub.cfg byte-identical to stock (15 entries, automated AND interactive) — a two-entry menu is purely a Felhom grub.cfg.tmpl change. - R-129 resolved: demo-hp's key is the operator's own, added post-install; demo-felhom's IS baked by an uncommitted profile. The reachable-before-rotation measurement FAILED twice and is recorded as failed, not inferred. Opens R-139..R-147; restates R-128. |
||
|
admin
|
9e079c7883 |
RECON: a Felhom-issued subdomain works in the product — the blocker is Cloudflare edge-cert depth
Question A: YES, no code change. customer.domain is a trimmed string with no UNIQUE, no CHECK, no format rule (store.go:114, configs.go:673), copied verbatim into controller.yaml (configgen.go:48), and every one of its 30 consumers on the box interpolates it without parsing. Zero hits for registrable/eTLD/publicsuffix across both repos. Nothing creates DNS records (zero hits for dns_records) — the two Cloudflare clients are WAF-only. And the zone-ownership assumption is a SWITCH, not a requirement: traefik.yml.tmpl selects DNS-01 when cf_api_token is set and HTTP-01 when it is empty. The real blocker is Cloudflare, proven live: the edge certificate covers exactly one wildcard level (SAN = demo-felhom.eu, *.demo-felhom.eu), so a two-label hostname — which a per-tester subdomain forces — gets "tls alert handshake failure" and no peer certificate at all. That makes Advanced Certificate Manager a prerequisite of the separate-domain plan, not an optional extra. Whether ACM is available on the account could not be established read-only: the only Cloudflare tokens in reach are the Zone:DNS:Edit tokens on the demo boxes, which the fence forbids using. Question C, measured rather than reasoned: r.Cookie returns the FIRST match and never tries the others (BOGUS+real = 302, real+BOGUS = 200), so a tossed cookie wins outright — DoS and confusion, not takeover, since it fails closed on mutations. CSRF is a single choke point (server.go:256) and the token carries the whole load against a same-registrable-domain attacker. But it is SKIPPED entirely when no session cookie is present, which with browser-cached Basic auth is cross-origin CSRF on every mutating route (R-135). Agreeing with the separate-domain recommendation, with the caveat the brief asked for: it is necessary but not sufficient. It does not solve Question D, because that is a shared-zone problem and the new domain is a shared zone. Filed R-133..R-138: duplicate domains accepted; hub/controller zone-resolvers disagree on depth; CSRF skipped on the no-cookie path; __Host- rename (one line, preconditions verified met); geo-WAF rules zone-scoped and non-namespaced (four cross-tenant faults, blocks shared-zone onboarding); shared-zone cf_api_token is a zone-wide DNS-write capability on a customer's box. Nothing created: no customer, DNS record, tunnel, route or code change. |
||
|
admin
|
eb5d05f496 | docs: host-addresses audit + capability-map row + REPORT (agent 0.119.0 / hub 0.85.0) | ||
|
admin
|
b4edc087fa |
Tester gate: golden re-baked to 0.188.0, fresh-install proof PASSED — a fresh box is safe to hand to a tester
§7.2 answer: YES. A real day-0 from the existing v1.25.0 ISO reached a claimable,
app-serving box in ~10 minutes unattended, and an app's data came back from the
drive with the guest's app.yaml gone — proven readable by the application over
its own TCP path, with a discriminator (PRE-BACKUP row = 1, POST-BACKUP row = 0).
Part 0: NO ISO rebuild needed, verified against the ISO on disk rather than from
source. It bakes only felhom-bootstrap.sh, its unit and the secret-free pairing
env (full-base64 match, 1 hit each) and 0 hits for any installer, controller or
golden marker. The installer is fetched at run time; the live URL is byte-identical
to repo HEAD (v1.22.0, six days newer than the ISO) and the fresh box ran it.
Part 1: baked 0.188.0 rather than the brief's 0.187.0 — 0.187.0 lacks D5, which
is the very claim Part 2 step 6 tests. Published (404 pre-gate with a 200 control;
anonymous download, 649310288 bytes, sha match), vouched, and consumed by a real
box. R-120's gate exercised BOTH ways: 0.185.1 refused with no write, 0.188.0
allowed — evaluated, not silently skipped.
Part 3: RUNBOOK-manual-build.md cited a "RECORDED" qemu line that is itself
labelled reconstructed and whose source says it was never saved. The real
invocation is now captured from this bake as §4.0, with the bake/publish/teardown
steps; the old entry is marked SUPERSEDED.
Teardown all three layers, hub disposition stated: VM destroyed, scratch storage
removed with space returned exactly, customer sess-g DELETED via full cascade.
sess-f deliberately left (R-131) with its command recorded.
Filed, none fixed: R-128 (false ISO_VERSION invariant comment), R-129 (demo-hp's
"no baked SSH key" is stale — key auth works), R-130 (HARD_MIN_LVM_GIB warns and
proceeds), R-131 (fourth orphaned scratch customer), R-132 (curl's %{redirect_url}
printed the hub operator password into a transcript — HUB_PW needs rotating).
|
||
|
admin
|
0a9bd3829d |
D5 SHIPPED: Tier-1/2 restore no longer depends on the whole-guest tier
Records controller v0.188.0 across the four coupled artifacts. 07-backup-architecture.md is the owning doc: - new 7.4 = the recovery chain AFTER D5 (7.1 leg 1 superseded; leg 2, the living-app dependency, explicitly unchanged so this is not read as more than it is) - 7.3 collapsed to history, with the correction that the target as written (data_key-only) was tested in Part 0 and rejected - 3 records that the two-lane split is now real, not just intended - matrix rows 3 / 3c (new) / 13; 10.1 D5 itself shipped Also: new capability-map row, D5 collapsed in ROADMAP + OPEN-ITEMS, and R-127 filed in both (data_key flag unreliable; O4 can regenerate a DB password that no longer matches the restored data directory). The audit is named D5-drive-alone-restore rather than "...secrets..." because .gitignore blocks *secret* -- a guard worth respecting, not forcing past. |
||
|
admin
|
d42d90fed7 |
R-108 CLOSED — D5's precondition is met (controller v0.187.0)
Four-artifact update per the coupling rule, plus the audit. 07-backup-architecture.md: §10.1 retitled CLOSED with the ruling and the D5 sentence; the FileBrowser network-share row flipped YES->NO, closed at the PLACEMENT rather than at the bind; the exposure chain annotated with the fifth surface (decommission-with-migrate guarded only its source) and the correction that the boundary is the deploy POST, not the dropdown; §7.3 retitled UNBLOCKED; register row collapsed; open question F answered. 00-capability-map.md: new §D row PROVEN-LIVE, with the un-exercised legs named — the deploy-POST and decommission refusals are unit-tested, not live-fired. OPEN-ITEMS.md: R-108 dispositioned; D5 given its OWN row as READY/UNBLOCKED (it had existed only inside other rows' prose — the R-123 thread-loss pattern); R-126 registered. ROADMAP.md: R-108 collapsed to a shipped one-liner; R-126 added. R-126 filed not fixed: a .fab bundle (plaintext secrets, optional password) can be exported ONTO a NAS. Split out of R-108 rather than folded in — it is an explicit customer-chosen export destination, not a browsing surface reaching a backup tree, so it was never part of D5's precondition. Live evidence: same-box before/after on demo-felhom through the real authenticated endpoint, the network-specific refusal on demo-hp, non-effect verified in the registry, and R-67's share-root bind diffed byte-identical across the deploy. |
||
|
admin
|
70f84941d4 |
R-106/R-109 audit + registers: shipped at agent 0.118.1, plus R-125
Adds the full audit: Part 0's three answers, the pre/post recipe for both boxes, the on-disk proof that `local` froze at the 2026-07-28 target move while felhom-backup kept running, all seven red-proofs, and the three publish observables. R-125 filed: v0.118.0's R-106 half shipped INERT. Two tests ran the real Collector.Collect() but both injected a fakeObserver, and the break was one layer below in mergeConfig, which dropped the pbs namespace. The recipe still said "root" — now with namespace_state "resolved" beside it, confident and wrong. Caught by live validation, not by the green suite. Fixed in 0.118.1; filed for the doctrine point that a production-path claim must name the seam it injects at. |
||
|
admin
|
3d504d58c8 |
docs(R-117): CLOSED — proven live on demo-hp; R-121 filed for agent-on-box drift
R-117 row → SHIPPED + PROVEN-LIVE (agent v0.117.0), with the full validation in audits/R117-v0117-2026-07-30.md. Both dead states detected on real hardware through the shipped predicate: RETURN raw 8:32 /dev/sdc | bind 8:16 shutdown → stale-device, usable false IN-PLACE both 252:11 emergency_ro, raw unit active → filesystem-aborted, usable false healthy → live 340-497us per call. No block I/O proven by strace (only /proc/self/mountinfo, 0 statfs) — the Part 1 CLAUDE.md fence applied to its own first consumer. No regression through the real pipeline: the live backup-target drive reads bound_under_parent=True via GET /disks with the controller's own credential, with 32 gate lines in 3 min as the positive observable and zero spurious transitions. The ruling asked for in §2.2 is recorded in full and flagged for overrule: Aborted must NOT self-heal. A re-bind lands on the same dead superblock and the call site runs every 20s, so repairing would be an infinite silent retry that masks the state. It surfaces instead. No operator decision was taken quietly — the reasoning is that it routes an already-broken state into the existing gate, event types and Hungarian copy, so no new concept reaches the customer. R-121 filed: a box's installed agent can sit releases behind the vouched one and nothing notices. demo-hp ran 0.113.0 against a vouched 0.116.0 through the whole R-116/R-117 arc. Confirmed at source that R-120's gate cannot catch it — it compares goldenVer against NewestReportedControllerVersion(), i.e. golden-artifact vs fleet-CONTROLLER. MinAgent is protective, not an alarm, and 0.113.0 equalled the floor. Fourth instance of the drift family. Also filed: R-117g (an aborted filesystem is never cleared automatically by design, so it alarms until a human acts, with no guided recovery) and R-117h (StablePathForRaw hardcodes the parent, so the repair path cannot be exercised on hardware without writing into a live customer guest's namespace). |
||
|
admin
|
e70b5feebe |
docs(R-117): the hang case measured — an I/O probe turns a wedged drive into an unkillable agent
Completes the spike once the venue came back. Q4's hang case and teardown are now measurements, not plans. Against a dmsetup-suspended device (I/O queues instead of returning EIO): - P1 (devno compare) and P2 (ext4 abort flags) completed in 364us / 206us. They read /proc, so no block device is involved. - statfs and getdents completed and reported HEALTHY — on a wedged device they do not even hang. R-117b confirmed in a second failure mode. - EVERY probe that touches the device blocked, including a buffered write with no fsync: the O_CREAT metadata path needs journal access (wchan=do_get_write_access). There is no cheap-and-safe write probe. - The blocked process survived SIGTERM AND SIGKILL (stat=D, wchan=folio_wait_bit_common, still alive 3m50s after kill -9) and died only when the device was resumed. So `systemctl restart felhom-agent` would hang, leaving the agent unrecoverable until the device returns or the host reboots. The thread count does not reveal the leak (5->5, 5->6). Filed as R-117f. A timeout protects the caller's control flow and nothing else, so "the fix must issue no block I/O" is now a fence rather than a preference — the thread-leak hypothesis the probes were built to test turned out to be the weaker half of the result. Teardown done, all three layers: guest 9301 destroyed, r117scratch removed, both dm and both loop devices gone, scsi_debug unloaded, local back to 37.02% against a 37.00% session start. Fences re-verified AFTER teardown: 9201 running, drill-r50 stopped, local-lvm 38.84% byte-identical, felhom-backup content unchanged, live /mnt/felhom-drives intact with both submounts, agent active. Layer 3 genuinely empty — 9301 had no NIC and ran no controller. Trap recorded: a suspended dm device must be resumed BEFORE any umount, or the teardown blocks on the same uninterruptible sleep. |
||
|
admin
|
c949389c95 |
docs(R-117): spike — the mechanism, a recipe, and a steady-state half nobody had looked for
Both halves of the R-113 conjunction are path-presence tests: GuestSeesMount (intermediary.go:276) and isHostMountpoint (:394) compare field 5 of a mountinfo line and never read field 3, so neither can see that the bind and the raw mount name different devices. Measured BoundUnderParent=TRUE over a namespace that EIOs on every read and write. Reproduced 3/3 on a purpose-built scratch LXC on demo-hp; predicates evaluated by a throwaway probe calling the real localapi code from d4eb259. Three results that change the shape of the fix: - Q7: a bind can die in STEADY STATE with no detach/return cycle. The gate produces no action and nothing is emitted on any channel. A Return-branch fix cannot reach this half, and a devno comparison does not detect it. - Q6/R-117d: AttachDrive's normalize leg already performs the repair, and three call sites already invoke it - including the controller's Return branch before it restarts apps. All defeated by one early return at :235. Unblock the existing path; do not add a new one. - Q1: the device-node change is a CONSEQUENCE, not a precondition. The stale bind pins the dead superblock, forcing the returning device onto a new number. Control test: released, the letter is reused. Not established: the hang case. Venue and probes built, run lost to a site internet outage; the thread-leak hypothesis is not claimed as a result. Teardown of the spike venue is owed - commands in the findings doc; nothing fenced was touched and no hub-side record was created. |
||
|
admin
|
29bcfeb214 |
docs(R-120): CLOSED on both halves — golden current, and the class has a gate that refuses
Half 1, the artifact: golden 0.186.0 baked, published, vouched, and proven on a REAL day-0 on demo-hp (not the fixture, per the rule committed in Part 1). With the target detached, the fresh box's endpoint returned the TargetAbsent copy -- "A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod..." -- with offer_path absent entirely. The day-old read on the 0.185.1 golden had returned the false system-disk message plus an offer of the other drive. That is the customer-visible defect closed. Half 2, the mechanism: operator ruled REFUSE, shipped as hub v0.82.0 and DEPLOYED. Proven live by re-attempting the original mistake -- vouching the stale 0.185.1 golden now yields HTTP 303 flash=golden_behind_fleet plus [WARN] artifact vouch REFUSED, and the manifest reads back unchanged at 0.186.0. Refused AND unwritten, against the real fleet signal rather than a unit fixture. Recorded on R-29's audit list as the first ENFORCED gate beside its three orphans, so the contrast is kept rather than lost. The orphans are unchanged -- this proves the pattern is available, not that the backlog moved. Teardown all three layers: VM 9402 purged, r120-images removed with the space measured back, hub layer gate-blocked on ONLINE with the command recorded. Last session's sess-e was deleted this run, discharging its recorded layer 3. |
||
|
admin
|
49b627684c |
docs(R-120): golden rebaked to 0.186.0, published, vouched, proven on a real day-0
The golden baked controller 0.185.1 -- confirmed from the golden's OWN record (drill/bake-0.185.1.log:1 and :330) and from build-golden.sh:345, which derives GOLDEN_VERSION from the controller tag. 0.185.1 predates R-114 + R-112, so every freshly installed box told a customer whose backup drive had fallen out that the backup was on the same disk as the system (false) and offered a different drive as the remedy. Baked golden 0.186.0 from main's controller in the DooPlex bake fixture: overlay2 OK, 3 mounts included, FATAL 0, exclusions 0, 618 MB, upload HTTP 201, GOLDEN_SHA256 b760ac6a33e70700..., token-leak grep 0, GL-1 teardown with drill.qcow2 back to virgin. Three observables, quoted as returned: PUBLISHED (anonymous GET -- what the installer does -- 200 / 648930639 bytes / sha identical to the bake); VOUCHED (manifest read BACK, not the 303); RESOLVED BY A CONSUMER (Artifact manifest served for customer sess-f, golden=0.186.0). Floor NOT touched per publish-train rule 2 -- it is a separate form and min_controller_version still reads 0.156.0. MinAgent left 0.113.0 because 0.186.0 declares it unchanged. Proven on a REAL day-0 on demo-hp, not the fixture, per the rule committed in Part 1: VM 9402 from the v1.25.0 ISO -> Controller elindult (0.186.0), box confirms felhom-controller:0.186.0 + agent 0.116.0. A fresh box now runs 0.186.0 where it ran 0.185.1. The procedure was NOT unwritten: RUNBOOK-manual-build.md:101-115 documents it and build-golden.sh carries its own usage and publishes to Gitea itself. One documentation-integrity finding: that runbook says to use the RECORDED qemu line and not reconstruct, while the line it cites is itself labelled reconstructed, the canonical one never having been saved. NOT done and not claimed: the TargetAbsent/empty-offer_path endpoint capture (the claim gate runs before auth with no Bearer escape -- R-119's fourth instance), and the Part 3 mechanism, which awaits the operator ruling. Recommendation and exact wiring recorded in the audit rather than built. VM 9402 + r120-images + customer sess-f retained pending that read, with teardown commands recorded. Previous session's sess-e layer-3 is now DISCHARGED -- it aged to STALE and the cascade completed, full residue purge logged. |
||
|
admin
|
772956d214 |
docs(R-116): CLOSED — proven live; capability row F to PROVEN-LIVE; R-120 filed
The events leg the previous commit reported as not-reached is now done. The operator relayed the claim code (the only route: bcrypt-hashed hub-side, emailed only), the two storage paths were registered through the real POST /api/storage/register, and the cycle ran on the fresh box: 07:20:04 backup_target_absent (error) Cel meghajto <- TARGET, specific 07:22:34 backup_target_restored (info) Cel meghajto <- its matching pair 07:24:04 storage_disconnected (error) Adat meghajto <- NON-target, generic 07:25:34 storage_reconnected (info) Adat meghajto All four at the hub; gate fired in 3 s. Two matched pairs, correctly discriminated -- and discrimination is proven NON-trivially for the first time, since both prior runs had the target itself emit the generic event. Over-correction passes on a positive observable, with two RETURNED lines proving the gate was ticking. 00-capability-map row F: PARTIAL -> PROVEN-LIVE with the evidence and the caveat. R-120 filed: the golden bakes controller 0.185.1, which PREDATES R-114 + R-112, so a freshly installed box shows the customer the WRONG absent-target message -- observed live on the drill box: the generic "the backup is on the same disk as the system" copy (false; the target is a drive that vanished) plus an offer of the other drive as the remedy. That is E2D 5.3's exact payload, still reachable on any new install. R-115's class one layer up -- R-111 closed by re-baking the golden, 0.186.0 then shipped, the golden did not move, and the gap reopened silently; this time the stale artifact carries a customer-facing falsehood in exactly the state R-116 now alarms about correctly. Teardown recorded for all three layers, hub layer gate-blocked with the command. |
||
|
admin
|
315c469fc8 |
docs(R-116): v0.116.0 proven live at the payload layer; events leg blocked on an emailed claim code
audits/R116-v0116-2026-07-30.md + the R-116 register row. WHAT PASSED, on real hardware. Agent 0.116.0 published (independent registry GET verified the bytes), vouched, and installed UNAIDED by a fresh box -- "Artifact manifest served for customer sess-e (agent=0.116.0 golden=0.185.1)", host sess-e-5d4427 ... 0.116.0 ONLINE. Real day-0 on a nested PVE on demo-hp (per runbooks/target-selection.md, which sent this run there rather than to the DooPlex fixture the previous run used), both drives enrolled through the real endpoints, device loss a real hot-detach. Captured live, absent state: the target is now ONE row carrying backup_target:true AND guest_path:/mnt/felhom-drives/cel with mount_path:"", so isTarget[/mnt/felhom-drives/cel] = TRUE -- it was false through v0.115.0. RETURNED gives true as well, so the pair matches. All three guards pass from the same payload: R-114 preserved (no row combines the flag with a non-empty mount_path), no over-correction (bound_under_parent:false), and discrimination at the payload layer (the non-target carries the flag on no row) -- the thing neither prior run could show. WHAT DID NOT HAPPEN, and is not claimed. No backup_target_absent or backup_target_restored event was observed on the wire. planDriveGates iterates registered StoragePaths and the drill controller has none ([WARN] Storage paths: no storage paths registered); every storage route answers 401 "dashboard not yet claimed". The claim code is bcrypt-hashed and emailed-only, and handleSelfBindLinkSend (selfbind_mint.go:139-161) renders a flash and never the token, so no operator-side route exists. A gen-2 code was re-sent; the drill VM, its storage and customer sess-e are DELIBERATELY RETAINED with teardown commands recorded, so the leg finishes without a rebuild. Reported as not-reached rather than as a third trivial pass. R-119 filed: the claim gate makes drive-gate legs unreachable to CC by design, and has now stopped three sessions at the same wall -- needs a ruling (operator-scoped test affordance, or a documented prerequisite step), not a fix. R-117 reproduced on real hardware with a read/write probe (EIO both directions while /disks reports attached + bound_under_parent:true) and §5 records how it colours the reattach leg. R-118's symptom vanishes incidentally on this one row; R-118 is NOT fixed. sess-c and sess-d verified GONE (404, absent from both tables) -- cleared by the operator using the previously recorded commands, not by this session. |
||
|
admin
|
699790b12d |
docs: write down which boxes are disposable (target selection by blast radius)
Nothing in the repo said which machines are safe to break. The host table gave access and role and stopped there, so a session needing a victim had to guess -- and the guessing inverted: the two boxes that exist to be broken were treated as sacred, and DooPlex (the recovery chain) got used because it was the only box no spec had fenced. New documentation/runbooks/target-selection.md -- one page, three tiers, and per machine what is freely permitted / needs care / forbidden, each carrying its REASON so a rule can be correctly narrowed later instead of ossifying. States the selection rule positively (start at Tier 0; a Tier 2 box only when a task says so explicitly; an absent fence is not permission) and that fences name ACTS, not machines -- demo-hp's over-subscribed local-lvm is one dangerous storage, not a dangerous box. CLAUDE.md: host table gains a Blast radius column, gains the missing demo-hp row (it was where the drill VMs ran and it was not in the table at all), and a pointer line to the new runbook. CORRECTION to the spec's problem statement: the designation was not missing. The 2026-07-25 operator ruling naming the t740 as drill+build VM host -- explicitly "moved off DooPlex" -- already existed in operations/nodes.md. It sat where no session reads at start, while the prohibitions were repeated in every task spec. The defect is reachability of the ruling, not its absence, and the R-116 drill on DooPlex contradicted a written ruling rather than filling a vacuum. CORRECTION to the R-116 record, same commit: the baseline claimed controller 0.186.0 on both demo boxes. Only felhom-pve was sampled and generalised; demo-hp re-checked directly runs 0.185.1, so the fleet is split and R-114's TargetAbsent branch is absent from demo-hp. Fixed in the audit table and REPORT-r116-diag. Docs only -- no code, no build, no deploy, no host reconfigured, no host renamed. |
||
|
admin
|
d56e395a2a |
docs(R-116): isolate the mechanism from the real /disks payload; file R-117 + R-118
The absent-state /disks payload was captured on a genuine device loss, after a present-drive control run proved the query works (Part 5's three attempts failed on token extraction, and its control returned 0 rows). The answer is theory #1 -- "the registry-union row writes false" -- which was raised, declared wrong and retracted. The retraction was the error. Absent state returns 4 rows, not 3. The drive appears twice and the two facts the controller needs sit on different rows: the Observe row has backup_target:true but mount_path:"" and guest_path:"", so it contributes no key to driveTargetByPath; the registry-union row owns /mnt/felhom-drives/<name> and omits BackupTarget from its struct literal (disks.go:301-306) => false. The union row is not deduped because seen is keyed on MountPath (:290-295), the one field the absent state empties, and its own MountPath comes from the systemd .mount unit FILE (registry_known.go:40-75), which never reads the mount table. Theory #2 (the basis of the shipped v0.115.0) is false on both halves; #3 is false too. v0.115.0 is provably inert: StablePathForRaw("") returns "". Also files the read path verbatim -- the token plaintext lives only in bootstrap.json on the Proxmox host; the agent's store keeps hashes only. New: R-117 (READY M, outranks R-116) -- a returned drive's guest bind is a DEAD mount (EIO both ways) while /disks reports attached + bound_under_parent:true, so the gate restarts the customer's apps onto it and reports healthy with no alarm. R-118 (READY XS) -- an absent drive's union row advertises the root filesystem's capacity as its own. Docs only. No code written, nothing built or published; v0.115.0 untouched. Both demo boxes read-only; drill fixture restored to virgin. |
||
|
admin
|
c3ce4c7b20 |
R-116 Part 5 FAILED: the fix shipped, C5 still fails, mechanism NOT isolated
A fresh box running the fully shipped stack -- agent 0.115.0 from the Day-0 manifest plus controller 0.185.1 from the vouched golden, no hand-deploy -- still fired the GENERIC storage_disconnected on detach and the SPECIFIC backup_target_restored on return. backup_target_absent count 0. Identical to Session C. The v0.115.0 fix changed nothing observable. Part 4's three positive observables were all obtained before the run (registry newest 0.115.0, hub vouches 0.115.0, felhom-pve running 0.115.0 clean), so the publish step forgotten twice was not forgotten a third time, and the box demonstrably installed the fix under test. Discrimination FAILS: the target itself produced the generic event, so the two cannot be told apart regardless of the non-target leg -- which was therefore not staged. Reported as a fail, not as Session C's trivial pass. Over-correction guard PASSES: 0 ABSENT lines with the drive present, target degraded:false. THE HONEST PART. The fix targets a shape that does not occur live, and which shape does occur is NOT ISOLATED. With the drive detached PVE reports the storage inactive with zeroed fields -- a shape the unit fixture did not model. Three attempts to read the real /disks payload failed on token extraction across the ssh -> guest -> container layers, and a present-drive CONTROL query also returned 0 rows, proving the query was broken rather than the payload. Without that control this run would have recorded a third false mechanism, after "the union row writes false" (wrong, corrected yesterday) and "no row carries the guest path" (unverified). The leading hypothesis -- an inactive storage reaching Observe with an empty MountPath, so StablePathForRaw returns "" -- is consistent with the pvesm output but is NOT evidence and is recorded as such. Next session's first job is a working /disks read, with a present-drive control run FIRST, before any further code. agent v0.115.0 is published, vouched and INERT. Not reverted: reverting is itself a change, the runbook forbids fixing mid-run, and the code is tested and harmless. Capability-map row F stays PARTIAL, now citing the re-test. Teardown clean: pvesm status after == before (local-lvm 38.83%), guest 9201 and drill-r50 untouched. Customer sess-d pending the usual ONLINE-ages-to-DOWN gate. |
||
|
admin
|
e87d6b26bb |
Correct the Session C audit: the union row is DEDUPED AWAY, not written false
The audit said the union row "writes false" for the guest-path key. That is wrong, and the next reader would have inherited the error. Isolated during R-116's Phase 0: RoleForStorage returns RoleSystem whenever backingDevice == "" (felhom-agent internal/storage/role.go:180-181). When the device vanishes the target row's role flips to system and it loses its guest path, but KEEPS its MountPath -- and the union loop skips any drive whose MountPath is already seen, so the registry row is never emitted at all. /disks therefore carries NO row with that guest path: isTarget[guestPath] is a MISSING KEY, not a false value. The practical difference is decisive -- the obvious fix (set BackupTarget on the union row) could not have worked, because that row does not exist in the state where the alarm is needed. The section's own "not isolated" caveat is replaced by the isolated answer. |
||
|
admin
|
952ebf4862 |
Record work, banked first: shrink the E-2d row, create the missing capability-map rows
Unconditional and three sessions overdue, so it commits before any code is
touched — E-2d itself stopped at Phase 0 and banked nothing.
E-2d row: 822 words -> 121, and the contradiction resolved. Its State read
CLOSED — PARTIALLY PROVEN while the cell's final sentence read "This row stays
OPEN only for the residue"; a reader could not tell which. It is CLOSED, with
R-116 the single named open leg.
Nothing unique was binned. Three facts existed ONLY in that cell and are moved
into audits/E2D-fresh-vm-2026-07-29.md as a new §1a: the local-lvm fence figures
with the 888 GB nvme alternative, the exactMount subdirectory caveat and why the
subdirectory is nonetheless the safe placement (no durable_id collision), and
the ISO/PAIRING -> DIRECT fall-through derived at source with its line
citations. drill-r50's blocked status was already in both audits.
Capability map: it had ZERO rows for the backup-target work — grep gives 0 hits
for backup_target and one for "E-2" that is a campaign date string. Three
scenario rows added, at today's honest status, not the value hoped for later:
C. Protection & recovery — installer Case A/B, DEGRADED recorded not hidden
PROVEN-LIVE, cites E2D-fresh-vm C1+C2
D. Storage & devices — the offer, and that registration confers no role
PROVEN-LIVE, cites SESSION-C C4 + the decline path
F. Notifications & monitoring — the absent-target alarm and its pairing
PARTIAL, cites SESSION-C C5, leg named, -> R-116
Row F is PARTIAL today per the doc's own strict enum (a leg not exercised live
is PARTIAL with the leg named, never PROVEN-LIVE). A later session may flip it;
this commit must not.
|
||
|
admin
|
06d7788392 |
Session C: R-113/R-114/R-112 PROVEN LIVE; C5 fails on a new defect (R-116)
Full ISO/PAIRING run on a fresh nested box. Agent 0.114.0 came from the Day-0 manifest -- the SHIPPED binary -- so C5 tested the real artifact. Controller 0.186.0 hand-deployed after install per the §3.1 ruling; the vouched golden bakes 0.185.1, so C3/C4 prove the code not the shipped golden, and that lag is filed against R-115 rather than a new ID. R-113 PROVEN: detach 18:43:50, gate fired 18:43:54 -- four seconds, where E-2d measured zero over 4.5 minutes -- and SetDisconnected was reached. It fired on exactly the shape that defeated it: raw /mnt/mentes NOT mounted while the bind /mnt/felhom-drives/mentes still read /dev/sdb[/felhom-data]. R-114 PROVEN: with the target absent the page rendered the absent copy, the system-disk copy 0 and the offer block 0. Both of E-2d's falsehoods are gone. R-112 PROVEN: the banner reached a customer's page for the first time. Healthy renders nothing, proven POSITIVELY -- idle delta 0 /backup/tiers calls, page load delta +1, single caller, so the seam ran and chose silence. C5 FAILED on a fourth, separate defect. The alarm fires but as the GENERIC storage_disconnected, while the recovery is the SPECIFIC backup_target_restored -- a pair an operator cannot match, which is what notifyDriveReturned's own comment forbids. backup_target_absent count 0 across the run. Root cause: the drive is TWO /disks rows and BackupTarget and GuestPath sit on different ones; absent they separate, on return they rejoin. v0.184.1 fixed the keying, not this. Only reachable because R-113 made the gate fire at all. Filed as R-116. Mirror + over-correction guard PASS: non-target drive -> storage_disconnected, backup_target_absent 0; both drives present -> 0 ABSENT lines and the target stayed healthy. Caveat recorded: the mirror passes trivially because the target also produced the generic event. E-2 and E-2d CLOSED as partially proven with R-116 the one named open leg, per the runbook's §9 rule decided in advance rather than mid-run. Capability map NOT touched: it has no E-2 rows at all, so nothing could move to PROVEN-LIVE. Creating them is a design act, not a validation act. Teardown clean: pvesm status after == before (local-lvm 38.78%), guest 9201 and drill-r50 untouched. Customer delete attempted and correctly refused while the host still reads ONLINE; command recorded for once it ages to DOWN. |
||
|
admin
|
d839ddcb60 |
E-2d teardown complete: drill customer + host removed from the hub
The delete was correctly refused at four successive gates while the host still read ONLINE (acknowledgements -> typed confirm_id -> expect_hosts stale-preview -> "host is ONLINE"). Rather than force it, the run waited for the destroyed host to age to DOWN; delete-impact then reported deletable:true and the documented cascade ran: host deleted (escrow demoted to retained custody), tenantsync deprovisioned, PBS tenancy deprovisioned, claim reset to unclaimed, residue purged (reports=5 app_telemetry=5 notif_prefs=1 appliance_registrations=1) Verified after: 0 occurrences of "e2d" anywhere on the hosts page; demo-felhom and demo-hp ONLINE on agent 0.113.0; drill-r50 and peti-felhom unchanged; demo-hp carries only guest 9201 and VM 300. Scoping checked rather than assumed: the single purged appliance_registration was this run's own appliance (810d10c5, bound to e2d-fresh). The unrelated stale 2026-07-25 appliance (206c8838 / QWA-WJE) was NOT touched by the cascade — the operator removed it separately. - OPEN-ITEMS.md: the drill-cleanup WATCHING row is removed (done, not open). - audits/E2D-fresh-vm-2026-07-29.md §8 + REPORT-e2d.md: teardown recorded as complete, with the cascade output and the appliance-scoping note. |
||
|
admin
|
f3975cf5bc |
E-2d executed on a fresh box: C1/C2 proven, C3/C4 partial, C5 FAILS — R-112/113/114
Full ISO/PAIRING route on a nested PVE VM on demo-hp, after R-111 was fixed earlier in the session. Bind -> running controller in 3m35s. The install fetched the artifacts published an hour before and restored the golden baked 20 minutes before, so the publish train is proven end to end on a real install. C1 PROVEN: "felhom-host-install v1.22.0", "Day-0 provision SUCCESS", guest 9201 running, bootstrap unit wrote its done-flag and self-disabled. This retires E-2's "installer-logic-tested, not install-tested". C2 PROVEN: both DEGRADED warning lines verbatim, backup.local_backup_target=local, no felhom-backup storage created, and the install did not abort. C3/C4 PARTIAL and C5 FAILED — three findings, none fixed: R-112 (P1): E-2's degraded banner and offer have NO UI CONSUMER. The endpoint returns byte-exact copy; grep 'backup-target' across every html/js/css is 0 hits and no page handler injects the state. Templates fetch 18 distinct /api/storage/* endpoints; these two are the only ones with zero references. v0.185.1 fixed the router mount and stopped one layer short of the render. Fifth instance of seam-built-but-never-wired. R-113 (P1): the drive-absent gate CANNOT FIRE on device loss. planDriveGates reads presence from BoundUnderParent = "is this path in the guest's mountinfo". The raw mount is a device-bound systemd unit and dies with the device; the agent's own bind is not device-bound and outlives it, so the gate sees "present" forever. Live: agent reported the drive absent every 20s for 4.5 minutes, the controller logged 0 [gate] lines, the hub received zero events -- neither backup_target_absent nor the generic storage_disconnected. Sixth instance of the class: E-2b wired the seam to a condition that cannot occur. R-114: on target-drive loss the message claims the backup is on the system disk (false) and offers the drive that just vanished. Invisible only because of R-112, so it must be fixed BEFORE R-112 is wired. Also filed as a second instance under R-110 rather than a new ID: host-install fetches nine files from raw/branch/main and the hub vouches a sha for one; E-2a's wrapper is installed 0755 to /usr/local/sbin, root-fenced in sudoers, validated only by bash -n. C4 is fully proven at API level: decline path (registration confers no role), restart_required:true, agent did NOT self-restart (in-flight check performed and recorded first), E-2a wrapper created the storage at the drive's own mountpoint, and healthy renders nothing. Teardown: VM destroyed, scratch storage removed, pvesm status after == before (local-lvm 38.77%), guest 9201 and drill-r50 untouched. Hub records for e2d-fresh remain -- delete correctly refused at four gates, finally "host is ONLINE"; deletable once it ages to DOWN. Command recorded in OPEN-ITEMS.md. capability-map NOT touched: the customer-facing legs are broken rather than proven, and the map has no E-2 rows at all. |
||
|
admin
|
f3f0d58844 |
E-2d: Phase 0 STOP — the Day-0 artifact channel cannot deliver the code under test
No VM created, no install run, no box touched. The run stopped at the Phase 0 gate per runbook §3, before provisioning. felhom-host-install.sh does not install what is on main. resolve_artifacts() (:423-436) reads the hub-vouched manifest and fetches Gitea GENERIC PACKAGES (agent :1945, golden :2573). Gitea's newest are agent 0.96.0 and golden 0.161.0; the hub manifest selects exactly those; the global floor v0.156.0 is below the golden's 0.161.0 so nothing self-updates. A fresh box therefore lands on agent 0.96.0 + controller 0.161.0 against main's 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by direct deploy and is not in the channel at all. Claim impact, each pinned to its introducing commit: - C1 (real rc=0 1.22.0 install) and C2 (Case B natural) — ACHIEVABLE, not run; both are installer-side and host-install is served at 1.22.0. - C3 — BLOCKED: banner + GET /api/storage/backup-target are controller v0.185.1 (cdaeb36), copy v0.185.0 (3f7cf2a). Unblocks cheaply by raising the hub floor to >=0.185.0; measured fleet impact nil (both demo boxes already 0.185.1). - C4 — BLOCKED: needs controller v0.185.1 + agent v0.113.0 (58b598b). - C5 — BLOCKED: needs controller v0.184.0 (c1a63de) + agent v0.112.0. Filed R-111 (P1): 17 unpublished agent releases (v0.97.0-v0.113.0) strand the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT, so a new customer's box installs without them. Mirror of R-110, not a duplicate. - audits/E2D-fresh-vm-2026-07-29.md — all four Phase 0 answers recorded so a resumed run does not re-derive them (cadence 30s; hot-detach available; ISO present; local-lvm fence re-measured at 38.77%, unchanged). - OPEN-ITEMS.md — R-111 opened; E-2d re-stated, NOT closed. - ROADMAP.md — R-111 under P1. - capability map NOT touched: nothing was proven live. The §5.1a operator STOP is retired — HUB_PW is in ~/.config/credentials and hub auth was verified, so CC can bind on a resumed run. |
||
|
admin
|
ff050cf409 |
C9-F1 (honesty half) + C9-F2 shipped in controller v0.183.0; C9-F1b and C9-F4 filed
Phase 0 sized C9-F1 properly before anything was designed: 43 of the 53 catalog apps have NO subtree the Tier-2 restore can read (not 2), 9 are covered only for their file legs and never their database or volumes, 1 is stateless. The asymmetry is Tier-2's alone — Tier-1 and offsite both restore the unit and replay volume dumps, so BookStack always had a working restore and only this button lied. Shipped: the restore refuses BEFORE stopping the app and names the action that does work; a run that proceeds claims only what it EXAMINED and discloses that the database and volumes are not covered. C9-F2 alarms after a 5-minute sustained-restarting threshold, set above the 120s deploy timeout, Mealie's 60s start_period and R-97b's 180s grace; StateRestarting is deliberately NOT added to IsDownState. Live: silent through ten 30s samples then app_start_failed at 5m25s, heartbeat now reads "1 currently down" where Campaign 9 recorded 0; a real deploy stayed silent; bookstack refused with its uptime unbroken; paperless re-restored 43/43 byte-identical, 16/16 docs clean. Filed, not fixed: C9-F1b (route to the Tier-1 restore — its own task because it puts a destructive operation behind a non-destructive button) and C9-F4 (nothing reads the Tier-2 copy's recovery-unit/ mirror, so the second local copy that exists for drive loss is unreachable by any customer action — potentially larger than C9-F1). |
||
|
admin
|
955083c0fc |
Campaign 9: the Tier-2 restore paths are PROVEN; 3 defects filed, none fixed
Phase A is the headline and it passed on live hardware, through the real endpoints the UI
posts to: a customer who deletes files — or their entire app data directory — gets everything
back byte-identical, and the app works afterwards (paperless served the restored bytes over
its own API at the exact pre-deletion sha256). A1's two non-destruction promises both hold.
Three defects, recorded not fixed:
C9-F1 (HIGH) the Tier-2 restore button is offered for apps it can never restore, takes a
real outage, and reports "nothing was missing" — indistinguishable from a
genuine result, while 156 MB of that app's data sits unread in the same copy.
C9-F2 (HIGH) an app in a crash loop never alarms on any channel; StateRestarting is in no
down-set, so F-OBS's own heartbeat printed "0 currently down" for 9 minutes.
C9-F3 (MEDIUM) an interrupted offsite run leaves a lock the self-heal cannot reach; the tier
is dead until a human unlocks and the operator is told "unknown reason".
This answers Phase C item 8.
Two candidates were deliberately NOT filed: a recovery-unit poisoning the catalog sync healed
in ~3 min, and a snapshot_id that is documented as logging-only. Reporting either would have
been reporting an artifact.
Stopped at the end of Phase B (plus D10), then full recovery — both boxes healthy, real
cadences, offsite tier proven working again, no leaked scratch guests, peti untouched.
D11's approved staleAfter compression turned out not to be a knob; reported, not worked around.
|
||
|
admin
|
5ffdcecb2d | F-LEAK: record all three attempts; the ACL fix is consumed by use, the fenced destroy is the fix | ||
|
admin
|
8c793955b1 |
Campaign 8: F-REBOOT/F-LEAK/F-OBS closed; R-100 filed (offsite staleness reset by failures)
F-LEAK's first fix (pool adoption) was refuted live and is recorded as such. R-100 is F-CRIT-2's defect class on the hub for the restic tier: isStale reads only LastRun, which the controller writes unconditionally on failure. Investigated, not fixed. |
||
|
admin
|
e168600148 |
docs: F-CRIT-1 + F-A1 shipped (controller v0.179.0); invariant rule
Both marked SHIPPED + PROVEN-LIVE in OPEN-ITEMS and the campaign doc. All three of Campaign 8's alarm findings are now closed (F-CRIT-1, F-CRIT-2, F-A1). Adds the standing rule earned by this arc to the versioned workspace CLAUDE.md: a comment asserting an invariant needs a test pinning it, or it is a wish — with all six shipped-false-guarantee instances catalogued, and the corollary that a test should assert the CONSEQUENCE (does the alarm fire?) not the MECHANISM (does suppression expire?). |
||
|
admin
|
d0cec9d3f3 |
docs: F-CRIT-2 shipped (agent v0.106.0) — OPEN-ITEMS, campaign doc, report
F-CRIT-2 marked SHIPPED + PROVEN-LIVE: NewestArchiveTime now counts only plausibly-complete entries (measured 1 MiB floor), verified by replaying campaign fault 2 on demo-hp in both directions. Files two follow-ups: - R-99: prune never REMOVES phantoms. It does NOT count them toward keep-last (dry-run kept 2 real + the phantom), so there is no retention/data-loss bug — but they accumulate one per aborted upload. - F-CRIT-1: filed READY-HIGHEST. Campaign 8's other HIGH finding, untouched by this fix. |
||
|
admin
|
33a7cc51d3 |
docs: CAMPAIGN 8 — GC run 2, fault 13 complete, final fleet state
GC run 2 (04:44Z, past the atime boundary) reclaimed 0 B, and the cause is the campaign itself: the datastore is relatime, so the night's three full verifies, ~14 backups, two GC runs and several restores kept refreshing chunk atimes. Sampled atimes confirm no chunk was ever 24h5m cold. Not a defect - GC made a correct, deliberate decision both times - but a real operational property: on a relatime datastore, activity defers reclamation. The spec's intended experiment therefore could not complete inside the window; recorded as such. Fault 13 now complete: backup-vs-restore-test single-flight holds, and a full two-tier backup ran successfully DURING GC run 2 (both artefacts landed, GC TASK OK, zero .bad chunks). True three-way overlap remains untested. Final fleet verification clean on all three hosts. Campaign-caused outage (F-REBOOT, 9m47s on demo-hp) stated plainly in the final-state section. |
||
|
admin
|
6c57658c1b |
docs: CAMPAIGN 8 — faults 11 and 18 results, F-LEAK root cause, evidence corrections
Tail-of-campaign additions after the Phase D revert (both re-injections declared): - fault 18 (delete a snapshot mid restore-test): detection PASS, and it ROOT-CAUSES F-LEAK — a failed restore-test cannot destroy its own scratch guest (403, missing VM.Allocate; the agent token is pool-scoped and a failed restore never joins the felhom pool) - fault 11 (guest reboot mid-backup): new finding F-REBOOT — the backup succeeds but the guest never comes back; ~9m47s outage until a manual pct start Two evidence corrections, both self-inflicted tooling errors: - pgrep -cf <pattern> matches its own ssh command line, which invalidated fault 11's first two injections and put one unsound line in fault 9 (withdrawn; that finding stands on the controller's own job state) - ep0 runs Etc/UTC, so its 03:30 prune fires at 05:30 CEST — nearly misread as a broken prune job Nine findings now, still two HIGH. Fleet healthy. |
||
|
admin
|
14b0939060 |
docs: CAMPAIGN 8 — adversarial backup & restore campaign report
Unattended 10h run against demo-felhom, demo-hp and ep0. No production code changed; findings recorded and ranked, not fixed inline. 8 findings, 2 HIGH — both in the system's ability to report that a backup did NOT happen: - F-CRIT-1: an app failing to restart after a quiesce never alarms (invariant I1 in main.go:1213 is false for the failed-restart path) - F-CRIT-2: a failed offsite backup leaves a phantom snapshot that resets the tier's freshness clock (NewestArchiveTime has no completeness check) Retires several never-validated items, including R-87 (first restic restore round-trip, byte-verified), the full R-88 backoff ladder, age_state=absent, and the crash-recovery unquiesce under a real SIGKILL. peti-felhom untouched; ep0 rollback copy intact; fleet healthy at end. |
||
|
admin
|
ad8057c4e3 |
docs(ep0): relocate the felhom-offsite PBS datastore onto the 100 GB volume
Supervised runbook execution. No code change, no version bump. felhom-offsite moved from ep0's 40 GB root disk (/srv/pbs-felhom) to a dedicated 100 GB Hetzner Cloud Volume (/mnt/pbs-datastore, ext4 -m 0, by-id fstab, relatime). Datastore NAME unchanged, so the PBS-DR descriptors, per-box storage ids, ACLs and namespaces are untouched. Capacity: 37.2 GB -> 98 GB total, 28.9% -> 13% used, headroom to the 80% warn 19 GB -> ~65 GB. This CLEARS the R-82 Phase 0 P0.3 STOP. Per-tenant encryption still precludes cross-customer dedup, so the slope is unchanged - the volume buys runway, not a better cost model. Verified: byte totals and chunk counts identical (9748), 7/7 snapshots across all three namespaces, backup:backup ownership, clean itemised dry-run, full verify job TASK OK with 0 errors, and a restore round-trip (source_tier pbs, pass true, mount_parity ok, clean teardown). Nothing deleted - the original 13 GB stays at /srv/pbs-felhom as the rollback until a new weekly backup lands. GC deliberately not run. Three findings recorded: - the `scratch` datastore points at a non-existent path (pre-existing; now logs ENOENT every start) - operator decision - the runbook's S6 guard test proves the wrong proposition: RequiresMountsFor re-mounts rather than refusing, so the test only bites when the device is genuinely unavailable (re-run that way, and the refusal was observed) - amendment recommended - S11: storage box u629193 has no live backup path, BUT ep0 carries an enabled sshfs mount unit against it that must be removed before the box is deleted Deviations: the volume arrived pre-formatted and mounted; S8 ran on demo-felhom rather than demo-hp (no SSH key for demo-hp); the window was contended by a stale in-memory 10-minute restore-test cadence whose config had already been reverted on disk. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018ARoadHBf8rHoscfiqeVZn |
||
 Claude Code
|
65e30f2e42 |
docs(audits): R-82 Phase 0 — P0.1 weekly CONFIRMED, P0.2 benign, P0.3 STOP (capacity)
Read-only gate. No code, no backup triggered, no config changed. P0.1 weekly CONFIRMED: the only 7-day-exposed state is the non-SMB half of settings.json (storage_paths, app_backup toggles, notification prefs, password_hash, launcher token) — none catastrophic, all recoverable. The two items that would have overturned it, encryption.key and the offbox credentials, are STABLE files unchanged since first boot, so a 7-day-old copy is byte-identical. Everything referentially coupled to app state is carried daily by Tier-1/2/3, and Tier-3 offsite was verified running and ok on both production boxes. CONDITIONAL: drill-r50 has offsite:null, so the verdict does not hold there. P0.2 RESOLVED, benign: PBS returns HTTP 200 with zeroed usage to the namespace-scoped token (DatastoreBackup, not Datastore.Audit), so PVE prints 0/0/0. Ground truth via the hub ep0 df op: 28.9% of 37.2 GB. Writes work — the 07-18 snapshot is owned by this token, 9.74 GB, verify ok. P0.3 STOP: 37.2 GB datastore, 10.8 GB used, 80% alert at 29.8 GB. Encrypted per-tenant backups do not dedup across customers. Current 3 boxes project to ~15-21 GB weekly keep-3; each additional customer costs ~5-10 GB, so the 80% alert fires at roughly the SECOND additional customer — inside the alpha horizon. Needs a ruling on datastore size / retention before recurring writes. |
||
|
Claude Code
|
add5b9bbbb |
docs(audits): R-80 DIAG — expected_backup_missed root-caused to in-memory agent backup store
Read-only diagnostic. No fix applied. Verdict: FALSE ALARM for demo-felhom + demo-hp, true-but-premature for drill-r50. Fired ONCE (2026-07-26 03:00 UTC), not nightly. No real external customer notified. Root cause: felhom-agent/internal/backup/store.go is in-memory by design; an agent restart empties the host-report 'backups' array until the next backup runs. The R-50 island migration restarted the fleet on 2026-07-25 12:44 UTC, opening an ~18h blind window that the 03:00 UTC deadline check fell into. Hub's assessBackupFreshness reads the empty array as 'no backup exists'. Separate real finding: the PBS/offsite-DR tier has no schedule at all (local_backup_target=local, no jobs.cfg) — demo-felhom has one snapshot from 07-18, demo-hp has zero since pbs_dr was applied 07-21. |