- OPEN-ITEMS: R-185 closed with the measurement, the corrected root cause (the installer's Scenario-F reuse arm, not PVE_STORAGES), and the live sequence. Records that demo-hp carried the same drift and was fixed too. - capability map: the whole-guest row's HOST-tier half was OPTIMISTIC and now says so — that tier was not merely unproven, it was unprovable on both demo boxes, and every live proof cited was on the offsite tier. - vzdump-target-move runbook: its item 5 predicted this; annotated (not rewritten) with what actually happened — the create arm did grant, the reuse arm did not, and it surfaced as a silent unreadable tier rather than the 403 the item expected, because vzdump writes through a root path. - CONTEXT: S-21 (an empty listing cannot distinguish forbidden from newborn; the measured trap that an ungranted path answers with INHERITED privileges) and S-22 (the Scenario-F arm must finish the job). - STATUS: rewritten for the operator, back to one screen.
This commit is contained in:
+40
@@ -17,6 +17,46 @@
|
||||
|
||||
## Standing rulings
|
||||
|
||||
**S-21 — an empty listing cannot distinguish FORBIDDEN from NEWBORN, so the box asks the permission
|
||||
question directly (2026-08-03, R-185; agent v0.123.0 + installer 1.24.0).**
|
||||
|
||||
*The defect.* On both demo boxes the agent's token had `FelhomAgentStore` on `local`, `local-lvm` and
|
||||
`felhom-pbs` and **not** on `felhom-backup` — the storage the same installer configured as
|
||||
`local_backup_target`. The content API answered `{"data":[]}` through the token while root listed
|
||||
three archives. `pickForThisRun` skipped the tier as *"no settled archive yet"*, which is exactly
|
||||
what a brand-new tier reports, so the host tier was never restore-testable and nothing said so.
|
||||
|
||||
*The rule.* The permission question has a definite answer where the listing does not. `Permissions`
|
||||
reads `/access/permissions?path=/storage/<target>` **as the agent's own token** — asking as root
|
||||
answers a different question and always says yes — and one `capability.Status` per configured tier
|
||||
reports it. The probed set comes from `BackupTiers()`, never a fixed list: a hardcoded probe list is
|
||||
the defect reproduced inside the fix.
|
||||
|
||||
*The measured trap, because the obvious reading is wrong.* An ungranted path answers **neither empty
|
||||
nor 403**: it carries the privileges inherited from the box-wide `/` grant
|
||||
(`Sys.Audit, SDN.Use, Datastore.Audit`). Testing for path-presence, or for `Datastore.Audit`, reports
|
||||
a blinded storage HEALTHY. The probe tests **`Datastore.AllocateSpace`**, and re-measuring is required
|
||||
before that constant is ever changed.
|
||||
|
||||
*Criticality, weighed once.* Critical, because the hub alerts only on critical and a non-critical
|
||||
entry would ride the report and alert nobody. **Except** the `local` fallback target, which
|
||||
host-install's own comment calls the DEGRADED configuration: still probed, still reported, but it
|
||||
does not page — turning an ordinary documented setup into an alert is how a signal becomes something
|
||||
an operator archives unread. It never consults content, so it cannot alarm on a newborn tier by
|
||||
construction, and it never reports ok when it could not ask.
|
||||
|
||||
**S-22 — the installer's Scenario-F arm must finish the job, not just leave the definition alone
|
||||
(2026-08-03, R-185).** `configure_backup_target` has two arms. Case A creates the storage and grants
|
||||
in the same breath. The reuse arm — *"the target already exists"* — returned **without granting**, and
|
||||
that, not `PVE_STORAGES`, is where the drift came from: a box whose target pre-dated the install
|
||||
(the vzdump-target-move runbook, or a reinstall) pointed `local_backup_target` at a storage its token
|
||||
could not read. The reuse arm now ensures the ACL through the same guarded wrapper. **Scenario F is
|
||||
unviolated** — the storage DEFINITION is untouched, and granting the role the agent is supposed to
|
||||
have on the target this script is about to write into `agent.json` is finishing the job, not
|
||||
retargeting the box. `$BACKUP_TARGET_ID` stays OUT of `PVE_STORAGES`: that list is granted a step
|
||||
before the target is resolved, and `--acl-storages` entries are preflight-checked for existence.
|
||||
A gate asserts every arm that resolves the target also grants on it.
|
||||
|
||||
**S-19 — a restore-test PROOF is durable and reportable; a FAILURE is neither, and that asymmetry is
|
||||
the design (2026-08-03, R-189; agent v0.122.0).**
|
||||
|
||||
|
||||
@@ -23,12 +23,18 @@ Proven end to end on real hardware.
|
||||
- **The off-site copy can be erased by the machine that made it.** The credential that writes it can
|
||||
also delete it. A daily snapshot is armed as a stopgap, and we have never restored from that copy.
|
||||
*(R-95, R-87)*
|
||||
- **Nothing else open that affects a customer.** The three faults that were on this list yesterday —
|
||||
the reserve watching the wrong step, the last app whose data was never saved, and the alert that
|
||||
told you about one app and swallowed the rest — are all fixed and proven on a real machine.
|
||||
|
||||
## What shipped recently
|
||||
|
||||
- **A backup copy the machine was never allowed to read — and could not tell you about.** One demo
|
||||
machine kept its whole-machine backups on a dedicated storage area the agent had never been granted
|
||||
permission to read. Asked what was there it was told "nothing", while an administrator saw three
|
||||
backups. **The permission was one command; the silence was the real fault** — a storage that answers
|
||||
"nothing" looks exactly like a brand-new one, which is a normal, healthy state, so that copy had
|
||||
never been test-restored and nothing had ever mentioned it. The machine now checks whether it is
|
||||
allowed to read each copy it depends on and says so when it is not — the alert reached you by email
|
||||
before the permission was granted, which is the whole point. **Both demo machines carried it and
|
||||
both are fixed**, and new machines no longer inherit it. *(R-185)*
|
||||
- **Three ways the alarm system was misreporting its own work — all fixed.** None of them ever risked
|
||||
data. **(1)** When the machine proved a backup restores, that result could vanish if the agent was
|
||||
restarted in the following quarter-hour — and yesterday's change made the gap a week rather than a
|
||||
@@ -39,17 +45,9 @@ Proven end to end on real hardware.
|
||||
published after the binary, and a new check catches the opposite mistake so nothing is traded away.
|
||||
**(3)** A released binary can now be rebuilt by anyone and checked against the fingerprint you
|
||||
approve — until today, rebuilding produced different bytes. *(R-189, R-188, R-186)*
|
||||
- **Each backup is now proved, instead of the clock being obeyed.** The machine used to re-test a
|
||||
backup every twenty-four hours on a timer that restarted whenever the software was updated, so the
|
||||
test happened at an arbitrary time and a fresh backup could sit unproven while an old one was
|
||||
re-checked. Now a copy is tested once, about a day after it is made, and not again until there is a
|
||||
newer one — so the daily copy is proved daily and the weekly off-site copy weekly, each on its own.
|
||||
The alert that says "this copy has not been proved lately" learned each copy's own rhythm in the
|
||||
same change; without that it would have started complaining every night about a system that is
|
||||
working. *(R-86)*
|
||||
- **A failed backup now tells you about every app, once**, with every failure written down whether or
|
||||
not it is emailed, and a message the machine decides not to send now leaves a record saying so.
|
||||
*(R-182)*
|
||||
- **Each backup is now proved, instead of the clock being obeyed** — tested once, about a day after
|
||||
it is made, and not again until there is a newer one; the "not proved lately" alert learned each
|
||||
copy's own rhythm in the same change. *(R-86)*
|
||||
|
||||
|
||||
## What we're working on
|
||||
@@ -67,17 +65,16 @@ Proven end to end on real hardware.
|
||||
- **One small question, not urgent.** The automatic check cannot see which version you have told
|
||||
machines to install, only which ones exist. Closing that needs either a password given to the build
|
||||
server or a check inside the hub itself. *(R-184)*
|
||||
- **Nothing else.** The question about whether the off-site endpoint counts as protected is
|
||||
**settled — it does**, and the machine list now says so instead of asking.
|
||||
- **Nothing else.**
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-03** — Found and fixed a backup copy the machine was never permitted to read, on both demo
|
||||
machines. The permission was one line; what mattered was that the machine now says so instead of
|
||||
treating "I am not allowed" and "there is nothing here yet" as the same answer. *(R-185)*
|
||||
- **2026-08-03** — Fixed three ways the alarm system misreported itself: a proof of a working backup
|
||||
that could vanish on a restart (seen happening), a release that emailed a failure for a release
|
||||
that worked, and a released binary nobody could rebuild and check. *(R-189, R-188, R-186)*
|
||||
- **2026-08-03** — Backups are now proved one at a time, each about a day after it is made, instead of
|
||||
on a timer; the "not proved lately" alert learned each copy's own rhythm so it does not complain
|
||||
about a healthy weekly copy. You settled that the off-site endpoint is protected, and the machine
|
||||
list records it. One thing found while testing: on the small demo machine the agent cannot see its
|
||||
own local backups at all — a permission that was never granted — so that copy has never been
|
||||
test-restored there. Written down, not yet fixed. *(R-86, R-185)*
|
||||
on a timer; the "not proved lately" alert learned each copy's own rhythm. You settled that the
|
||||
off-site endpoint is protected. *(R-86)*
|
||||
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -431,7 +431,7 @@ label. Filed under E-2.
|
||||
| 2 | **Assignment in the storage wizard** — suggestion by attribute, refusal of the absurd (a 32 GB FAT thumb drive), never a decision by transport or `removable` (§1.2 shows both fail on the reference hardware). |
|
||||
| 3 | **Unassigned drives do nothing automatically** — §2's rule, enforced in code. A drive must never acquire a role by appearing. |
|
||||
| 4 | **Stickiness** — an assigned target must not move because a new drive appeared, and must never silently retarget when absent. |
|
||||
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. |
|
||||
| 5 | **New installs**: `felhom-host-install.sh` must create the target storage with `--is_mountpoint 1` **and issue the `FelhomAgentStore` grant** (§4), or a new box's first backup 403s. **ANNOTATION 2026-08-03 (R-185) — this happened, in the half nobody looked at.** The installer's CREATE arm did issue the grant, exactly as this item asked. Its **reuse** arm — *"the target already exists, leave it as it is"* — returned without granting, so a box whose target pre-dated the install (i.e. one moved by THIS runbook) ended up pointing `local_backup_target` at a storage its own token could not read. It did not surface as a 403 on backup, which is what this item predicted: `vzdump` writes through a root path, so backups kept landing. It surfaced as the agent's **read** returning `{"data":[]}` while root saw three archives — so the tier was silently never restore-tested. Both demo boxes carried it. Closed by installer **1.24.0** (the reuse arm grants too, with a gate asserting every arm that resolves the target also grants on it) and agent **v0.123.0** (the box now asks whether it may read each tier, because an empty listing cannot distinguish forbidden from newborn). |
|
||||
| 6 | **Absent-target policy** per §6: decide fallback-vs-fail, and if fallback, alarm that protection is degraded rather than reporting a healthy tier. |
|
||||
| 7 | **Retention and space accounting** on a drive the customer also uses — today `keep-last=3` competes with customer data with no reservation and no ceiling. |
|
||||
| 8 | The honest **single-drive label**. |
|
||||
|
||||
Reference in New Issue
Block a user