docs: R-165 merge built and proven at the bake; R-163 + R-175 closed, R-178 filed
gates / gates (push) Successful in 8s
gates / gates (push) Successful in 8s
07-backup-architecture.md gains §7.5.1 (S-1: the contract changed in the same session): the ceiling §7.5 describes no longer exists for a box built from golden >= 0.192.0, the bulkhead's replacement is recorded, and R-175 is FIXED here rather than left standing — the bound is restated as a function of mp1 and scoped to split-layout boxes, naming all three real shapes. Capability map: new row as IMPLEMENTED, deliberately NOT proven-live, with the missing leg named — no box has been reinstalled from the golden, and "the golden baked" is not "a box built from it works". R-163 CLOSED: the ceiling it recorded stops existing. R-176(a) answered by P1; (b) WITHDRAWN, since every node is reinstalled rather than migrated. R-178 filed for the reinstalls, which were not done this session. CONTEXT S-13 (the variant chosen on measurement; pruning rejected with its reason) and S-14 (prove first, then vouch — the golden is published but deliberately unvouched, because vouching is what makes a fresh install pick up a layout no box has been proven from). STATUS: plain-language section; both operator questions now answered, so the waiting-on-you item is cleared. Two older entries trimmed so the page did not grow.
This commit is contained in:
+29
@@ -17,6 +17,35 @@
|
||||
|
||||
## Standing rulings
|
||||
|
||||
**S-13 — the `mp1` merge landed, and the variant was chosen on measurement (2026-08-03, R-165 / D-a).**
|
||||
The appliance's two data volumes are one. **Variant V-c**: the volume mounts at the NEUTRAL path
|
||||
`/var/lib/felhom`, and both `/var/lib/docker` and `/mnt/sys_drive` are binds of subdirectories of it.
|
||||
**Three shapes were built and rebooted before choosing** (`audits/SPIKE-r165-phase0-2026-08-03.md`) —
|
||||
all three boot, reboot 3/3, give ONE `df` figure and keep a container's `statfs("/")` on the merged
|
||||
volume, so **the ordering risk that motivated the probe was not what mattered.** They differ only in
|
||||
which documented guarantee they break: volume-at-`/var/lib/docker` puts customer backups inside
|
||||
Docker's data-root, so the ordinary "clear `/var/lib/docker`" reflex destroys every local unit;
|
||||
volume-at-`/mnt/sys_drive` puts Docker's entire data-root under `/mnt`, which the controller container
|
||||
mounts wholesale — **measured: it then sees `/mnt/sys_drive/docker`**, falsifying the bootstrap's own
|
||||
comment that `/mnt` holds only Felhom's namespace mounts. **V-c breaks neither**, for one extra path.
|
||||
|
||||
**B2 is the bulkhead replacement, and "or prune the oldest" is REJECTED with its reason**, because the
|
||||
question will be asked again: nothing on that filesystem is generational — a unit is ONE fixed path per
|
||||
app (`backups/primary/<app>`) refreshed in place, and a DB dump is `<stack>-<dbtype>.sql`, also fixed —
|
||||
so pruning could only mean deleting a **different** app's only local recovery unit.
|
||||
`pruneStalePrimaryDirs` is an ORPHAN sweep with no notion of age and must never be repurposed.
|
||||
|
||||
**No migration exists, and that is a ruling not an omission:** every node is REINSTALLED. Both demo
|
||||
boxes are Tier 0; the colleague's box carries none of our customer data and is clean-installed shortly.
|
||||
So R-176's in-place migration rehearsal is **withdrawn**, not deferred.
|
||||
|
||||
**S-14 — prove first, then vouch (2026-08-03).** Golden **0.192.0** is baked, published and verified in
|
||||
the registry, and is **deliberately UNVOUCHED**. Vouching is what makes a fresh install pick a golden
|
||||
up, so vouching one that no box has been proven from would put an unproven disk layout in front of the
|
||||
next install anywhere. The bake script already treats the hub record as a separate deliberate step; this
|
||||
makes the ordering a rule. **The remaining work is R-178** — reinstall each demo box from the merged
|
||||
golden, prove claim → deploy → back up → restore, **one box at a time**, and only then vouch.
|
||||
|
||||
**S-11 — D-c's routing, and why R-158's own proposal was overruled (2026-08-02, R-167 SHIPPED).**
|
||||
Decision D-c splits two signals by AUDIENCE, and the split is the ruling: **a fill warning is the
|
||||
CUSTOMER's** (they can free space, delete files, add a drive) and **a per-app backup capture failure
|
||||
|
||||
@@ -1,147 +1,83 @@
|
||||
# REPORT — hub v0.89.0: the two halves of decision D-c, plus the R-165 merge spike (2026-08-02)
|
||||
# REPORT — R-165: the `mp1` merge, built and proven at the bake (2026-08-03)
|
||||
|
||||
**Overwritten** per the standing rule. The prior contents (R-168, the CI runner, same day) have their
|
||||
durable record in `scripts/CHANGELOG.md` and `CONTEXT.md` S-8/S-9/S-10.
|
||||
**Overwritten** per the standing rule. The prior contents (hub v0.89.0 / R-167, 2026-08-02) have their
|
||||
durable record in `hub/CHANGELOG.md` and `CONTEXT.md` S-11/S-12.
|
||||
|
||||
**Companion report:** `felhom-controller/REPORT.md` holds the controller side (v0.191.0/.1/.2), the
|
||||
full red-proof table, the Hungarian copy, and the live evidence for all three flows. This file covers
|
||||
the hub change, the documentation coupling, and **Part 3's spike**.
|
||||
**Companion:** `felhom-agent/REPORT.md` holds the full session detail — probes, variant evidence, the
|
||||
bake transcript, red-proofs and teardown. **This file covers what changed in THIS repo, and the CI
|
||||
verification for all three.**
|
||||
|
||||
> **Scope, stated first.** The merge is **built and green through Phase 5**. **Phases 6–7 — reinstalling
|
||||
> the two demo boxes from the merged golden and proving one end to end — were NOT done, and nothing
|
||||
> was wiped.** The golden is therefore deliberately **unvouched**. Remaining work: **R-178**.
|
||||
|
||||
---
|
||||
|
||||
## 1. Baseline drift — recorded, because the task's §1 was wrong
|
||||
## 1. What changed in `felhom.eu`
|
||||
|
||||
The task targeted hub **v0.87.0 → v0.88.0**. On arrival `main` was at `8ef92a3f` with hub **v0.88.0
|
||||
already shipped** (R-172, the WAL fix), not `d5774d318941`/v0.87.0. Target corrected to **v0.89.0**.
|
||||
Highest register ID in use was **R-173**, not R-171.
|
||||
**No hub change; no hub version bump.** This repo changed in two places, one of them forced.
|
||||
|
||||
## 2. Hub change (v0.89.0)
|
||||
**`scripts/felhom-host-install.sh` — forced by a census, not planned.** `step_grows` computed **two**
|
||||
volume sizes and the install call passed both, so it had to change with the agent or every install
|
||||
would have provisioned a half-sized box. It now computes ONE total, summing the old 80/20 split
|
||||
(`226` = the previous `184 + 42`), so **a standard appliance keeps exactly the 250 G it had** — it is
|
||||
simply no longer split by a wall. **The size still comes from the physical disk**: `step_grows`
|
||||
already read the thin pool's real free space, and the merge only collapsed its two outputs into one.
|
||||
That is the answer to the task's §8.2 — no row needed filing. `--sysdata-grow` is deprecated but still
|
||||
honoured, because the agent **folds** a hand-passed value in rather than dropping it.
|
||||
|
||||
**One new event type, not two.** The task called for a new customer-facing type *and* a new
|
||||
operator-only one. Reconnaissance found `disk_warning`/`disk_critical` already allowlisted here, with
|
||||
Hungarian `customerMessages`, in the controller's `DefaultEnabledEvents` and behind a UI checkbox —
|
||||
**and with no producer in any repo.** The operator chose to wire that inert pair rather than mint a
|
||||
near-duplicate, so only the operator type is new.
|
||||
**Documentation**, per the coupling rule — see §3.
|
||||
|
||||
| Change | File | Why |
|
||||
|---|---|---|
|
||||
| `+ "recovery_unit_capture_failed"` | `internal/api/handler.go` (`allowedEventTypes`) | without it the controller's POST 400s and the event vanishes |
|
||||
| `+ "recovery_unit_capture_failed"` | `internal/notify/dispatcher.go` (`operatorOnlyEvents`) | **this** is what makes it operator-only; the allowlist does not, and v0.78.0 claimed otherwise and shipped the defect |
|
||||
| `- customerMessages["disk_warning"]`, `- ["disk_critical"]` | `internal/notify/templates.go` | `FormatCustomerEmail` PREFERS the entry over the message, so a static template would discard the drive label and the free-space figures the controller now sends. Same reason `offbox_enlarge_blocked` and `disk_health_degraded` have none |
|
||||
| `+ func IsOperatorOnly` | `internal/notify/dispatcher.go` | lets the `api` package pin BOTH registers in ONE test; checked separately, allowlisted-but-not-operator-only is invisible. Read-only — the register stays unexported so nothing can widen it at runtime |
|
||||
| `REUSE.md` §5 "new event type" rewritten | `REUSE.md` | it told readers to always add a `customerMessages` entry, which is **wrong** for operator-only types and **harmful** for dynamic-message ones |
|
||||
## 2. The bake, as this repo's audit trail records it
|
||||
|
||||
**Tests 574 → 579**, full suite green (`go build ./... && go vet ./... && go test ./...`), all five
|
||||
`repo_gates.py` gates OK.
|
||||
`documentation/audits/SPIKE-r165-phase0-2026-08-03.md` (new) holds P1, P2 and P3 with method,
|
||||
measurement and ruling, including the teardown of every probe artefact at all three layers. The
|
||||
headline measurements:
|
||||
|
||||
**Red-proof (Scenario G), demonstrated not argued:** removing `recovery_unit_capture_failed` from
|
||||
`operatorOnlyEvents` fails two tests, one reading *"a customer was emailed the OPERATOR-ONLY
|
||||
recovery_unit_capture_failed (customer@example.com)"*. The dispatch test runs under the **breaking**
|
||||
configuration — the customer has the event enabled and an email set — because that is the only
|
||||
configuration in which the missing entry is visible.
|
||||
- **P1 PASS** — a real pre-merge archive restore-tests clean, `mount_parity: ok`, 84 s, `mountParity`
|
||||
untouched.
|
||||
- **P2** — all three variants boot and reboot 3/3; they are separated by **scoping**, not mechanics.
|
||||
The container's view of `/mnt` is 8.0K under V-a and V-c, and **17.9M — Docker's entire data-root —
|
||||
under V-b**. The operator chose **V-c**.
|
||||
- **P3** — the four retargeted golden assertions, run against a deliberately wrong shape: **8/8**.
|
||||
|
||||
**Live (guest 9201 → hub):** both event types accepted and stored; `operator | sent`; and the positive
|
||||
observable `customer | recovery_unit_capture_failed | skipped | operator_only` read from
|
||||
`notification_log`. The customer half: `customer | disk_warning | sent` and `customer | disk_critical
|
||||
| sent` with the dynamic Hungarian intact.
|
||||
|
||||
**Deploy:** GitOps only — `manifests/hub.yaml` bumped 0.88.0 → 0.89.0 (`6d359a5`), pushed, then a
|
||||
deliberate ArgoCD hard-refresh + sync. Never `kubectl set image`. App `felhom` **Synced / Healthy**,
|
||||
`deploy/hub` rolled out, running `gitea.dooplex.hu/admin/felhom-hub:0.89.0`, startup log clean.
|
||||
|
||||
## 3. Part 3 — the R-165 spike. **M1-M5 each answered; nothing was changed.**
|
||||
|
||||
Full document: `documentation/audits/SPIKE-r165-mp1-merge-2026-08-02.md`. No partition was created,
|
||||
resized, moved or deleted; no golden rebuilt; no guest config edited. `ep0` and Peti's box were not
|
||||
contacted (D-d, `runbooks/target-selection.md`).
|
||||
|
||||
**M1 — what is actually there. ANSWERED, and it contradicts the architecture doc.**
|
||||
|
||||
| | demo-felhom | demo-hp | golden default |
|
||||
|---|---|---|---|
|
||||
| `mp0` `/var/lib/docker` | **200 G** (13 G used) | **50 G** (5.4 G used) | 16 G |
|
||||
| `mp1` `/mnt/sys_drive` | **50 G** (2.0 G used, 5%) | **20 G** (92 M used, 1%) | 8 G |
|
||||
|
||||
§7.5 documents the appliance as `mp0 50G / mp1 20G` — that is demo-hp exactly and **not** demo-felhom.
|
||||
Any merge plan expressed as a fixed pair is already wrong for one of the two boxes that exist. §7.5's
|
||||
headline bound (*"≈ 19 GB … ≈ 10 GB"*) is derived from `mp1 = 20 G` and is therefore one box's, not
|
||||
the fleet's → **R-175**, filed and §7.5 annotated in this session.
|
||||
|
||||
**M2 — what lives on `mp1`. ANSWERED, and it is not only backups.** Four things would move:
|
||||
Tier-1 units of driveless apps (269 M, ~30 apps on demo-felhom), **Tier-2 mirrors (1.7 G — i.e. the
|
||||
MAJORITY is Tier 2, not Tier 1)**, the `userdata/import` drop zone which lives on the system drive by
|
||||
**contract** (R-75), and the system-data userdata namespace. Observed fill is 5% / 1%: the constraint
|
||||
is a **ceiling** problem, not a current-fill one.
|
||||
|
||||
**M3 — which merge shapes exist. ANSWERED for three shapes, with ONE item explicitly unmeasured.**
|
||||
The golden **fails closed on the split in four places**, not one (`build-golden.sh:126,130`
|
||||
separate-mount asserts + `:315,319` vzdump-exclusion guards). The archive scope `rootfs+mp0+mp1` stays
|
||||
complete after a merge (the data moves onto `mp0`). `mountParity` holds for new archives. **Unmeasured
|
||||
and reported as such:** whether a *pre-merge* archive restore-tests into a *merged* guest — reading
|
||||
`mountParity` says it should, but that is reasoning from source about an unvalidated mechanism, which
|
||||
this project has got wrong four times → **R-176**.
|
||||
|
||||
**M4 — the bulkhead. ANSWERED, and it is the important one.** `mp1` is not only a ceiling: today an
|
||||
overflow is refused per app with the last good unit byte-identical **and cannot reach
|
||||
`/var/lib/docker`**. After the merge it can, and a full Docker data-root is a stopped box, not a slow
|
||||
one. Four replacements costed — a reserved block percentage, **a refusal threshold in the capture
|
||||
path**, a project quota, or deeming R-167's warnings sufficient — with the trade-off of each.
|
||||
**Deliberately not chosen: this is the operator's ruling.**
|
||||
|
||||
**M5 — existing boxes. ANSWERED for the measurable population; one part honestly UNMEASURED.** The
|
||||
hub's `/hosts` register holds four hosts, **two ONLINE**, both demo boxes — and both are **Tier 0,
|
||||
therefore reinstallable rather than migratable (D-d)**, so migration cost for the measurable population
|
||||
is **zero**. **D-a's condition (1) — "before any external install" — is currently SATISFIED**, which
|
||||
makes this the cheapest this decision will ever be. **`peti-felhom` exists as a customer with NO host
|
||||
in the register**, so its layout is not knowable from the hub and the box was not contacted; whether it
|
||||
needs converting or reinstalling is the operator's information. The in-place migration procedure has
|
||||
**never been rehearsed**, so "is the box restorable at every point of it?" is currently unknown → also
|
||||
**R-176**.
|
||||
|
||||
**Ranked options and recommendation:** (1) **S1 — one volume with the two paths as directories — plus
|
||||
B2, a refusal threshold in the capture path**, shipped as a fresh-install shape with the demo boxes
|
||||
reinstalled; (2) S1 + warnings only; (3) S3, grow `mp1` and keep the split (D-a's rejected baseline,
|
||||
measured for comparison); (4) S2, two mounts on one pool — **not recommended at all**, it satisfies
|
||||
every assertion while delivering none of the benefit and converts a clean per-app refusal into a
|
||||
shared-pool exhaustion neither `df` can see coming.
|
||||
|
||||
**STOPPED at the operator's question**, per the task. The merge is next session's supervised work.
|
||||
|
||||
## 4. Documentation coupling
|
||||
## 3. Documentation coupling
|
||||
|
||||
| File | Change |
|
||||
|---|---|
|
||||
| `documentation/backlog/OPEN-ITEMS.md` | **R-158** closed (by R-167 — *no second row for the same wire*); **R-167** closed; **R-165** updated with M1-M5 + the operator question, stays open; **4 new rows** R-174/175/176/177 |
|
||||
| `documentation/backlog/ROADMAP.md` | R-158 collapsed to a shipped one-liner; R-167 added as shipped; R-165 added as spiked/waiting-on-operator |
|
||||
| `documentation/architecture/00-capability-map.md` | **two new rows**, both **PROVEN-LIVE** with live citations |
|
||||
| `documentation/architecture/07-backup-architecture.md` §7.5 | **S-1: the contract changed in the same session.** The section's closing claim *"nothing warns when an app crosses the line"* is now false; the alerting is written in, and the one-box-vs-fleet caveat added |
|
||||
| `CONTEXT.md` | **S-11** (D-c's routing, and why R-158's own `backup_failed` proposal was overruled) and **S-12** (the monitoring landed *before* the merge, not with it) |
|
||||
| `STATUS.md` | new plain-language section; the merge decision added to *Waiting on you*; **two older entries trimmed** so the page did not grow — one screen, per its own rule |
|
||||
| `REUSE.md` | the "new event type" extension point rewritten (see §2) |
|
||||
| `documentation/architecture/07-backup-architecture.md` | **S-1: the contract changed in the same session.** New **§7.5.1** — the ceiling §7.5 describes no longer exists for a box built from golden ≥ 0.192.0, the bulkhead's replacement (B2) is recorded, and **R-175 is FIXED here**: the bound is restated as a function of `mp1` and scoped to split-layout boxes, naming all three real shapes |
|
||||
| `documentation/architecture/00-capability-map.md` | new row — **IMPLEMENTED, not PROVEN-LIVE**, with the missing leg named (no box reinstalled, R-178) and the bake cited as the evidence it is |
|
||||
| `documentation/backlog/OPEN-ITEMS.md` | **R-165** → shipped-not-yet-proven-live; **R-163 CLOSED** (its ceiling no longer exists); **R-175 CLOSED**; **R-176** (a) answered by P1, (b) **withdrawn** — every node is reinstalled, not migrated; **R-178 NEW** |
|
||||
| `CONTEXT.md` | **S-13** (the variant, chosen on measurement; B2's floor; pruning rejected with its reason; no migration exists) and **S-14** (prove first, then vouch) |
|
||||
| `STATUS.md` | rewritten section in plain language; the *Waiting on you* item cleared — both questions are answered; **two older entries trimmed** so the page did not grow |
|
||||
| `scripts/CHANGELOG.md` | the host-install change, with why it was forced |
|
||||
|
||||
## 5. Register IDs
|
||||
## 4. CI — run ids and conclusions
|
||||
|
||||
**Opened:** R-174, R-175, R-176, R-177. Each established free by
|
||||
`grep -ro "R-17n\b" documentation/ *.md` → **0 hits**, run before minting.
|
||||
**Closed:** R-158, R-167, R-174. **Updated, still open:** R-165, R-163 (unchanged — it stays the
|
||||
record of the constraint until the merge lands).
|
||||
|
||||
## 6. CI — run ids and conclusions
|
||||
|
||||
Checked by PULL from `…/actions/tasks`, matching `head_sha` to each commit — CI emails only on
|
||||
failure, so a green that was never looked at is an assumption, not an observation. **Every commit this
|
||||
session, both repos, is green.**
|
||||
Checked by **PULL**, matching `head_sha` to each commit — CI mails only on failure, so an unchecked
|
||||
green is an assumption.
|
||||
|
||||
| Repo | Commit | Task id | Run # | Conclusion |
|
||||
|---|---|---|---|---|
|
||||
| `felhom-controller` | `cf48214` (v0.191.0) | 31 | 11 | **success** |
|
||||
| `felhom-controller` | `5adae4d` (v0.191.1) | 34 | 12 | **success** |
|
||||
| `felhom-controller` | `9a3c485` (v0.191.2) | 35 | 13 | **success** |
|
||||
| `felhom.eu` | `179dd79` (hub v0.89.0) | 32 | 17 | **success** |
|
||||
| `felhom.eu` | `6d359a5` (manifest 0.89.0) | 33 | 18 | **success** |
|
||||
| `felhom.eu` | `41dbecb` (docs) | 36 | 19 | **success** |
|
||||
| `felhom-controller` | `4be6467` (v0.192.0) | 39 | 15 | **success** |
|
||||
| `felhom-agent` | `cd6e267` (v0.120.0) | — | — | *see below* |
|
||||
| `felhom-agent` | `4bb84fc` (REPORT) | — | — | *see below* |
|
||||
| `felhom.eu` | `7406ac7` (phase-0 audit) | 40 | 21 | **success** |
|
||||
| `felhom.eu` | `e3525e6` (host-install) | — | — | *see below* |
|
||||
|
||||
## 7. `--no-verify`
|
||||
*(The remaining rows are filled in from `…/actions/tasks` after the final push; any that had not
|
||||
finished at write time are listed with their status rather than assumed green.)*
|
||||
|
||||
**Not used anywhere.** Every push in this session ran `.githooks/pre-push` (`repo_gates.py --fast` /
|
||||
`controller_gates.py --fast`) and passed.
|
||||
## 5. `--no-verify`
|
||||
|
||||
**Not used anywhere.** Every push in this session ran its repo's `.githooks/pre-push` and passed.
|
||||
|
||||
## 6. What remains — R-178
|
||||
|
||||
1. Reinstall **demo-hp** from golden 0.192.0 through the real installer path; prove claim → deploy an
|
||||
app → back up → restore; show `df` proving one filesystem and a recovery unit landing on it.
|
||||
2. Only then reinstall **demo-felhom** (it carries the PBS-DR/offsite tier, so it is the box whose
|
||||
backup chain a reinstall actually disturbs).
|
||||
3. **Then** vouch golden 0.192.0 in the hub, and flip the capability-map row to PROVEN-LIVE.
|
||||
4. Re-run P1's restore-test against agent v0.120.0 — one command, turns a sound inference into an
|
||||
observation.
|
||||
|
||||
@@ -65,12 +65,32 @@ barrier against a runaway backup filling the space the machine needs to run; put
|
||||
first means that when it comes down, the thing watching is already working and already tested.
|
||||
*(R-167, R-158)*
|
||||
|
||||
**The backup partition is gone from the base image.** A machine built from now on has one storage area
|
||||
instead of two, so a backup can use whatever space the machine actually has free rather than a fixed
|
||||
slice decided when it was built. The wall does not move; it stops existing.
|
||||
|
||||
**What replaced the wall.** It was quietly doing a second job — keeping a runaway backup from eating
|
||||
the space the machine needs to keep running. That job is now explicit: if a backup would push the disk
|
||||
below a safe reserve, **that one app's backup is refused, its last good copy is left exactly as it
|
||||
was, and you are told.** Nothing is ever deleted to make room; every app has only one local copy, so
|
||||
"delete the oldest" would always mean destroying some other app's only copy.
|
||||
|
||||
**How the shape was chosen — worth one line, because it was not the obvious one.** Three ways of doing
|
||||
it were built and rebooted rather than argued about. All three worked. They differed in what they
|
||||
quietly broke: one put your backups inside Docker's own storage, where the normal way of fixing a sick
|
||||
Docker would wipe them; another exposed all of Docker's internals to the part of the system that
|
||||
manages your drives. The third does neither, and costs one extra line of configuration.
|
||||
|
||||
**Nothing has changed on any existing machine.** They keep their current layout and go on working
|
||||
exactly as before; they get the new shape only when they are reinstalled. The new base image is
|
||||
deliberately **not switched on yet** — nothing will pick it up until a machine has been rebuilt from
|
||||
it and checked, which is the next step. *(R-165, R-163)*
|
||||
|
||||
## What we're working on
|
||||
|
||||
- **Now:** the last app whose data was never saved; today's decisions written down.
|
||||
- **Next:** merging the small backup partition into the large one — **the warnings for it are already
|
||||
done and working**, so this step is now only the partition change. It needs one decision from you
|
||||
first (below).
|
||||
- **Next:** finishing the partition merge — **the build is done and the decision is made**; what is
|
||||
left is to reinstall the two demo machines from the new base image and check one end to end.
|
||||
- **After:** rebuilding how the machine records whether an app is meant to be running.
|
||||
|
||||
## Waiting on you
|
||||
@@ -82,35 +102,23 @@ first means that when it comes down, the thing watching is already working and a
|
||||
settle both. *(R-110, R-115)*
|
||||
- **A job, not a decision: the hub password needs changing.** A diagnostic command printed it into a
|
||||
session log; nothing suggests anyone else saw it. *(R-132)*
|
||||
- **The partition merge: one decision, now measured.** Removing the backup partition also removes a
|
||||
barrier — today a runaway backup is refused on its own and cannot touch the space the machine needs
|
||||
to run; afterwards it can, and a machine out of that space is stopped, not slow. So: do we add a
|
||||
hard stop that refuses a backup before it eats the last of the room, or do we rely on the new
|
||||
warnings? The recommendation is the hard stop, because it keeps exactly what the barrier gave us.
|
||||
**Second question, which only you can answer:** does the tester's box need converting in place, or
|
||||
can it be reinstalled? It does not report to the hub, so nothing here can tell. *(R-165, R-176)*
|
||||
- **Nothing — both partition-merge questions are answered.** You chose the storage shape and the hard
|
||||
stop; both are built. The tester's box needs no conversion: it will simply be reinstalled. *(R-165)*
|
||||
|
||||
## Changed since last update
|
||||
|
||||
- **2026-08-02** — The false "host offline" warning is fixed, and the cause was not what it looked
|
||||
like. The hub's database was supposed to be in a mode where reading a page cannot block a machine's
|
||||
status update — the code said so, but a one-word syntax difference meant the setting had **never
|
||||
taken effect**, for the hub's whole life. So opening an operator page could make a machine's report
|
||||
fail; two failures in a row crossed the half-hour threshold and sent you an alert about a machine
|
||||
that was up and healthy. It had already done that twice that day. Now genuinely fixed and verified
|
||||
live. **Also found while checking it: the hub's own database is not in any automatic backup** — it
|
||||
holds every machine's emergency password and the escrow records. Filed, not yet fixed.
|
||||
- **2026-08-02** — The false "host offline" warning is fixed. The hub's database was supposed to be in
|
||||
a mode where reading a page cannot block a machine's status update; a one-word difference meant that
|
||||
setting had **never taken effect**, for the hub's whole life. Fixed and verified live. **Also found:
|
||||
the hub's own database is in no automatic backup** — it holds every machine's emergency password.
|
||||
Filed, not yet fixed.
|
||||
|
||||
- **2026-08-02** — Boot recovery finished: the machine records what the customer asked for, and waits
|
||||
for the system to finish starting before deciding what is missing. Six hard resets, everything back
|
||||
every time. A hole the previous day's change had opened — starting an app whose external drive was
|
||||
missing — was found by reading the code, reproduced on the demo box first, and fixed the same day.
|
||||
**A second instance of the same hole was found and fixed today**, on the path that restarts an app
|
||||
after an interrupted backup.
|
||||
- **2026-08-02** — Boot recovery finished; six hard resets, everything back every time. Two instances
|
||||
of the same hole — starting an app whose external drive was missing — were found by reading the code
|
||||
and fixed the same day.
|
||||
|
||||
- **2026-08-02** — Thirteen mechanical checks had built up across the four repositories and nothing
|
||||
ran most of them; two were failing quietly, one since 14 July. Both fixed, and the arrangement that
|
||||
replaced them is described above.
|
||||
- **2026-08-02** — Thirteen mechanical checks had built up and nothing ran most of them; two were
|
||||
failing quietly. Fixed, and the arrangement that replaced them is described above.
|
||||
- **2026-08-02** — Decided: the 20 GB backup partition goes away and shares space with app data. That
|
||||
changes the disk layout, so it happens before any machine is installed outside the house. **Measured
|
||||
since:** no machine outside the house is registered yet, so this is as cheap now as it will ever be;
|
||||
|
||||
@@ -86,6 +86,7 @@
|
||||
| Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE (2026-07-25)** | **R-50 SHIPPED and deployed to the whole fleet.** The control plane now rides a host-internal, portless island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original `bind: cannot assign requested address` daemon-death) + cold-reboot survival (`SPIKE-island-bridge-2026-07-25.md`), the migration runbook run verbatim (`RUNBOOK-island-migration.md`), a fresh provision auto-attaching the island `net1` (A4), and the live migration of **both demo boxes** (demo-hp + demo-felhom, 2026-07-25) — island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), **apps served throughout (0 container restarts)**, hub reporting 0.96.0. **Origin:** `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box | **Fleet: DONE.** Remaining: **R-74** — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped |
|
||||
| **The customer is warned BEFORE a filesystem fills** — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered | controller **v0.191.0/.1/.2**, hub **v0.89.0** (R-167, decision D-c) | **PROVEN-LIVE (2026-08-02)** | `audits/SPIKE-r165-mp1-merge-2026-08-02.md` (context) + `felhom-controller/REPORT.md`. Exercised on guest 9201 against a REAL filesystem (`/mnt/sys_drive` filled with `fallocate`): **`disk_warning` at 90% used / 4.7 GB free** → hub `notification_log` `customer | disk_warning | sent` with the dynamic Hungarian rendered; grown to 1.7 GB free → **`disk_critical`** → `customer | sent`; file removed → `critical → ok … cleared silently, re-armed` and the persisted state emptied. **Exactly two events across three boots** — the boot in between produced none, which is the edge trigger holding | **Nothing warned before this.** The only prior signal was the healthcheck's generic `health_degraded` at 90%, for REGISTERED STORAGE PATHS ONLY — it never looked at the docker area or the system-data area, never gave a free-byte figure and never named a drive. **The two event types already existed with NO PRODUCER** (`disk_warning`/`disk_critical`: allowlisted, copy'd, in `DefaultEnabledEvents`, checkbox'd) — the **sixth** *built-but-never-wired* instance here; this ships their producer rather than a seventh near-duplicate type. **Two threshold terms, whichever trips first, and the live proof vindicated the design:** the critical crossing fired on the FREE-BYTE term (1.7 GB) at only **91%** used — a percentage-only rule would have missed it. The hub's generic `customerMessages` entries were REMOVED, because `FormatCustomerEmail` prefers the entry over the message and would discard the label and figures. **Known gap → R-177:** there is no operator-triggerable run-now path; the check is daily 03:30 + once at startup, so confirming a cleared warning on a support call needs a controller restart or a wait |
|
||||
| **A failed per-app Tier-1 backup reaches the OPERATOR** (app, error, and the target filesystem's used/free bytes at the moment of failure) | controller **v0.191.0**, hub **v0.89.0** (R-158, closed by R-167) | **PROVEN-LIVE (2026-08-02)** | `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing | **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it |
|
||||
| **A local backup is bounded by the box's FREE SPACE, not by a partition set at build time** — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime | golden `build-golden.sh` **v3.0.0**, agent **v0.120.0**, controller **v0.192.0** (R-165 / D-a / B2) | **IMPLEMENTED — NOT PROVEN-LIVE** | `audits/SPIKE-r165-phase0-2026-08-03.md` (P1/P2/P3) + the bake transcript. **The golden bake is real evidence and is cited as such:** `build-golden.sh v3.0.0` produced `including mount point mp0 ('/var/lib/felhom')` with **no `mp1` line at all**, and its own guards printed `/var/lib/docker is a real mount`, `/mnt/sys_drive is a real mount` and `both paths are ONE filesystem`. Archive published (registry HTTP 200, sha `54e2a4c4…`). The B2 floor is unit-proven with 3 red-proofs and live on 9201 | **NOT PROVEN-LIVE, and the missing leg is named: no box has been reinstalled from this golden (R-178).** Per this map's own rule a PROVEN-LIVE claim needs an end-to-end citation, and "the golden baked" is not "a box built from it works". **The golden is deliberately UNVOUCHED** so no fresh install picks up an unproven layout — prove first, then vouch (`CONTEXT.md` S-14). Every box in the field is still on the SPLIT layout and is unaffected: nothing assumes the merged shape at runtime, the controller's system_data_path is a path rather than a volume, and agent v0.120.0 FOLDS the retired `-sysdata-grow` into the single grow so an older `felhom-host-install.sh` still provisions the same total capacity |
|
||||
| Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | |
|
||||
| **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. **2026-07-20: H4 closed in controller v0.153.0 (R-47) on BOTH paths, AND THE EVIDENCE RUN HAPPENED.** *(The "closing in v0.149" wording above was wrong — v0.149.0 was the F3 dashboard fix; R-47 shipped in v0.153.0.)* The C6 drill ran end-to-end **through the UI**: photos deleted, **trash emptied**, the full files+database restore pressed on `/backups/restore`, 40 files placed + 1 DB dump replayed rc-0, 11 assets back, no drift, timeline visually confirmed. The method note below is now DEMONSTRATED, not merely written down. Evidence: `felhom-controller/REPORT.md` 4e. **Residual: the run was performed by the OPERATOR, not by a customer** — for this row literal wording the alpha still owes one genuinely customer-driven pass, but no product gap blocks it. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count **Lane split → `07-backup-architecture.md` §3**: this row is Lane 1 (customer, unassisted). §8 rows 1–5 are the routes it would exercise |
|
||||
|
||||
|
||||
@@ -545,11 +545,38 @@ decision D-c; controller v0.191.x + hub v0.89.0).** The last sentence of this se
|
||||
**operator-tier** (`notify.operatorOnlyEvents`) and deliberately not `backup_failed`: a customer can
|
||||
take no action on a capture failure.
|
||||
|
||||
**A caveat this section must carry, because the bound below depends on it.** The `mp0 50G / mp1 20G`
|
||||
table above is **demo-hp's** shape, not the fleet's — demo-felhom ships `mp0 200G / mp1 50G`, where
|
||||
the same bound is ≈ 49 GB / ≈ 24 GB, and the golden's own defaults are `16 G / 8 G` before provision
|
||||
grows them. **The bound below is a FUNCTION of `mp1`, not a constant.** Measured 2026-08-02,
|
||||
`audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1; correcting the numbers throughout is **R-175**.
|
||||
### 7.5.1 — THE CEILING THIS SECTION DESCRIBES HAS BEEN REMOVED (2026-08-03, R-165 / decision D-a)
|
||||
|
||||
**Everything above describes the SPLIT layout, which is now the legacy shape.** A golden built by
|
||||
`build-golden.sh` **v3.0.0** ships **one** data volume; `mp1` does not exist. Both consumer paths are
|
||||
binds of subdirectories of it (variant **V-c**):
|
||||
|
||||
```
|
||||
mp0 -> /var/lib/felhom ├─ docker/ --bind--> /var/lib/docker
|
||||
└─ sys_drive/ --bind--> /mnt/sys_drive
|
||||
```
|
||||
|
||||
**So the size bound below no longer applies to a box built from that golden.** A driveless app's
|
||||
recovery unit is limited by the box's actual free space, not by a partition set at build time. The
|
||||
mismatch table above (`mp0` 50 G vs `mp1` 20 G) describes what a merged box no longer has.
|
||||
|
||||
**R-175, fixed here rather than left standing.** The bound below was stated as the fleet's and was
|
||||
**one box's**: it is derived from `mp1 = 20 G`, which is demo-hp exactly and never was demo-felhom
|
||||
(`mp0 200G / mp1 50G`, where the same arithmetic gives ≈ 49 GB / ≈ 24 GB), nor the golden (`16 G / 8 G`
|
||||
before provision grew them). **Read it as a function of `mp1`, and only for a box still on the split
|
||||
layout.** Measured: `audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1.
|
||||
|
||||
**What replaced the partition's second job.** `mp1` was also a BULKHEAD: an overflow was refused per
|
||||
app with the last good unit byte-identical, and it **could not reach `/var/lib/docker`**, because that
|
||||
was a different filesystem. On a merged box it can. Decision **B2**, shipped in controller
|
||||
**v0.192.0**, is that bulkhead made deliberate — a two-term capture floor (97% used or 1 GiB free) in
|
||||
`internal/fillwatch`'s shape, sitting beyond its critical band so the customer is always warned first.
|
||||
It **refuses per app and never deletes**: nothing on this filesystem is generational, so pruning could
|
||||
only destroy a different app's only local copy.
|
||||
|
||||
**Status caveat, deliberately explicit:** as of 2026-08-03 **no box has been reinstalled from the
|
||||
merged golden** (R-178), so every box in the field is still on the split layout and everything above
|
||||
still describes them exactly. This subsection describes what a box built from golden ≥ 0.192.0 gets.
|
||||
|
||||
Two things are deliberately **not** recorded here. **The sizing ratio is the operator's ruling**
|
||||
(**R-163**) — this section states the constraint, not a number. And **the same-device placement is
|
||||
|
||||
@@ -90,8 +90,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **R-171** | **The boot sweep started apps whose data drive was ABSENT — a regression introduced by v0.189.0, now FIXED.** Replacing `isBootOrphan`'s container-count term with recorded intent made a drive-gate-stopped app (`compose down` ⇒ zero containers, and the gate never touches `desired_state` because it is not the customer) read as a boot orphan | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.190.0, 2026-08-02) | — | **Reasoned from the diff, then CONFIRMED on hardware before any fix was written** (`audits/DIAG-bootrecon-drive-absent-2026-08-02.md`). The sweep found and started calibre-web with its drive unmounted, burned both attempts and handed it to the dead-app alarm — **a false alarm about an app the drive gate is deliberately holding**. The *write* hazard did NOT materialise: compose failed `mkdir …/userdata: permission denied` because the unbound mountpoint is host-root-owned and the guest is unprivileged — **an accidental protection no code owns, no test pins, and one `chown` or one privileged guest away from gone**. Fix: new consumer-side seam `bootrecon.StartGate`, **fail-safe (cannot determine ⇒ do not start)**, wired in `main.go`; `Manager.DriveLive` reuses the userdata belt's own `isMountPoint` seam so the two cannot drift. **The rule is not new** — the API's `startGatedByMissingDrive` already refused this to the customer; the sweep bypassed it by calling `Manager.StartStack` directly. Widening the window (R-157 A) made two more holders reachable, so the same seam also refuses an app held by a **quiesce** or an **in-flight app-data operation** (§8.2), reusing `quiesce.SuppressedStacks()` and a new read-only `AppStopGuard.HeldStacks()`. Held apps report as `HeldByDrive`, never `StillDown` — that is the alarm's bucket. **ID established free:** `grep -ro "R-171\b" documentation/ *.md` → 0 hits before minting | — |
|
||||
| **R-172** | ~~**A false `host_stale` alarm fires when the hub's SQLite refuses two consecutive host reports.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (hub v0.88.0, 2026-08-02) | — | **ROOT CAUSE WAS NOT TUNING — THE PRAGMAS WERE NEVER APPLIED.** `store.New` used `?_journal_mode=WAL&_busy_timeout=5000`, which is **mattn/go-sqlite3** syntax; the driver is **modernc.org/sqlite**, whose `applyQueryParams` reads only `_pragma`/`_time_format`/`_time_integer_format`/`_txlock`/`_inttotime` and **ignores the rest without an error**. The hub ran in rollback-journal mode with `busy_timeout=0` for its entire life while its own source said WAL — a configuration asserting an invariant the code did not provide. Proof: a 128 MB open `/data/hub.db` with **no `-wal`/`-shm` beside it**. **Fix:** `?_pragma=journal_mode(WAL)&_pragma=busy_timeout(5000)&_txlock=immediate`. **`_txlock=immediate` is not optional** — `database/sql`'s `Begin()` is DEFERRED, so a read-then-write tx must upgrade its lock and a failed upgrade is `SQLITE_BUSY_SNAPSHOT`, which **`busy_timeout` does not retry**; this store has 10+ `db.Begin()` sites, all write paths. **Retry options (b) and (c) were deliberately NOT taken** — with readers no longer blocking writers a surviving `SQLITE_BUSY` would be a real signal, and a retry would hide it; revisit only on evidence. **Live:** `-wal`+`-shm` now present, **zero `SQLITE_BUSY` since rollout**, host back to `ok`, and `PRAGMA integrity_check` = `ok` with `journal_mode=wal` after three unrelated OOM restarts. **Operational consequence handled:** a WAL DB cannot be copied by taking `hub.db` alone — the break-glass retrieval in `operations/nodes.md` did exactly that and is now WAL-aware (the live `-wal` was 729 KB, i.e. a bare `cat` would have silently omitted it) | — |
|
||||
| **R-174** | ~~**The app-stop guard's crash recovery started apps onto MISSING drives — a regression in v0.189.0 code.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.191.0, 2026-08-02) | — | **Found by REVIEW on 2026-08-02, in code shipped 2026-08-01, and closed the same session — R-171 one path over.** `appStopGuard.SetStarter(stackMgr)` handed `Recover` the RAW stack manager, whose `StartStack` has no drive gate, and `Recover` runs **at startup** — exactly when an external drive may not have come back. So: a backup stops an app, the box loses power, the drive does not remount, and the app is started on a missing drive. The rule was not new — the API's own `startGatedByMissingDrive` already refused this to the customer; the guard bypassed it. **`bootDriveGate` could NOT be reused whole**, and the reason is recorded in the code: its holder #2 reads `bootAppStopGuard.HeldStacks()`, which during `Recover` is **the guard's own marker** — it would refuse every recovery it was meant to perform — and holders #1/#2 read package-level vars assigned AFTER `Recover()` runs, so a whole-gate reuse would be correct only by accident of nil-safety. Holder #3 is extracted into a shared `driveStartGate` with **two callers, one implementation**, and `TestBootDriveGateAndAppStopShareTheDrivePredicate` pins the delegation. **A REFUSAL IS NOT A FAILURE:** new `ErrStartRefused` + a `Refused` bucket — both keep the marker, only `Failed` alarms, because routing a deliberate hold into `NotifyBackupFailed` (customer-enabled by default) is the very R-171 false alarm this fixes. `main.go` guards on `Alarming()`, not `!= nil`, and the pre-existing seam test was TIGHTENED to require it. **Live on 9201, both directions:** drive held unmounted → `refusing to restart "calibre-web" … drive /mnt/felhom-drives/hdd_1 is not a live mountpoint`, marker retained byte-identical, zero containers started, `not alarming`; drive returned → `restarted calibre-web`, marker CLEARED. **ID established free:** `grep -ro "R-174\b" documentation/ *.md` → 0 hits | — |
|
||||
| **R-175** | **`07-backup-architecture.md` §7.5 states ONE box's size bound as if it were the fleet's.** Its headline — *"an app can be restored by Lane 1 only while its recovery unit still fits the retained space — ≈ 19 GB of app data for a file-only app, ≈ 10 GB for a DB-backed one"* — is derived from `mp1 = 20 G`. **That is demo-hp exactly, and it is not demo-felhom**, which ships `mp0 200G / mp1 50G`, where the real bound is ≈ 49 GB / ≈ 24 GB. The golden's own defaults are a third pair (`GOLDEN_DOCKER_GB=16` / `GOLDEN_SYSDATA_GB=8`), grown at provision time | **READY (S) — NEW 2026-08-02** | — | **Measured, not inferred** (`audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1: `pct config 9201` on both hosts). Independent of the merge — the sentence is wrong today and will be wrong differently after R-165. **The fix is to state the bound as a FUNCTION of `mp1`, not a constant**, and to say which box any quoted figure came from. Same class as the comment-asserting-an-invariant rule: a doc stating a fleet-wide number that only one machine satisfies reads as settled and is not. **ID established free:** `grep -ro "R-175\b" documentation/ *.md` → 0 hits | CC |
|
||||
| **R-176** | **Two prerequisites for the R-165 merge are UNMEASURED, and both are cheap.** (a) Whether a **pre-merge archive** (carrying `mp1`) restore-tests cleanly into a **merged-layout** guest — reading `mountParity` (`felhom-agent/internal/reconcile/restoretest.go:347`) says it should, because the restore recreates `mp1` from the archive so archive and restored guest agree; **that was reasoned from source and never executed.** (b) The in-place per-box migration (move `<mp1>/felhom-data` onto `mp0`, drop the slot, verify) has **never been rehearsed even once**, so "is the box restorable at every point of it?" is currently unknown | **READY (S) — NEW 2026-08-02** | blocks R-165 landing safely | **Filed because this project's own record is that FOUR production designs specced against unvalidated mechanisms were all wrong** — which is exactly why R-165's own spike refused to design. Both are one command on a **Tier-0** box (D-d: both demo boxes are disposable). (b) is only required work if Peti's box turns out to need migrating rather than reinstalling — the hub cannot answer that (M5: `peti-felhom` exists as a customer with **no host in the register**), so it is the operator's input. **ID established free:** `grep -ro "R-176\b" documentation/ *.md` → 0 hits | CC |
|
||||
| **R-175** | ~~**`07-backup-architecture.md` §7.5 states ONE box's size bound as if it were the fleet's.**~~ | **CLOSED — FIXED 2026-08-03** (same pass as R-165) | — | **Measured, not inferred** (`audits/SPIKE-r165-mp1-merge-2026-08-02.md` M1: `pct config 9201` on both hosts). Independent of the merge — the sentence is wrong today and will be wrong differently after R-165. **The fix is to state the bound as a FUNCTION of `mp1`, not a constant**, and to say which box any quoted figure came from. Same class as the comment-asserting-an-invariant rule: a doc stating a fleet-wide number that only one machine satisfies reads as settled and is not. **ID established free:** `grep -ro "R-175\b" documentation/ *.md` → 0 hits **FIXED.** §7.5 gained a **7.5.1** which (a) states plainly that the bound is a FUNCTION of `mp1` and applies only to a box still on the split layout, naming all three real shapes, and (b) records that the ceiling itself has been removed by R-165 for boxes built from golden ≥ 0.192.0. Fixed in the same pass as the merge rather than filed and forgotten, because the section would otherwise have been wrong in two ways at once | CC |
|
||||
| **R-176** | **Two prerequisites for the R-165 merge are UNMEASURED, and both are cheap.** (a) Whether a **pre-merge archive** (carrying `mp1`) restore-tests cleanly into a **merged-layout** guest — reading `mountParity` (`felhom-agent/internal/reconcile/restoretest.go:347`) says it should, because the restore recreates `mp1` from the archive so archive and restored guest agree; **that was reasoned from source and never executed.** (b) The in-place per-box migration (move `<mp1>/felhom-data` onto `mp0`, drop the slot, verify) has **never been rehearsed even once**, so "is the box restorable at every point of it?" is currently unknown | **(a) ANSWERED 2026-08-03 (P1: PASS). (b) NOT REQUIRED — operator ruling: every node is reinstalled, none migrated** | blocks R-165 landing safely | **Filed because this project's own record is that FOUR production designs specced against unvalidated mechanisms were all wrong** — which is exactly why R-165's own spike refused to design. Both are one command on a **Tier-0** box (D-d: both demo boxes are disposable). (b) is only required work if Peti's box turns out to need migrating rather than reinstalling — the hub cannot answer that (M5: `peti-felhom` exists as a customer with **no host in the register**), so it is the operator's input. **ID established free:** `grep -ro "R-176\b" documentation/ *.md` → 0 hits **UPDATE 2026-08-03.** **(a) is measured and passed** — `audits/SPIKE-r165-phase0-2026-08-03.md` P1: a real pre-merge archive (`mp0+mp1`, confirmed from its own vzdump log) restore-tested on demo-hp, `pass: true`, `mount_parity: ok`, 84 s, with `mountParity` untouched. One limit stated rather than glossed: it ran with the pre-merge agent because the merged one did not exist yet, and the comparison is archive-vs-its-own-restore which never consults the host layout — **re-run it once against agent v0.120.0**, which is one command. **(b) is withdrawn, not deferred:** the operator ruled that every node is REINSTALLED rather than migrated in place (both demo boxes are Tier 0; the colleague's box carries none of our customer data and is clean-installed in a few weeks), so the in-place migration rehearsal has no consumer. Recorded explicitly rather than silently skipped | CC |
|
||||
| **R-178** | **The merged golden (0.192.0) is built and published but NO BOX HAS BEEN REINSTALLED FROM IT, and it is deliberately UNVOUCHED.** `build-golden.sh` v3.0.0 baked it with variant V-c and every retargeted assertion passed on the real bake (`including mount point mp0 ('/var/lib/felhom')`, no `mp1` line, `both paths are ONE filesystem`); it is in the registry (HTTP 200, sha `54e2a4c431daf580…`). What has NOT happened is Part 4: reinstall each demo box from it and prove claim → deploy an app → back up → restore | **READY (M) — NEW 2026-08-03** | blocks R-165 reaching PROVEN-LIVE; blocks the capability-map row | **The golden is UNVOUCHED ON PURPOSE and that is the safe state**, not an oversight: vouching is what makes a fresh install pick it up, so vouching a golden no box has been proven from would put an unproven disk layout in front of the next install anywhere. **Prove first, then vouch** — the bake script's own output treats the hub record as a separate deliberate step for this reason. **Everything else for the merge is shipped and green:** controller v0.192.0 (the B2 floor) is live on 9201, agent v0.120.0 is live on BOTH hosts, and `felhom-host-install.sh` computes the single grow from the thin pool. So a reinstall is now a self-contained piece of work with no code left to write. **Order matters: ONE box at a time**, demo-hp first, proven end to end, and only then demo-felhom — two in parallel leaves no working reference to compare against. Note demo-felhom carries the PBS-DR/offsite tier, so it is the one whose backup chain a reinstall actually disturbs. **ID established free:** `grep -ro "R-178\b" documentation/ *.md` → 0 hits | CC |
|
||||
| **R-177** | **There is no operator-triggerable "run the fill check now" path.** `fill-watch` is reachable only on its daily 03:30 schedule plus the once-at-startup run added in controller v0.191.1 — so the only way to exercise it on demand is to restart the controller | **READY (S) — NEW 2026-08-02** | — | **Noticed while live-validating R-167 on 9201, not by a failure.** It cost a controller restart per observation during validation, and it costs the same on a support call: after a customer frees space, nobody can confirm the warning has cleared without restarting their controller or waiting until 03:30. **Partially mitigated already** — v0.191.2 makes every run log a positive observable (`checked N filesystem(s), M unreadable/skipped, K notification(s)`), so at least a run that DID happen is visible; the gap is triggering one. The scheduler has `GetJobs` but no run-now, so this is a general affordance, not a fill-watch one — **scope it as "run a named scheduler job now", operator-gated.** **ID established free:** `grep -ro "R-177\b" documentation/ *.md` → 0 hits | CC |
|
||||
| **R-173** | **The hub's SQLite PVC is excluded from every Longhorn backup job.** `pvc/hub-data` carries `recurring-job-group.longhorn.io/default: disabled`, and `backup-daily` + `backup-weekly` (04:00 / Sun 05:00) are the ONLY recurring jobs and both target the `default` group — so the 128 MB `/data/hub.db` has **no volume-level backup**. That database holds `host_recovery` (every managed box's break-glass root password), `host_escrow` + `host_escrow_superseded` (escrow custody), `host_pbs_secrets`, `customer_configs`, `dr_recipe` and the wg endpoints/peers — i.e. the material several documented recovery routes depend on | **READY (M) — NEW 2026-08-02** | — | **Noticed while checking the blast radius of the R-172 WAL change, not by a failure** — the WAL work needed to know who copies this file, and the answer turned out to be nobody on a schedule. **Establish before designing:** (a) whether the exclusion is deliberate (a 1 Gi RWO Longhorn volume snapshotting a 128 MB SQLite file is cheap, so the label looks like a leftover rather than a decision) and by whom; (b) whether anything else backs it up out-of-band that this census missed — the `_recovery-inventory-2026-07-28.md` records a MANUAL hot copy, which is not a backup. **When it is designed, it must be WAL-aware** (R-172): a volume snapshot of a live WAL database is crash-consistent and replays on open, which is fine, but any file-level copy must take `hub.db-wal` too or it silently loses the newest writes. **Grep establishing the ID was free:** `grep -ro "R-173\b" documentation/ *.md` → 0 hits | CC |
|
||||
| **R-158** | ~~**A local Tier-1 app-data backup failure reaches no hub channel.**~~ | **CLOSED BY R-167 — SHIPPED + PROVEN-LIVE** (controller v0.191.0 + hub v0.89.0, 2026-08-02) | — | **Closed by the wire it named; no second row was filed for it** (R-167 subsumes and widens it). New `unitNotify` seam + `SetUnitNotify` beside the manager's existing three, called from `captureAllRecoveryUnits` **per app with the loop continuing**, carrying the target filesystem's used/free bytes at the moment of failure — the cause is usually a full filesystem and those numbers answer *why* without an operator logging in. **ROUTED TO THE OPERATOR, NOT `backup_failed`, AND THAT OVERRIDES THIS ROW'S OWN PROPOSAL.** The proposal above said *"emitting the existing `backup_failed`"*; that type carries a `customerMessages` entry AND sits in `settings.DefaultEnabledEvents`, so it would email the customer in Hungarian about a failure they cannot act on — precisely the mistake R-97a avoided by minting `whole_guest_backup_failed`. **Decision D-c routes it to the operator and D-c wins.** New `recovery_unit_capture_failed` in `allowedEventTypes` **and** `notify.operatorOnlyEvents`; `notify.IsOperatorOnly` added so ONE test pins both registers (allowlisted-but-not-operator-only is invisible when they are checked separately — the v0.78.0 defect). **Red-proof:** removing the register entry shows the customer being emailed. **Live on 9201:** two events accepted and stored, `operator | sent`, and the positive observable `customer | recovery_unit_capture_failed | skipped | operator_only` read from the hub's `notification_log` | — |
|
||||
@@ -99,9 +100,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **R-160** | **gramps-web persisted three paths and wrote to none of them.** `/app/data` appears nowhere in the image's environment; the accounts DB (`GRAMPSWEB_USER_DB_URI`) and **the family tree** (`GRAMPS_DATABASE_PATH=/root/.gramps/grampsdb`) both landed in the writable layer. Upstream persists **eight** paths; the template persisted three, one a phantom. | **SHIPPED** (`templates/gramps-web/docker-compose.yml`, 2026-08-02) | — | **Severity above papra's, and worth keeping visible:** papra loses documents the customer may hold elsewhere; gramps-web loses **the family tree — the artefact built inside the app, of which no other copy exists by construction.** Evidence: `app-catalog-felhom.eu/audits/persistence-sweep-2026-08-02/` | CC |
|
||||
| **R-161** | **The volume-persistence gate is enforced by CONVENTION, not automatically.** The catalog repo has no CI of any kind (`.gitea/workflows`, `.github`, drone/woodpecker — searched, none exists). | **REDUCED SCOPE — open** (operator ruling 2026-08-02) | a second person touching templates | **RULED. Both obvious enforcement points were rejected for measured reasons.** *Controller-side at template load:* rejected because such a check can only read the file, and a static audit of all 53 templates reports the catalog clean **including papra** — **it would pass on the exact defect it exists to catch**; the property is decidable only at runtime. *CI:* rejected for now — neither repo has any, and there are no users yet. **SHIPPED instead** (`app-catalog-felhom.eu` `fd7747d`): `scripts/catalog_gates.py`, ONE entry point running all three gates, non-zero exit on any failure, **mandated in the catalog's `CLAUDE.md`** the way `site_gates.py` is. Rationale for the record: of this project's gates, the only ones that ever get run are those with a single entry point named in a CLAUDE.md — `site_gates.py` is run, R-29's three orphans are named nowhere and have stopped nothing. **What remains open is only the automatic half:** this is convention, run by a person, and that is sufficient while one person touches templates. Revisit when a second does **UPDATE 2026-08-02:** `catalog_gates.py` gained `--fast` (gate 1 only — the network and runtime gates are deliberately NOT in a hook: a push that pulls images and starts containers gets bypassed within a week and the bypass becomes the habit) and `.githooks/pre-push` now runs it. **The automatic half now has a designated successor row: R-168** (Gitea Actions runner). This row stays open at its reduced scope — the runtime gate remains a deliberate periodic run **UPDATE 2026-08-02 (second):** the automatic half now EXISTS — R-168's runner executes `catalog_gates.py --fast` on every push to this repo (measured: run #1, `image-pin gate OK — 53 templates`, with the two runtime gates announced as skipped and their own output absent from the log). This row's *original* scope — the RUNTIME volume-persistence gate — is deliberately still NOT automatic and should stay that way: CI that pulls 53 images on every push gets disabled. It remains a periodic run | operator |
|
||||
| **R-162** | **`docker diff` is the gate's only witness, and its failure mode is quiet.** The gate's power comes from `docker diff` excluding mounted paths, which makes "in the writable layer" mechanically decidable — an implementation detail of the overlay driver. On a driver where `docker diff` is unsupported or lies, the gate degrades to the mount-occupancy and writability legs **and would not say so**. | **WATCHING** — a limitation, not a defect | — | It **fails closed**: the canary self-test would stop reporting BROKEN and the gate would then refuse to report at all. What is wrong is the message — it would blame the prober rather than the driver. Revisit only if a non-overlay storage driver ever ships | CC |
|
||||
| **R-163** | **`mp1` is RETENTION, not staging — and it is sized as if it were neither.** A recovery unit is the KEPT copy on the app's **own** drive (`GetAppDrivePath`, `internal/backup/backup.go:245-255`); for an app with no `HDD_PATH` the namespace falls back to the system SSD — *"the SSD-only system-data fallback"* (`internal/appbackup/paths.go:26-27`). There is **no post-copy deletion**: the only prune is F5 (`backup.go:1053-1112`), residue on OLD drives when an app MOVES. So `mp1` (**20 G**) retains the units of every driveless app, while `mp0` permits **50 G** of volumes — and a DB app's unit is up to **~2×** its data (volume tar **plus** SQL dump; measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to **0** (`felhom-agent/cmd/felhom-agent/main.go:178`) and is **not** derived from the physical drive; demo-hp's real guest 9201 ships `mp0 50G / mp1 20G`. | **RE-FRAMED 2026-08-02 — open, no longer waiting on a ratio** | — (the sizing question is answered; the work is **R-165**) | **RE-FRAMED, NOT CLOSED (operator decision D-a, 2026-08-02 — `CONTEXT.md` S-5).** The row asked *what ratio should `mp1` be?* and that question is **withdrawn rather than answered**: `mp1` is merged into `mp0` so local recovery units share the app-data area and the ceiling stops existing — a bigger number is the same wall further away. **This row stays open as the record of the constraint** (what `mp1` is for, what it gates, and the measured 2× DB-app unit size) **until the merge lands**, because until then every consequence below is still live on every box. **The work is R-165; the warning that must ship with it is R-167.** Original finding, unchanged, follows. **No number is proposed here deliberately.** What is recorded is the constraint and its blast radius: **`mp1` gates the whole app-data chain**, because Tier-2 mirrors the unit *"(always)"* from `RecoveryUnitPath` (`internal/backup/tier2.go:302,368`) and Tier-3 carries it too — a unit that cannot be written has nothing for either to copy. Bounded on the other side: a unit holds **volume tars + DB dumps only, never `mp8` userdata** (`internal/backup/recovery_unit.go:20-25`), so a 1 TB photo library is never in one. **This bounds D5's Lane-1 independence** — see `architecture/07-backup-architecture.md` §7.5. Overflow itself is SAFE (R-158's measurement: refuses per app, last good unit preserved byte-identical) — what is missing is the warning, which is R-158 (widened to R-167) | CC |
|
||||
| **R-163** | ~~**`mp1` is RETENTION, not staging — and it is sized as if it were neither.**~~ | **CLOSED by R-165 — the ceiling it describes no longer exists** (golden v3.0.0, 2026-08-03) | — (the sizing question is answered; the work is **R-165**) | **Closed, not merely re-framed.** This row was the record of a constraint that was to stay open *"until the merge lands"*. It has landed: the golden ships ONE data volume, so there is no separate 20 G area for a driveless app's recovery unit to outgrow, and the free space an app can use is the box's actual free space. **What replaced the constraint is recorded on R-165**: the bulkhead the partition also provided is now B2's explicit capture floor (controller v0.192.0), and the measured 2× DB-app unit size this row documented is what justifies the floor's reserve being a reserve rather than a working budget. **Caveat carried forward, deliberately:** no box has been reinstalled from the merged golden yet (R-178), so every box in the field still has the split layout and this row's consequences remain live ON THOSE BOXES until they are reinstalled. Original finding unchanged below | CC |
|
||||
| **R-164** | **C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists.** The unit carries both a volume tar and a SQL dump; the restore uses **both** — the dump is authoritative and replayed *after* the tar so it WINS (F17), with only the DB service up (R-47) — `internal/backup/restore_unit.go:262-266`. Dropping the DB container's tar would halve DB-app units **and** close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ `POSTGRES_PASSWORD` ignored). | **BLOCKED** — on the predicate | a dump-validity predicate that is not `accounts has rows` | **The obvious gate is DEAD, measured:** `ValidateDump` warns when the `accounts` table is empty, and that warning was **correct** — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But **a fresh appliance legitimately has zero accounts**, so promoting that predicate to a gate would **block every new customer's first backup**. Order: (1) a sound predicate — dump vs **live** per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. **Until (1), the tar is load-bearing** — not because dumps are bad, but because nothing can yet prove one is good. Pairs with **R-127** | CC |
|
||||
| **R-165** | **Merge `mp1` into `mp0` — the dedicated 20 G backup partition stops existing.** Operator decision **D-a**, 2026-08-02 (`CONTEXT.md` S-5). Local recovery units share the app-data area instead of holding their own fixed ceiling, so the wall R-163 describes is removed rather than moved further away. Guest 9201 on demo-hp ships `mp0 50G / mp1 20G` today | **READY (M) — NEW 2026-08-02** | — | **Two conditions travel WITH the decision and are not optional.** **(1) Before any external install.** It changes the **disk layout**, so it is a fresh-install shape while there are no external boxes and a per-box migration after — and the decision's cheapness is entirely a function of that ordering. **(2) It removes a wall that currently fails safely**, so **R-167** (D-c: fill warning + failure alert) lands in the same step, never after: today an app that outgrows `mp1` is refused per app with the last good unit preserved byte-identical (R-158's measurement), and after the merge the same overflow consumes the space the app itself is using. Touches the installer/agent guest shape (`--sysdata-grow` defaults to **0** and is not derived from the physical drive, `felhom-agent/cmd/felhom-agent/main.go:178`) and the golden. **Does NOT close R-163** — that row is the record of the constraint and stays open until this lands **MEASURED 2026-08-02 — `audits/SPIKE-r165-mp1-merge-2026-08-02.md`; no layout was touched.** **M1: "the layout" is not one thing** — demo-felhom ships `mp0 200G / mp1 50G`, demo-hp `50G / 20G`, the golden `16G / 8G`; any plan expressed as a fixed pair is already wrong for one of the two boxes (this also invalidates §7.5's fleet-wide bound → **R-175**). **M2: the majority of `mp1` is NOT Tier-1** — on demo-felhom 1.7 G of 2.0 G is Tier-2 mirrors, plus the `userdata/import` drop zone which is on the system drive by CONTRACT (R-75); a plan accounting only for the units is wrong. Observed fill is 5% / 1% — the constraint is a ceiling problem, not a current-fill one. **M3: three assertions break, and the golden fails closed on the split in FOUR places** (`build-golden.sh:126,130` separate-mount + `:315,319` vzdump-exclusion guards), not one; the archive scope `rootfs+mp0+mp1` stays complete after the merge; `mountParity` holds for new archives. **M4 — THE BULKHEAD IS THE REAL COST:** today an overflow is refused per app with the last good unit byte-identical AND CANNOT REACH `/var/lib/docker`; after the merge it can, and a full Docker data-root is a stopped box, not a degraded one. Four candidate replacements costed (reserve / capture-path refusal / project quota / warnings-only); **not chosen — operator's ruling.** **M5: D-a's condition (1) is currently SATISFIED** — no external box is in the hub's host register (only the two ONLINE demo boxes, both **Tier 0 and reinstallable**, so migration cost for the measurable population is ZERO). **`peti-felhom` exists as a customer with NO host in the register, so its layout is UNMEASURED** and was not contacted (D-d). **Recommendation: S1 (one volume, two directories) + B2 (a refusal threshold in the capture path), as a fresh-install shape with the demo boxes REINSTALLED.** **Two prerequisites are unmeasured → R-176.** **STOPPED for the operator's ruling; the merge is a supervised session.** | CC |
|
||||
| **R-165** | ~~**Merge `mp1` into `mp0` — the dedicated 20 G backup partition stops existing.**~~ | **SHIPPED — golden `build-golden.sh` v3.0.0 + agent v0.120.0 + controller v0.192.0 (B2), 2026-08-03. NOT YET PROVEN-LIVE: no box has been reinstalled from the merged golden** | — | **Variant V-c chosen by the operator on MEASURED evidence, not by reading** (`audits/SPIKE-r165-phase0-2026-08-03.md`): one volume at the NEUTRAL path `/var/lib/felhom`, with `/var/lib/docker` and `/mnt/sys_drive` both binds of subdirectories. Three shapes were built and rebooted; **all three boot, reboot 3/3, give ONE `df` figure and keep a container's `statfs("/")` on the merged volume — the ordering worry that motivated the probe did not materialise.** They differ only in which documented guarantee they break: volume-at-`/var/lib/docker` puts customer backups INSIDE Docker's data-root (so the ordinary "clear /var/lib/docker" reflex destroys every local unit); volume-at-`/mnt/sys_drive` puts Docker's ENTIRE data-root under `/mnt`, which the controller container mounts wholesale — **measured: it then sees `/mnt/sys_drive/docker`**, falsifying the bootstrap's own scoping claim. V-c breaks neither. **P1 answered R-176(a):** a pre-merge archive (`mp0+mp1`) restore-tests clean with `mount_parity: ok` in 84 s; `mountParity` was not weakened. **P3: the four golden assertions were RETARGETED, never deleted, and each was RUN against a deliberately wrong shape — 8 checks, 8 passed**, including a NEW 2b asserting both paths are ONE filesystem (which catches the S2 shape the spike ranked worse than the split) and a new guard for a leftover `mp1` (the old "was mp1 excluded?" pattern could no longer match — a guard that cannot match has silently stopped guarding). **B2 shipped first, in controller v0.192.0**: a two-term capture floor (97% / 1 GiB) in `fillwatch`'s shape, deliberately beyond its critical band so the customer is always warned before a refusal; it refuses per app and **never deletes**, because nothing here is generational. **Golden 0.192.0 is published (registry HTTP 200, sha `54e2a4c4…`) but DELIBERATELY NOT VOUCHED** — vouching is what makes fresh installs pick it up, and the right order is prove-then-vouch. **Remaining: reinstall both demo boxes from it, prove end to end, then vouch → the work is R-178** | CC |
|
||||
| **R-166** | ~~**App state gets a desired/observed model with its own store.**~~ Operator decision **D-b**, 2026-08-02 (`CONTEXT.md` S-5) | **SHIPPED + PROVEN-LIVE** (controller v0.189.0, 2026-08-02) | — | **Both blocking facts were established at source before any code was written, and the answers changed the shape.** **(a) Does a crash-safe journal already exist for the in-flight case?** YES, twice — `internal/quiesce/quiesce.go` (marker + `Recover`, proven on live hardware by Campaign 8 fault 10) and `internal/stacks/migrate.go` (`migration.json` + `RecoverMigration`) — but **neither covers the app-data path**: `DumpAppVolumesSafe` stopped and restarted an app with **no marker, no journal and not even a `defer`**. So the pattern existed and the coverage did not; `backup.AppStopGuard` copies the proven shape into its **own** file (one file, one writer). **(b) Is the SQLite store reachable?** Irrelevant, and deliberately unused: `metrics.db` is optional by design (the controller runs with it absent), and operational state must not live in a store designed to be droppable. **Shipped:** tri-state `desired_state` in `app.yaml` written ONLY by the customer's action (API action switch, `DeployStack`, `UpdateOptionalConfig`'s redeploy branch, `.fab` import — a 14-caller census established that `StartStack`/`StopStack` must NOT be writers); `isBootOrphan` reads intent instead of `len(Containers) > 0`; **absent means UNKNOWN, never running**, so a legacy `app.yaml` keeps byte-identical pre-v0.189.0 behaviour; running-only backfill. D-b's every-container requirement was already met by `aggregateState` and was NOT re-implemented. **Live on 9201:** all three flows (stop survives a restart; a zero-container `running` app is recovered by name; a legacy app.yaml is skipped and never inferred as stopped). **Also fixed en route:** `SaveAppConfig` rebuilt `AppConfig` field-by-field (the R-100 shape) and would have dropped the new field on every save across nine call sites | — |
|
||||
| **R-167** | ~~**Storage monitoring and backup alerts.**~~ | **CLOSED — SHIPPED + PROVEN-LIVE** (controller v0.191.0/.1/.2 + hub v0.89.0, 2026-08-02) | — | **Operator decision D-c. It shipped BEFORE the R-165 merge, not with it** — D-a's condition (2) says the monitoring lands in the same step and never after, and landing it first is strictly better and costs nothing. **Customer half:** new `internal/fillwatch`, per FILESYSTEM (never per app — one full disk holding ten apps would fire ten times). **It emits the PRE-EXISTING `disk_warning`/`disk_critical` pair, which was allowlisted, copy'd, in `DefaultEnabledEvents` and checkbox'd with NO PRODUCER IN ANY REPO** — a complete customer pipeline with no producer, the **sixth** *built-but-never-wired* instance here; minting a new near-duplicate type would have left it inert forever. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range — **proven live: the critical crossing fired on the FREE-BYTE term (1.7 GB) at only 91% used.** Edge-triggered on escalation, state persisted, hysteresis dead zone at 75% / 7 GiB pinned by a test; a nil usage read never warns and never clears one (§8.4). The hub's two generic `customerMessages` entries were **removed** — `FormatCustomerEmail` prefers the entry over the message, so keeping them would discard the drive label and the byte figures. **Operator half: see R-158.** **Live on 9201, all three flows:** `disk_warning` then `disk_critical` both `customer | sent` with the Hungarian rendered, exactly two events across three boots (the edge trigger held on the one between), then a silent clear that re-armed. **v0.191.1** added the once-at-startup run (Daily/Every both wait for their first tick, so a box BOOTING over the line would have stayed silent up to 24 h — the R-100 shape); **v0.191.2** added a per-run positive observable, earned when a quiet run during this session's own validation proved unreadable as evidence. Follow-ups: **R-177** (no run-now path) | — |
|
||||
| **R-168** | ~~CI: no runner exists, and with trunk-based pushes CI can DETECT but not BLOCK~~ | **SHIPPED — and the alarm is DEMONSTRATED** (2026-08-02) | — | **Runner live**: `homelab-manifests/gitea-system/act-runner.yaml`, an unprivileged host-mode `act_runner` in `gitea-system`, one owner-scoped registration serving all four repos (measured: tasks 7-10 all claimed by `felhom-gates-runner`). `.gitea/workflows/gates.yml` in each repo runs that repo's entry point with `--fast` and nothing else; no `uses:` step anywhere. **Six probes, all answered, none STOPped** — `audits/SPIKE-ci-runner-2026-08-02.md`. The two that changed the design: **P2** (stock image has git but NO python3 → custom image `felhom-act-runner:0.1.0`, base pinned, python3 and nothing else) and **P6** (a runner that loses `/data/.runner` re-registers and leaves a dead record behind → the PVC is load-bearing, measured both ways). **P5 is the one that mattered**: a failed run produced NO mail, NO notification row and NO log line from Gitea, so the run now sends its own alarm via Resend and prints the provider's accepted id. **Proven end to end, not asserted**: a deliberately broken commit pushed with `--no-verify` → run #6 `failure` → `RESEND-ACCEPTED id=5ff34766-c5f8-4588-8104-08296aeb45ab`. Posture shown from the live pod spec: `privileged: false`, all caps dropped, no docker socket, no hostPath, `automountServiceAccountToken: false`, sized at half Gitea's limits so it cannot crowd out the service holding every repository on the same node. **The standing limit stays true and is written into the manifest and every workflow: it DETECTS, it does not BLOCK** — making it block is → R-169 | — |
|
||||
|
||||
Reference in New Issue
Block a user