docs: R-182 closed, R-90 closed on measurement, R-86 unblocked, ep0 record corrected
gates / gates (push) Successful in 7s
gates / gates (push) Successful in 7s
R-182 CLOSED (controller v0.194.0 + hub v0.90.0/.1), proven live on demo-hp. The hub's notification_log for the run reads: two per-app failures RECORDED, one digest SENT naming both, and the customer channel SKIPPED with operator_only. Against the measured previous behaviour — two failures, one email naming one app, one leaving no trace anywhere. Scenario D proved itself on an event I had not planned: disk_critical alarmed on two filesystems, the second was collapsed by the cooldown, and that collapse is now visible WITH ITS KEY. Yesterday it would have left nothing at all. A gap the spec did not anticipate is recorded with its fix: the per-app event also fires from the periodic sweep, outside any run, so making it record-only would have created a NEW silence. The sweep emits a digest too, with no run_id, so it stays under the ordinary hourly cooldown. ep0: MEASURED on the box — 7757 MB (8 GB), 4 vCPU, and the 4 GiB swapfile SURVIVED the resize and is active (checked, because a resize is a stop/start). The 40 GB local disk is UNCHANGED, so no disk figure was touched anywhere. Five documents corrected — three of which the task's list did not name, found by searching. Two audit/evidence documents ANNOTATED, body untouched: they record what was true when written and that is their value. R-90 CLOSED. R-86 unblocked and re-ranked, stated honestly: 8 GB is comfortable, not unbounded — the original OOM was a 14.46 GB restore — so the restore-test cadence should still be paced, just not by fear of the endpoint. target-selection.md's "D-d did not name ep0 either way" is deliberately left standing. It is the operator's question, not CC's. STATUS.md 127 -> 83 lines, items rather than sentences.
This commit is contained in:
1 parent
f21e7caed1
commit
e34b614e5b
13 files changed
+279
-279
No files matched your search
@@ -86,7 +86,7 @@
|
||||
| An app can be **withdrawn from the catalog without orphaning the customers running it** (available / hidden / abandoned) | controller v0.158.1, catalog metadata | **PROVEN-LIVE** (2026-07-21) | TASK-F Part 1. Verified on 9201 through the real endpoints: `lifecycle: abandoned` arrived via the normal catalog sync; plant-it renders 0 times on the Alkalmazások page (control app renders 10); a direct `POST /api/stacks/plant-it/deploy` → **HTTP 409 "Ez az alkalmazás jelenleg nem telepíthető."**; the app page carries the permanent notice and offers no Telepítés button. `felhom-controller/REPORT.md` (2026-07-21) | Deployed instances keep FULL function in every state — lifecycle governs what is offered, never what runs. Orphan detection deliberately never sees the field (red-proofed): a withdrawn template stays in the catalog tree, or every deployed instance would read `Elavult` and be offered deletion. Unknown values fail OPEN; the deploy gate fails CLOSED. R-57 |
|
||||
| Box survives a **site/network change** (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE (2026-07-25)** | **R-50 SHIPPED and deployed to the whole fleet.** The control plane now rides a host-internal, portless island bridge (`vmbr9`, `169.254.253.1/30`↔`.2/30`) with a fixed private address that no LAN/DHCP/site move can invalidate. Proven end-to-end: the spike's F1 replay (renumber the LAN → agent stays bound on the island, control plane HTTP 200; the LAN-literal contrast reproduces the original `bind: cannot assign requested address` daemon-death) + cold-reboot survival (`SPIKE-island-bridge-2026-07-25.md`), the migration runbook run verbatim (`RUNBOOK-island-migration.md`), a fresh provision auto-attaching the island `net1` (A4), and the live migration of **both demo boxes** (demo-hp + demo-felhom, 2026-07-25) — island `/storage` HTTP 200, LAN DNS pinned to the LAN IP (Finding-1), **apps served throughout (0 container restarts)**, hub reporting 0.96.0. **Origin:** `audits/AUDIT-vacation-remote-ops-2026-07-20.md` — the real relocation where the agent's LAN-literal bind took storage/PBS/quiesce/restore-test/DR down silently; that is now structurally impossible on a migrated box | **Fleet: DONE.** Remaining: **R-74** — bring the island to Peti's 2-node cluster (SDN vnet / bridge parity), its own supervised runbook. Related historical: R-51 (dead-primary alerting), R-52 (boot desired-state reconciliation), both shipped |
|
||||
| **The customer is warned BEFORE a filesystem fills** — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered | controller **v0.191.0/.1/.2**, hub **v0.89.0** (R-167, decision D-c) | **PROVEN-LIVE (2026-08-02)** | `audits/SPIKE-r165-mp1-merge-2026-08-02.md` (context) + `felhom-controller/REPORT.md`. Exercised on guest 9201 against a REAL filesystem (`/mnt/sys_drive` filled with `fallocate`): **`disk_warning` at 90% used / 4.7 GB free** → hub `notification_log` `customer | disk_warning | sent` with the dynamic Hungarian rendered; grown to 1.7 GB free → **`disk_critical`** → `customer | sent`; file removed → `critical → ok … cleared silently, re-armed` and the persisted state emptied. **Exactly two events across three boots** — the boot in between produced none, which is the edge trigger holding | **Nothing warned before this.** The only prior signal was the healthcheck's generic `health_degraded` at 90%, for REGISTERED STORAGE PATHS ONLY — it never looked at the docker area or the system-data area, never gave a free-byte figure and never named a drive. **The two event types already existed with NO PRODUCER** (`disk_warning`/`disk_critical`: allowlisted, copy'd, in `DefaultEnabledEvents`, checkbox'd) — the **sixth** *built-but-never-wired* instance here; this ships their producer rather than a seventh near-duplicate type. **Two threshold terms, whichever trips first, and the live proof vindicated the design:** the critical crossing fired on the FREE-BYTE term (1.7 GB) at only **91%** used — a percentage-only rule would have missed it. The hub's generic `customerMessages` entries were REMOVED, because `FormatCustomerEmail` prefers the entry over the message and would discard the label and figures. **Known gap → R-177:** there is no operator-triggerable run-now path; the check is daily 03:30 + once at startup, so confirming a cleared warning on a support call needs a controller restart or a wait |
|
||||
| **A failed per-app Tier-1 backup reaches the OPERATOR** (app, error, and the target filesystem's used/free bytes at the moment of failure) | controller **v0.191.0**, hub **v0.89.0** (R-158, closed by R-167) | **PROVEN-LIVE (2026-08-02)** | `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing | **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it |
|
||||
| **A failed per-app Tier-1 backup reaches the OPERATOR — EVERY failing app, in ONE mail per run, and every failure recorded whether or not it is mailed** | controller **v0.194.0**, hub **v0.90.1** (R-158 → R-167 → **R-182**) | **PROVEN-LIVE (2026-08-03)** | `felhom-controller/REPORT.md`. Two real capture failures on guest 9201 (`mkdir …/backups: permission denied`) → both accepted and stored by the hub, `operator | recovery_unit_capture_failed | sent`, and the positive observable **`customer | recovery_unit_capture_failed | skipped | operator_only`** read from the hub's `notification_log`. One event per app, loop continuing | **Before this the failure was a `[WARN]` line and nothing else** — the manager carried three notify seams and none for the unit capture, so `/backups/apps`, the page you open to ask whether ONE app is backed up, was the one page that never said. **Deliberately NOT `backup_failed`:** that type is customer-enabled by default and carries Hungarian copy, so reusing it — which R-158's own proposal said — would email the customer about a failure they cannot act on. **D-c routes it to the operator and overrides the proposal.** Operator-only is enforced by `notify.operatorOnlyEvents`, NOT by the absence of a `customerMessages` entry (the v0.78.0 defect); a red-proof removing the register entry shows the customer receiving it. **ROW REWRITTEN 2026-08-03 (R-182) — the 2026-08-02 claim was TRUE OF ONE APP AND FALSE OF THE REST, and it is worth saying which.** The signal existed and worked; what it did not do was scale past the first failing app. Measured: nine per-app events reached the hub in one day and **two** operator mails went out, because the cooldown key is `customerID:eventType(+tier)` and this type carries `app` but no `tier` — so the first refused app took the hour and the rest were dropped **before `LogNotification`**, leaving no row on any channel. The old row said "One event per app, loop continuing", which was true of what the CONTROLLER emitted and not of what the operator received. **Now:** the per-app event is the RECORD (hub `recordOnlyEvents`: stored + logged every time, never mailed) and `backup_run_failures` is the NOTIFICATION — one mail per run listing every failed app, its leg and its reason, with the counts and free space. A suppressed operator event of ANY type now leaves a `suppressed` row naming its key. **Proven live on demo-hp 2026-08-03** by a real 64.6 GiB fill (thin pool held 30.78 → 30.78): `notification_log` shows `recovery_unit_capture_failed | operator | recorded` ×2, `backup_run_failures | operator | sent` ×1 naming BOTH apps, and `backup_run_failures | customer | skipped | operator_only`. A second run in the same hour produced a second digest; after freeing space a run completed with 2 volume dumps and **no** digest. The suppression row proved itself on an unplanned event — `disk_critical | suppressed | key=demo-hp:disk_critical` — a collapse that yesterday would have left nothing at all. **The digest's silence is safe only because** the hub's deadline check raises `expected_backup_missed` from report freshness independently of any mail (`monitor/deadline.go:396,417`); that check is load-bearing for this row |
|
||||
| **A local backup is bounded by the box's FREE SPACE, not by a partition set at build time** — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime | golden `build-golden.sh` **v3.0.0**, agent **v0.120.0**, controller **v0.193.1** (R-165 / D-a / B2, completed by R-181) | **PROVEN-LIVE (2026-08-03) — BOTH halves** | `REPORT.md` (R-178 reinstalls) + `audits/SPIKE-r165-phase0-2026-08-03.md` (P1/P2/P3) + the bake transcript. **The golden bake is real evidence and is cited as such:** `build-golden.sh v3.0.0` produced `including mount point mp0 ('/var/lib/felhom')` with **no `mp1` line at all**, and its own guards printed `/var/lib/docker is a real mount`, `/mnt/sys_drive is a real mount` and `both paths are ONE filesystem`. Archive published (registry HTTP 200, sha `54e2a4c4…`). The B2 floor is unit-proven with 3 red-proofs and live on 9201 | **The row's FIRST clause is now PROVEN-LIVE; its SECOND is not, and they are separated deliberately.** **Proven (R-178, 2026-08-03):** *"a local backup is bounded by the box's FREE SPACE, not by a partition set at build time"* — both demo boxes reinstalled from this golden, by two different supply paths (demo-hp `--golden <local volid>`; demo-felhom the normal manifest route with **`verified sha256 54e2a4c431daf580… matches the hub manifest`**), each showing `mp0` at `/var/lib/felhom` with **no `mp1`**, both consumer paths real mounts on ONE filesystem (`stat -c %d` = `64519` on all three), 3/3 reboots each, and claim → deploy → backup → **restore** with a planted marker returning byte-identical. Space available to a recovery unit measured at **65 GiB / 233 GiB**, against the **19 GiB / 45 GiB** those boxes' `mp1` slices offered. **NOT proven — and measured FALSE in part:** *"a capture that would exhaust it is refused per app rather than allowed to stop the container runtime"*. The floor fired live for the first time (demo-hp 06:40:03) and does refuse per app, delete nothing, and alert — **but it is checked only in `captureAllRecoveryUnits`, while `runVolumeDumps` writes the bulk with no floor check at all**, so the leg that exhausts the volume is the unguarded one; and the refusal's claim that the previous unit is untouched was measured false (a 182,272 B dump replaced by 2,147,666,432 B under a manifest still dated 06:34:26). → **R-181, CLOSED THE SAME DAY (controller v0.193.0 + v0.193.1) and the second half is now PROVEN-LIVE TOO.** The reserve became a **per-app, per-run ADMISSION decision** taken before the app's FIRST write and covering all three legs (DB dump, volume dump, capture) — they write under one per-app root, which is what lets one verdict cover them honestly — and it gained a **size term**, so an app is no longer admitted at 96% and then allowed to write 2 GB. **Re-proven by filling demo-hp deliberately, once for EACH term, using the method that found the defect.** *Headroom @ 08:59:46* (906 MB free / 99%): both apps refused, **the whole `backups/primary` tree byte-identical — `TREE_SHA` 111d1760c18d3440f700634ab325f8b8 before and after**, opengist's tar still at its original 182,272 B; **no `Stopping <app> for safe volume dump` line at all**, which is the positive-by-absence observable that matters because that line IS present in the 08:58 baseline run; 0 volume dumps; one alert per app, HTTP 200. Space freed, re-run @ 09:01:33 → both captured normally. *Size @ 09:03:00*, reproducing the original sequence with a real 2 GiB file in opengist's volume (previous tar **2,147,666,432 B**, the exact figure the defect was measured at) and the filesystem at **91% used / 2.9 GB free — both headroom terms deliberately clear**: opengist refused `(size)` while **privatebin was ADMITTED and dumped normally**, proving the term is per-app rather than a global halt. **The refusal's wording was NOT weakened to fit** — the behaviour moved so the wording became true, and it is verified by tree fingerprint rather than by reading the log line, which is what lied. The `fallocate` instrument was re-proven on the rebuilt box before use (5 GiB step moved guest `df` while thin-pool `data_percent` held **36.83 → 36.83**), and teardown returned the pool to **29.43%**, below its own baseline. The golden **is now VOUCHED** (2026-08-03, hub `Artifact manifest set: … golden=0.192.0`), so fresh installs pick up the merged layout. Every box in the field that has not been reinstalled is still on the SPLIT layout and is unaffected: nothing assumes the merged shape at runtime, the controller's system_data_path is a path rather than a volume, and agent v0.120.0 FOLDS the retired `-sysdata-grow` into the single grow so an older `felhom-host-install.sh` still provisions the same total capacity |
|
||||
| Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | 6D/6E; hub OffsiteChecker | |
|
||||
| **A customer (not the operator) performs a restore via UI alone** | all | **MISSING** (as evidence) | — | Alpha will produce this; script it into R-3. **2026-07-19:** the C6 evidence attempt ran and found a **product gap instead of evidence** — `audits/DIAG-immich-restore-2026-07-19.md`. A customer-driven UI restore of a DB-indexed app cannot currently succeed (R-43 file-only restore, R-44 stale dump), so this row cannot flip until those close. Row stays MISSING **by finding, not by absence of attempt** — the rehearsal system working, not failing. **2026-07-19: the blocking product gaps are CLOSED in controller v0.148.0** (R-43 + R-44 shipped), so this row is now blocked only on the evidence run itself, not on missing capability. It flips the moment the §9 acceptance produces screenshots + the outcome flash + a snapshot ID. **2026-07-19 round 2 — PARTIAL EVIDENCE ONLY, row NOT flipped** (`audits/DIAG-immich-restore-round2-2026-07-19.md`): a deliberate run from snapshot `49e7cb46` did recover all 11 assets (`status=active`, files resolve), but the operation **reported failure** and left immich reporting schema drift, because the replay aborted against the running app (H4). Photos back ≠ clean acceptance. **2026-07-20: H4 closed in controller v0.153.0 (R-47) on BOTH paths, AND THE EVIDENCE RUN HAPPENED.** *(The "closing in v0.149" wording above was wrong — v0.149.0 was the F3 dashboard fix; R-47 shipped in v0.153.0.)* The C6 drill ran end-to-end **through the UI**: photos deleted, **trash emptied**, the full files+database restore pressed on `/backups/restore`, 40 files placed + 1 DB dump replayed rc-0, 11 assets back, no drift, timeline visually confirmed. The method note below is now DEMONSTRATED, not merely written down. Evidence: `felhom-controller/REPORT.md` 4e. **Residual: the run was performed by the OPERATOR, not by a customer** — for this row literal wording the alpha still owes one genuinely customer-driven pass, but no product gap blocks it. Method note for R-3's script: deleting in an app's own UI usually means *trash*, not deletion, so a drill written that way merges 0 files, flashes success and proves nothing — a real drill must empty the trash **and** verify the app's *content*, not the file count **Lane split → `07-backup-architecture.md` §3**: this row is Lane 1 (customer, unassisted). §8 rows 1–5 are the routes it would exercise |
|
||||
|
||||
@@ -574,6 +574,30 @@ because that was a different filesystem. On a merged box it can. Decision **B2**
|
||||
It **refuses per app and never deletes**: nothing on this filesystem is generational, so pruning could
|
||||
only destroy a different app's only local copy.
|
||||
|
||||
**WHAT IS RECORDED, WHAT IS E-MAILED, AND HOW OFTEN (controller v0.194.0 + hub v0.90.x, R-182).**
|
||||
The two are deliberately different mechanisms, because conflating them is how seven failures went
|
||||
missing on 2026-08-03 without leaving a trace.
|
||||
|
||||
| | Record | Notification |
|
||||
|---|---|---|
|
||||
| what | `recovery_unit_capture_failed`, one per failed app | `backup_run_failures`, one per RUN |
|
||||
| when | every time, unconditionally | at the end of a run, **only if something failed** |
|
||||
| gated by | nothing — not cooldowns, preferences or delivery | the hub's operator cooldown |
|
||||
| where it lands | the events table **and** `notification_log` (status `recorded`) | the operator's inbox |
|
||||
|
||||
- **A clean run e-mails nothing.** Silence means the run finished and found nothing wrong — and that
|
||||
is only safe because the hub's daily deadline check raises `expected_backup_missed` from the box's
|
||||
REPORT freshness, independent of any mail the box sends. That check is load-bearing for this
|
||||
design; weakening it re-opens a silent-failure path.
|
||||
- **A suppressed operator notification leaves a `suppressed` row** naming the key that suppressed it.
|
||||
Deciding not to tell someone is itself an event worth recording.
|
||||
- **Deliberate skips are not failures** and never appear in the digest — a disconnected or
|
||||
decommissioned drive has its own alert, and a nightly digest about an unplugged drive is one the
|
||||
operator stops reading.
|
||||
- **Cadence:** a nightly run gives at most one mail a day. A manual run always reports, even within
|
||||
the hour, because someone pressing the button is actively trying to get a backup. The periodic
|
||||
capture sweep is capped by the ordinary hourly cooldown.
|
||||
|
||||
**THE CONTRACT, stated as what the code provides (controller v0.193.0, R-181).** The reserve is a
|
||||
**per-app, per-run ADMISSION decision, not a capture check.** It is taken once for an app, immediately
|
||||
before that app's FIRST write of the run, and it covers **all three write legs — the database dump, the
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
**Class:** SPIKE (empirical validation; no product code). **Repos:** felhom.eu (this doc only);
|
||||
felhom-agent read-only for grounding (`internal/pbs/{client,pin}.go`, `configs/build-golden.sh`,
|
||||
`internal/hub/cloudflared.go`, `internal/escrow/identity.go`).
|
||||
**Probe ends:** `felhom-hetzner` = Hetzner CX23, Debian 13.4, public IPv4 `167.233.158.164`,
|
||||
**Probe ends:** `felhom-hetzner` = Hetzner CX23 (**note added 2026-08-03: rescaled to a CX33, 8 GB RAM — this spike records the machine as probed and its body is deliberately unchanged**), Debian 13.4, public IPv4 `167.233.158.164`,
|
||||
global IPv6 `2a01:4f8:...::/64` (throwaway — NOT the live jarrs.eu box) ⟷ **demo-felhom** =
|
||||
the real PVE 9.2.2 host on the operator's home line (One Hungary fixed cable, Budapest), driven
|
||||
over the existing LAN SSH path; the tunnel itself always dialed **out**.
|
||||
|
||||
@@ -26,11 +26,11 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **R-114** | **On target-drive loss the customer is told the wrong story and offered the drive that just vanished.** With the assigned target absent, the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"a rendszermentés ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a drive that has disappeared, not the system disk — **and** `offer_path` pointing at the missing drive as the remedy | **SHIPPED + PROVEN-LIVE** (controller v0.186.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** With the target absent the page rendered the ABSENT copy (1), the system-disk copy 0, the offer block 0 — both of E-2d's falsehoods gone. API carried `message:"A rendszermentés meghajtója nem érhető el…"` with `target:felhom-backup`. **FIXED: the third state exists.** New `BackupTargetState.TargetAbsent` separates *configured-and-gone* from *never-configured*. `Degraded` keeps its meaning (is there a problem) so the wire contract is unchanged for every consumer; `TargetAbsent` answers which problem, because the remedies are OPPOSITE — attach any second drive vs reconnect *that* one. Copy routed through `degradedMessageFor` (still one decision point) and taken **verbatim** from the hub's `backup_target_absent` email so banner and mail tell one story. **Offer suppressed on the branch itself**, deliberately not left to `firstOfferableDrive`'s `Disconnected` skip — that flag is set by R-113 in another repo, and this state must be right without it. Red-proof: deleting the branch reproduces E-2d's exact payload, offering `/mnt/felhom-drives/mentes2`, the drive that had vanished. **MinAgent unchanged 0.113.0** — R-114 reads `BackupTarget`/`MountPath`/`GuestPath`/`Role`, none of which R-113 altered, so demo-hp is not held. **NOT live-validated: Scenario C cannot occur on a healthy box.** `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing **never configured** from **configured and now missing**. Shares R-113's root cause — two disagreeing presence signals — but is a different code path with a different fix. **Currently invisible ONLY because of R-112; fix this before wiring that.** Also seen: after reattach the drive returned as `/dev/sdc` while the stable bind still recorded `/dev/sdb`, and the state read healthy. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 | CC |
|
||||
| **R-29** | **The green gates are not enforced anywhere — one was RED for 16 releases before anyone ran it.** This is the **class**, not an instance: a gate that exists, asserts something true, is red, and is invoked by nothing reads as coverage it is not providing. `controller/scripts/docker_run_volume_path_gate.py` failed continuously from **2026-07-14 (v0.129.0)** until R-7b's close-out ran it by hand at v0.145.0 — sixteen releases in which every REPORT said "green" | **CLOSED — both halves shipped** (2026-08-02) | — | **This item has existed at `ROADMAP.md:158` since before the register was rebuilt (2026-07-27) and was never carried across — that omission is itself part of the finding**, because it is an open item *about work not getting done* that then went missing from the page that decides what gets done. Two separable parts, per R-29's own analysis: **(a)** the `docker_run_volume_path_gate` finding is benign and the fix is a 3-line ALLOWLIST addition with its why — **not** a rewrite of the flagged call — and it gets its own reviewed diff, never bundled into a feature commit; **(b)** the systemic half, the real item: decide where gates run (pre-push hook, `build.sh` step, or CI) and make a red gate block the train the way the Go green gate does. **Two further orphans confirmed 2026-07-29** by repo-wide grep across all file types + sibling repos + `~/.claude` settings/skills/hooks + `.git/hooks` (none non-sample) + Makefile/justfile/Taskfile find (only `hub/Makefile`, zero `gate` occurrences) + CI-directory find (**this repo has no CI at all**) — every one of the 19 hits is a docstring, a code comment or prose, and **not one is an invocation**: `scripts/hostinstall_gates.py` — **RED today** (`hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0`, exit 1), the same finding as **R-94 leg (b)** — and `scripts/hub_confirm_gate.py`. Of the four gates in `scripts/`, only `site_gates.py` is mandated anywhere (`CLAUDE.md:153`) and `manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`. **In R-29's own words, carried forward deliberately: do not mint a new ID for a new instance** — the 2026-07-18 rehearsal independently re-raised this item and no second ID was minted then either **UPDATE 2026-08-02 — leg (a) CLOSED** (`felhom-controller` `c432f70`, its own reviewed diff as specified): `appexport/estimate.go`'s `-v` is a NAMED VOLUME mounted read-only into a throwaway container, no host path, structurally identical to the allowlisted `backup/backup.go` entry — allowlisted with its why; `realVolumeSize` untouched. **Leg (b) HALF-SHIPPED:** the 'decide where gates run' ruling is now made and half-implemented — **every repo has ONE entry point** (`felhom.eu/scripts/repo_gates.py`, `felhom-controller/controller/scripts/controller_gates.py`, `felhom-agent/scripts/agent_gates.py`, `app-catalog-felhom.eu/scripts/catalog_gates.py`), each mandated in its `CLAUDE.md` and each wired to `.githooks/pre-push` via `--fast`. **THE CENSUS, which is the finding:** thirteen gate scripts across four repos; **every gate a `CLAUDE.md` names was GREEN, and two of the four nobody names were RED** — `hostinstall_gates.py` (red since 2026-07-14) and `reuse_refs_check.py` (red on all four repos); a third, `docker_run_volume_path_gate.py`, was named only in `REUSE.md:284` and was also red. Correlation with 'named in a CLAUDE.md' was exact. **STAYS OPEN for the automatic half** — a hook is per-clone and `--no-verify` skips it; the unbypassable half is CI → **R-168** **CLOSED 2026-08-02, on the demonstrated ALARM and not on a green run.** Leg (b)'s automatic half is now live: a Gitea Actions runner re-runs every repo's entry point on every push, independent of who pushed and of what they typed (→ R-168). The class this row opened — *a gate that exists, asserts something true, is red, and is invoked by nothing* — is answered at both ends: the pre-push hook refuses locally, and CI catches a `--no-verify` bypass and **emails the operator**, proven with a real red run and a provider accepted-id. What remains is not this row's finding but a working-style choice — CI reports rather than blocks because there is no merge to gate (→ R-169) | CC |
|
||||
| **R-95** | restic offsite credential **can delete** (`readonly=False`, `forget --prune` runs from the box); SFTP cannot express append-only | **READY** | — | Root exposure still open. Mitigation now ARMED — split prune off-box or move to REST `--append-only` | CC |
|
||||
| **R-86** | Restore-tests are interval-scheduled, not backup-aligned | **READY** | R-90 (ep0 headroom) informs cadence | Trigger a tier ~24 h after **its own** newest archive | CC |
|
||||
| **R-86** | Restore-tests are interval-scheduled, not backup-aligned | **READY (M) — UNBLOCKED and re-ranked 2026-08-03** | — | Trigger a tier ~24 h after **its own** newest archive. **The R-90 dependency is discharged:** that row informed the cadence because ep0 had 3.8 GB and a 14.46 GB restore read had OOMed it, so a more frequent restore-test risked knocking the offsite endpoint over. ep0 is now a **CX33 with 8 GB RAM plus a 4 GiB swapfile** (measured 2026-08-03), so headroom is no longer what sets the cadence and this can be designed on its own merits. **Do not read that as "the ceiling is gone":** the OOM was a 14.46 GB restore against 3.8 GB, and 8 GB is comfortable rather than unbounded — the restore-test cadence should still be paced, just not by fear of the endpoint | CC |
|
||||
| **R-87** | The restic tier is never restore-tested | **READY** | — | Design a controller-side test (no scratch-guest analogue transfers) | CC |
|
||||
| — | Storage Box **snapshots** on `storage-box-pool-1` — plan SET (daily 00:00, keep 7) but **0 taken yet** | WATCHING | first run tonight 00:00 | Confirm `size_snapshots > 0` tomorrow; until then the mitigation is armed, not proven | CC |
|
||||
| — | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | WAITING-ON-OPERATOR | operator console | Delete the box | operator |
|
||||
| **R-90** | ep0 RAM headroom — 4 GiB swap **survived its first reboot** 2026-07-27; 3.8 GB RAM unchanged | **BLOCKED** (interim proven) | Hetzner CX33 availability — **confirmed unavailable even powered OFF**, so it is the Cost-Optimized "Limited availability", not the power state | Re-check CX33; escape hatch if urgent = CPX/CCX lines (no availability warning, higher cost) | operator |
|
||||
| **R-90** | ~~ep0 RAM headroom — 4 GiB swap survived its first reboot 2026-07-27; 3.8 GB RAM unchanged~~ | **CLOSED — the operator rescaled ep0 to a CX33 on 2026-08-03** | — | **MEASURED ON THE BOX, not read from an invoice:** `felhom-hetzner` reports `Mem: 7757` MB total (**8 GB**, was 3.8) and `nproc` **4**. **The interim lever survived and was checked rather than assumed** — a resize is a stop/start, so "the swapfile is still there" was an assumption until measured: `/swapfile`, 4 GiB, dated `Jul 27 14:40`, **active** (`swapon --show` → `/swapfile file 4G 0B -2`), 0 B in use on an idle box. **THE 40 GB LOCAL DISK DID NOT CHANGE** and must not be "corrected" alongside the RAM: `/` is 38 G, 58% used. This was a CPU/RAM resize only, so every disk figure in the runbooks still stands — the separate 98 G volume at `/mnt/pbs-datastore` (R-82 P0.3) is unaffected. **Why this was BLOCKED and no longer is:** the row recorded CX33 as *"confirmed unavailable even powered OFF"* — the Cost-Optimized line's limited availability, not a power-state problem. It became available and the operator took it. **Documentation corrected** (`RUNBOOK-ep0-datastore-volume`, `RUNBOOK-pbs-prune-serverside` ×2, `runbooks/offsite-endpoint.md` ×2, `runbooks/target-selection.md`) and **audit/evidence documents ANNOTATED, not revised** (`SPIKE-connectivity-wireguard-2026-07-03`, campaign-10 `phaseA-journal`) — they record what was true when written and that is their value. **Still open and still the operator's, deliberately untouched:** `target-selection.md`'s *"D-d did not name ep0 either way. Confirm it explicitly."* | — |
|
||||
| **R-91** | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC |
|
||||
| — | First-ever **GC** on `felhom-offsite` (armed today 13:11 UTC, never run) | WATCHING | schedule | **Sun 2026-08-02 04:30 UTC** — confirm it completes | CC |
|
||||
| — | demo-felhom's next weekly PBS backup (newest is 2026-07-26) | WATCHING | schedule | ~2026-08-02; also releases R-91 | CC |
|
||||
@@ -94,7 +94,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **R-176** | **Two prerequisites for the R-165 merge are UNMEASURED, and both are cheap.** (a) Whether a **pre-merge archive** (carrying `mp1`) restore-tests cleanly into a **merged-layout** guest — reading `mountParity` (`felhom-agent/internal/reconcile/restoretest.go:347`) says it should, because the restore recreates `mp1` from the archive so archive and restored guest agree; **that was reasoned from source and never executed.** (b) The in-place per-box migration (move `<mp1>/felhom-data` onto `mp0`, drop the slot, verify) has **never been rehearsed even once**, so "is the box restorable at every point of it?" is currently unknown | **(a) ANSWERED 2026-08-03 (P1: PASS). (b) NOT REQUIRED — operator ruling: every node is reinstalled, none migrated** | blocks R-165 landing safely | **Filed because this project's own record is that FOUR production designs specced against unvalidated mechanisms were all wrong** — which is exactly why R-165's own spike refused to design. Both are one command on a **Tier-0** box (D-d: both demo boxes are disposable). (b) is only required work if Peti's box turns out to need migrating rather than reinstalling — the hub cannot answer that (M5: `peti-felhom` exists as a customer with **no host in the register**), so it is the operator's input. **ID established free:** `grep -ro "R-176\b" documentation/ *.md` → 0 hits **UPDATE 2026-08-03.** **(a) is measured and passed** — `audits/SPIKE-r165-phase0-2026-08-03.md` P1: a real pre-merge archive (`mp0+mp1`, confirmed from its own vzdump log) restore-tested on demo-hp, `pass: true`, `mount_parity: ok`, 84 s, with `mountParity` untouched. One limit stated rather than glossed: it ran with the pre-merge agent because the merged one did not exist yet, and the comparison is archive-vs-its-own-restore which never consults the host layout — **re-run it once against agent v0.120.0**, which is one command. **(b) is withdrawn, not deferred:** the operator ruled that every node is REINSTALLED rather than migrated in place (both demo boxes are Tier 0; the colleague's box carries none of our customer data and is clean-installed in a few weeks), so the in-place migration rehearsal has no consumer. Recorded explicitly rather than silently skipped | CC |
|
||||
| **R-183** | **A fresh install fetched the vouched agent BINARY and its sixteen CONFIG files from two different refs, and nothing compared them.** `felhom-host-install.sh` resolved the agent version from the hub manifest and sha-verified the binary — then took `felhom-agent.service`, `felhom-agent.sudoers` and fourteen more from `raw/branch/main`, i.e. whatever the agent repo's tip happened to hold at that second. One install, two refs, no comparison | **CLOSED — SHIPPED** (installer v1.23.0, 2026-08-03) | — | **Found while implementing R-110, by reading `fetch_raw`'s call sites rather than the spec's description of them** — the task said nine files from `felhom.eu`; they are **sixteen** and they come from **`felhom-agent`**. **Why it is a defect and not only untidiness:** these files are the agent's own operating surface — its systemd unit, its sudoers, its guarded wrappers — and `configs/felhom-backup-target-apply` is installed **0755 into `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n`. A config newer than the binary is a root-executed artifact the vouched version was never tested against. **Not hypothetical in shape:** the fleet has shipped exactly this class before, where a config and the code that reads it moved independently. **Fixed by pinning to the agent version the install is already committed to**, on the operator's ruling: `raw/tag/v$ART_AGENT_VER/`, resolved from the hub manifest that the binary's sha is already checked against — so binary and configs now come from ONE ref by construction. **No fallback to a branch**, deliberately: a missing tag dies loudly rather than quietly serving `main`. Pinned by `hostinstall_gates.py` gate 6 (no `raw/branch/` anywhere; the `$ART_AGENT_VER` pin still present), red-proofed by reverting one of the sixteen and by removing both assertions. `felhom-agent` now carries `v<semver>` tags (`v0.120.0` created retroactively at `cd6e267`, the commit the published binary was built from; `configs/` is byte-identical there and at `main`, so nothing depended on the choice) and `release-agent.sh` creates them as part of releasing | — |
|
||||
| **R-184** | **Nothing prevents the hub from vouching an agent version that was never released.** The R-115 gate proves every RELEASED version is installable, but it works from git tags — so a hub artifact-manifest entry naming a version with no tag and no package is invisible to it. The installer would then die at step 5 on a virgin machine, as root | **READY (S) — NEW 2026-08-03** | — | **Filed BECAUSE the R-115 gate deliberately does not cover it, rather than leaving the gap unstated.** CI cannot check it: the hub's `/api/v1/artifacts/<customer>` answers **401** without a per-customer retrieval passphrase and Gitea's package **listing** api answers **401** without a token (both measured 2026-08-03, P-C), so a credential-free gate can ask *"is this version installable"* but never *"which version is vouched"*. **Two shapes, and the second is better:** (a) give CI a hub credential — expands what CI can reach, and is the operator's call not a gate author's; (b) **validate at vouch time, in the hub**: the operator UI's Day-0 artifact form refuses a version whose package is not downloadable. (b) fails closed at the moment of the decision, needs no new credential anywhere, and puts the check where the mistake is actually made. **Exposure is low and should be said so:** vouching is a deliberate operator action against a version they have just released, and R-115's release path now makes released-but-unpublished nearly impossible. This is the residue, not the main risk | CC |
|
||||
| **R-182** | **A full disk tells the operator about ONE app and silently swallows every other app's refusal for an hour.** The hub's operator cooldown key is `customerID + ":" + eventType + cooldownTierSuffix(details)` (`hub/internal/notify/dispatcher.go:268`). `recovery_unit_capture_failed` carries **`app`** in its details and **no `tier`**, so the suffix is empty and the key contains **no app identifier**: the first refused app's alert takes the 1-hour slot and the second app's is dropped — and dropped BEFORE `LogNotification`, so it leaves **no row on any channel**. It cannot even be audited after the fact | **READY (M) — RE-SCOPED 2026-08-03, and the direction REVERSED** | — | **FILED AS THE OPPOSITE DEFECT AND THE MEASUREMENT OVERTURNED IT.** It was filed 2026-08-03 as *"the reserve re-alerts on every status refresh"* — too MANY alerts — from the controller-side observation that a second push followed 13 s after the first. That was the sending end. **Measured at the receiving end** (hub `notification_log` + `events`, read from a copy taken WITH its `-wal`, freshness confirmed by the newest row post-dating the session): **9 events received today → 2 operator e-mails sent.** **06:40:03** privatebin AND opengist both refused → **opengist e-mailed, privatebin's alert has no row at all**. **08:59:46/47** opengist AND privatebin both refused → **privatebin e-mailed, opengist's absent**. **08:59:59, 09:03:00, 09:07:06** → **no operator row whatsoever**, all inside the 1-hour cooldown opened at 08:59:47. So the controller pushing repeatedly is not the defect; the hub emitting at most one operator e-mail per customer per hour is, and the loser is silent. **This is R-97a's failure mode exactly, in a second event type.** That row's own comment states it: *"`felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20 for the whole hour"*. `cooldownTierSuffix` was written NARROW on purpose — empty unless the producer sends a `tier` — so no existing type's behaviour changed; `recovery_unit_capture_failed` simply never opted in. **CORRECTION OWED, and it is the reason this was worth measuring:** the 2026-08-03 R-181 report said *"one `recovery_unit_capture_failed` per app, HTTP 200"*. That was **true of what the CONTROLLER pushed** and would be read as *the operator was told about each app* — which is **false**. The distinction between an accepted event and a sent e-mail is the whole of this row. **Fix shape (NOT implemented — Part 0 was investigation only, by instruction):** let the producer opt into a per-app cooldown key, the way R-97a let the whole-guest producer opt into a per-tier one — the narrow mechanism already exists and needs no widening. **And a suppressed operator alert should leave a `skipped` row rather than nothing**, or this class stays undiagnosable from the hub's own records | CC |
|
||||
| **R-182** | **A full disk tells the operator about ONE app and silently swallows every other app's refusal for an hour.** The hub's operator cooldown key is `customerID + ":" + eventType + cooldownTierSuffix(details)` (`hub/internal/notify/dispatcher.go:268`). `recovery_unit_capture_failed` carries **`app`** in its details and **no `tier`**, so the suffix is empty and the key contains **no app identifier**: the first refused app's alert takes the 1-hour slot and the second app's is dropped — and dropped BEFORE `LogNotification`, so it leaves **no row on any channel**. It cannot even be audited after the fact | **CLOSED — SHIPPED** (controller v0.194.0 + hub v0.90.0/.1, 2026-08-03) | — | **FILED AS THE OPPOSITE DEFECT AND THE MEASUREMENT OVERTURNED IT.** It was filed 2026-08-03 as *"the reserve re-alerts on every status refresh"* — too MANY alerts — from the controller-side observation that a second push followed 13 s after the first. That was the sending end. **Measured at the receiving end** (hub `notification_log` + `events`, read from a copy taken WITH its `-wal`, freshness confirmed by the newest row post-dating the session): **9 events received today → 2 operator e-mails sent.** **06:40:03** privatebin AND opengist both refused → **opengist e-mailed, privatebin's alert has no row at all**. **08:59:46/47** opengist AND privatebin both refused → **privatebin e-mailed, opengist's absent**. **08:59:59, 09:03:00, 09:07:06** → **no operator row whatsoever**, all inside the 1-hour cooldown opened at 08:59:47. So the controller pushing repeatedly is not the defect; the hub emitting at most one operator e-mail per customer per hour is, and the loser is silent. **This is R-97a's failure mode exactly, in a second event type.** That row's own comment states it: *"`felhom-pbs` failing at 09:00 would swallow `local` failing at 09:20 for the whole hour"*. `cooldownTierSuffix` was written NARROW on purpose — empty unless the producer sends a `tier` — so no existing type's behaviour changed; `recovery_unit_capture_failed` simply never opted in. **CORRECTION OWED, and it is the reason this was worth measuring:** the 2026-08-03 R-181 report said *"one `recovery_unit_capture_failed` per app, HTTP 200"*. That was **true of what the CONTROLLER pushed** and would be read as *the operator was told about each app* — which is **false**. The distinction between an accepted event and a sent e-mail is the whole of this row. **Fix shape (NOT implemented — Part 0 was investigation only, by instruction):** let the producer opt into a per-app cooldown key, the way R-97a let the whole-guest producer opt into a per-tier one — the narrow mechanism already exists and needs no widening. **And a suppressed operator alert should leave a `skipped` row rather than nothing**, or this class stays undiagnosable from the hub's own records. **CLOSED 2026-08-03 — controller v0.194.0 (`88897a2`) + hub v0.90.0/.1 (`dd40f85`, `f21e7ca`), proven live on demo-hp.** **The operator ruled against the obvious fix** (putting `app` in the cooldown key): on a full disk that is one e-mail per app, the volume problem wearing the correctness problem's clothes. What shipped instead is **one digest per run** plus **a record for every failure**. **(1) The silent drop is gone.** `processOperator`'s cooldown no longer returns bare — a suppressed operator event writes a `suppressed` row carrying the message and the key that suppressed it, **for every operator event type, not only the one that exposed this**. The cooldown's duration and semantics are deliberately unchanged: this makes the drop visible, not absent. **(2) `backup_run_failures`** — one operator mail at the end of a run listing every failed app, its leg and its reason, with the counts and the free space; in `allowedEventTypes` AND `operatorOnlyEvents`, with a test demonstrating a customer who has enabled it receiving nothing. **(3) `recordOnlyEvents`** — a third routing class. `recovery_unit_capture_failed` is now stored + logged every time and never mailed: **the record and the notification are different things**, which is the durable half. A register rather than a severity downgrade, which would have relabelled a genuine failure as informational everywhere it is queried. **(4) `cooldownRunSuffix`** — a sibling of `cooldownTierSuffix`, so `tier` keeps byte-identical semantics and R-97a is untouched. A unique `run_id` per real run means the cooldown can never collapse a manual run into the nightly one (the operator ruled on that explicitly). **A GAP THE SPEC DID NOT ANTICIPATE, AND ITS FIX.** `recovery_unit_capture_failed` also fires from `GetFullStatus`'s periodic sweep, outside any run. Making it record-only would have left those failures recorded and **never notified** — a NEW silence created while closing one. So the sweep emits a digest too, deliberately with **no `run_id`**, so it stays under the ordinary hourly cooldown exactly as before while the mail now liLine truncated
|
||||
| **R-181** | **The capture floor guards the cheap leg and not the leg that fills the volume — and its refusal message asserts an invariant the code does not provide.** B2 (controller v0.192.0) is recorded on R-165 as the deliberate replacement for the bulkhead the `mp1` partition used to give. It is consulted in exactly one place — `m.unitFloorBlocked(stack.Name)` at `recovery_unit.go:328`, inside `captureAllRecoveryUnits`, which writes a manifest and a compose copy: **a few KB.** The leg that writes the bulk, `runVolumeDumps` (`backup.go:535`), has **no floor check at all** — its gates are protected-stack, volume-less, disconnected, decommissioned — and it runs FIRST, by design (*"MUST run before captureAllRecoveryUnits so the manifests enumerate the fresh tars"*, `backup.go:483`). So the write that fills the filesystem is unguarded, and the floor then refuses the write that would have cost almost nothing. **Second limb: the refusal message is false.** `recovery_unit.go:331` prints *"the previous unit is untouched and NOTHING was deleted"*. Nothing was deleted — true. Untouched — **measured false**: privatebin's `volume-dumps/privatebin_privatebin_data.tar` went `26c546c2…` → `b538ab89…` and opengist's went **182,272 B → 2,147,666,432 B**, both rewritten by the earlier leg, while each unit's `manifest.json` kept `created_at: 2026-08-03T06:34:26Z` and its `checksums` block covers only the three compose files — so a unit's payload can be swapped under a stale descriptor and **nothing in the unit can detect it** | **CLOSED — SHIPPED** (controller v0.193.0 + v0.193.1, 2026-08-03) | unblocks **R-165** | **FIRST LIVE FIRING OF B2, and it is why the runbook asked for one.** Proven on demo-hp 2026-08-03 06:40:03 on a box reinstalled from the merged golden (R-178). Method: a real 2 GiB file in opengist's data volume, then `fallocate` to bring the filesystem to 96 % used / 3.0 GiB free — both floor terms deliberately still clear, so the run started. **The `fallocate` instrument was proven before use** (5 GiB moved guest `df` 977M→6.0G while thin-pool `data_percent` stayed 29.03 → 29.03: zero blocks allocated), because demo-hp's thin pool is 53.93 GiB and a real fill to 97 % of a 70 G volume would have exhausted it and corrupted every guest on the box including the `drill-r50` fixture. Sequence observed: opengist's volume dump wrote **2.0 GB unguarded** → free fell to 1.0 GB → **both** apps' recovery-unit captures were then REFUSED on the `1.0 GiB free` term, each pushing `recovery_unit_capture_failed` (severity `error`) to the hub, accepted HTTP 200. **What DOES hold: it refuses per app rather than aborting the run, it never deletes, and the alert reaches the operator.** **Fix shape, not written this session by design (§7 of the runbook):** the floor belongs before the write in `runVolumeDumps` too, the message must stop claiming what the earlier leg has already falsified, and per `CLAUDE.md` *"a comment asserting an invariant needs a test pinning it"* the pinning test must assert the **consequence** (after a refusal, is the previous unit's payload byte-identical?) and not the mechanism. **Class:** the sixth entry in `CLAUDE.md`'s own table of shipped guarantees the code did not provide — found, as four of those were, only on live hardware. **CLOSED 2026-08-03 — controller v0.193.0 (`fef07c3`) + v0.193.1 (`6c43bf6`), proven live on demo-hp.** **The fix is ONE admission verdict per app per run** (`internal/backup/admission.go`), taken before that app's FIRST write and consulted by all three legs — the three write under one per-app root (`appbackup.RecoveryUnitPath`), which is exactly why one verdict can honestly cover them. **Decided LAZILY at the app's first write, never once at run start**: app A's dump can put app B under the reserve, so a run-start verdict would read a disk that no longer exists — the same class of mistake one level up. **Never re-decided between an app's own legs** (that IS the split this closes) and **reset per run**. Placed ahead of `DumpAppVolumesSafe`, which stops the stack as its first act, so a refused app is never bounced; placed AFTER the volume-less check, which has no write to gate. Exactly ONE operator alert per refused app per run. Leg order unchanged. **The floor is now SIZE-AWARE**, which is the second half of the defect: it asks whether THIS app's write would cross the reserve, not only whether the filesystem is already below it — the term whose absence admitted an app at 96% and then let it write 2 GB. Estimate = the app's previous `.sql`+`.tar` on disk; **no history → headroom-only** deliberately, or the first backup becomes the one that can never happen, and the alert says so. **A container-based `du` was MEASURED and rejected, not assumed**: 66 timed runs on demo-hp guest 9201, **median ~355 ms/volume (341–404)** on volumes holding tens of KB — the cost is container start-up, not the walk. Decisive on top: `docker run` needsLine truncated
|
||||
| **R-180** | **`--archive-storage` is accepted without checking the agent's token will ever be granted on it, and the failure lands at step 8/8 — after the root@pam password has already been rotated.** `felhom-host-install.sh` validates the archive storage EXISTS (`pvesm status --storage`, `:1583`) and that the golden volid RESOLVES on it (`:1661`), both in pre-flight. It never checks that storage against the ACL set it is about to grant, which is the fixed default `local local-lvm felhom-pbs` (`--acl-storages`, which `runbooks/day0-install.md` tells the operator **not** to pass). A storage outside that set therefore passes every pre-flight gate and dies at the last step | **READY (S) — NEW 2026-08-03** | — | **Hit live on demo-hp 2026-08-03** during R-178 Phase A, self-inflicted and therefore a clean demonstration: the golden was staged on `felhom-backup` (the enrolled NVMe, where the box's vzdumps live) and `--archive-storage felhom-backup` passed. Pre-flight passed; steps 1–7 ran; step 8 returned `reconcile: bring-up restore: proxmox: POST /nodes/felhom-host/lxc -> HTTP 403: permission denied at /storage/felhom-backup (missing privilege Datastore.AllocateSpace)`. **The cost is the ORDER, not the error** — by the time it fires, step 2 has minted the PVE token, step 4b has **rotated root@pam and vaulted it** (so the old console password is already dead), and step 5 has installed the agent. Recovery was `--resume` after moving the golden to `local`, which worked cleanly. **This is statically checkable in pre-flight**: `ARCHIVE_STORAGE ∈ PVE_STORAGES` is a one-line assertion over two variables both known at `:1583`. Same class as R-29 — the checkable thing that nothing checks | CC |
|
||||
| **R-179** | **`--uninstall` leaves the NAS network-storage systemd units behind, with the automount in `failed` state and the parent bind still mounted.** The teardown's residue-diff provenance (`day0-install.md` Part E: *"a full-filesystem diff against the pre-install baseline showed zero Felhom-named leftovers"*) is from **v1.9.1**, which predates the NAS network-storage feature. A box that has ever had a network share configured keeps `/etc/systemd/system/mnt-felhom\x2ddrives-<share>.mount` and `.automount` after a full uninstall | **READY (S) — NEW 2026-08-03** | — | **Observed on demo-hp 2026-08-03** after `--uninstall --vmid 9201`: `mnt-felhom\x2ddrives-Felhom\x2dShare.automount` **loaded failed failed**, its `.mount` `loaded inactive dead`, and `mnt-felhom\x2ddrives.mount` still `active mounted` — the uninstall's own output had warned `/mnt/felhom-drives/Felhom-Share is busy — NOT forcing` and `/mnt/felhom-drives root bind left mounted`, which is correct behaviour (it never forces an unmount) but is not teardown. Cleared by hand before the reinstall: stop both units, remove both unit files, `daemon-reload`, unmount the autofs then the parent. **NEGATIVE CONTROL, same day:** demo-felhom's uninstall left **nothing** (`ls /etc/systemd/system | grep -i felhom` → only the unrelated `felhom-bootstrap.service`; no felhom mounts) — because that box had no network share configured. **So the residue is conditional on the feature having been used, which is exactly why a diff taken on a box that never used it reported clean.** `felhom-bootstrap.service` is NOT residue — it is the ISO first-boot unit, `disabled`+`inactive`, exactly-once and already fired | CC |
|
||||
|
||||
@@ -144,7 +144,7 @@
|
||||
| R-83 | **Ratify or retire `07-backup-architecture.md`** | S | **DISCHARGED (2026-07-26) — brought current, NOT ratified** | Per CONTEXT ruling S-2 the doc now opens with an honest status header: what each part was verified against, and an explicit ⚠️ **STALE outside the PBS tier** verdict (the controller-side tiers were last verified 41 controller versions ago). The §2 PBS row was corrected — it claimed "PBS on DooPlex" (the retired spike store) with no cadence; it now names `felhom-pbs` → `felhom-offsite` on ep0 over `wg-felhom`, weekly, keep_last=2 — and §9.1 records the measured capacity. **Deliberately NOT marked ratified: ratification is Viktor's review of the §10 list, not an editor's.** R-82 re-verified only the rows it touched and says so, rather than implying a freshness it did not earn. |
|
||||
| R-84 | **The agent's cold backup `Store` no longer causes a redundant backup** | S | **SHIPPED (agent v0.103.0, 2026-07-26)** | Observed live: three redundant local backups on demo-felhom in one afternoon of deploys. The `Store` is in-memory ("lost on restart; the cadence re-populates"), so after every restart `/backup/due` said *no successful backup recorded yet* and the controller took another — on the OFFSITE tier a wasted multi-hour WAN upload after every agent deploy. **Resolved by asking the STORAGE rather than persisting the store** (`BackupRunner.NewestArchiveTime` + the optional `localapi.BackupArchiveLister`): ground truth, so a pruned archive correctly stops counting where a persisted record would keep claiming a backup that no longer exists; no new on-disk state, no migration. It answers only *when did a backup last land* — the richer fields stay with real records, so the host-report never carries invented numbers. Fail-safes: read error → fall back to memory (never fake freshness, never suppress); genuinely empty → due; old archive → still due. **Proven live on both boxes** immediately after a restart with the store cold: demo-felhom `local age=5298s / felhom-pbs age=14373s`, demo-hp `local age=498s / felhom-pbs age=2346s`, all `due:false`. |
|
||||
| R-85 | **The DR tier must be restore-tested UNATTENDED, and its failure must be HEARD** | M | **Code SHIPPED (agent v0.104.0 + hub v0.77.0, 2026-07-27); rotation NOT YET OBSERVED LIVE** | R-82 proved a PBS restore round-trip **by hand**. Three defects kept the unattended path from covering it, each verified at source: **(a)** `buildRestoreTestScheduler` took `cfg.Backup.BackupTarget()` and built ONE runner on it, so the offsite tier's archives were never candidates; **(b)** the `Spec` was an immediately-invoked function evaluated ONCE at daemon start, so nothing tier-varying was expressible and any storage-type/config change stayed stale until restart; **(c)** a failed restore-test was a `[WARN]` log line with **no event, no notification, no gauge** — true for the LOCAL tier that was already being tested, so the loudest DR signal this system produces was inaudible. (c) is the one that mattered: rotating tiers without it would only mean two tiers can fail silently instead of one. **Selection ruling (operator, 2026-07-26): oldest-first (Option 1)** — never-proven sorts first, which is where the offsite tier starts; no new cadence knob. The per-tier last-proven timestamp is now reported **so the achieved interval can be MEASURED before anyone decides whether a knob is needed** — measure, then configure, not the reverse. **Shipped:** per-run `SpecBuilder`; `PickRestoreCandidateOn` (a tier with no archive NO-OPS, never fails); persisted `RestoreTestState` (rotation survives a restart — credit only on SUCCESS, or a permanently failing tier looks freshly proven); `backup.InFlight`, the host-wide one-heavy-op gate shared with the backup path (a LINK concern, not a lock one — an offsite restore pulls multi-GB over the tunnel a backup pushes one); and hub `restore_test_failed` / `restore_test_stale`, kept **distinct** and anchored on R-81's verdict structure, operator-tier only. **NOT yet proven:** rotation has not been observed selecting both tiers across consecutive unattended cadences — a multi-day window at a 24h cadence. Manual restore-proof IS proven (R-82). |
|
||||
| R-86 | **Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup** | S | **NEXT — operator ruling 2026-07-27** | R-85 schedules restore-tests on a free-running INTERVAL and rotates oldest-proven-first. That cannot express *"the day after the PBS backup"*: any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. **The ruling:** weekly per tier (daily is overkill), timed **~1 day after that tier produced a new backup** — so the gap between an archive being written and being proven stays small, and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **Shape:** trigger a tier from **its own last successful backup** rather than a clock — test tier X once its newest archive is ~24 h old and not yet proven. Self-aligning when a backup runs late, and it naturally stops testing a tier that has stopped producing backups (today an interval keeps re-testing the same stale archive). Needs the scheduler to consult per-tier backup times (already available: `BackupRunner.NewestArchiveTime`, R-84) alongside `RestoreTestState` (R-85). **Interim in force:** `restore_test_cadence_seconds = 302400` (3.5 d) on both boxes — oldest-first over two tiers lands each tier ~weekly, which matches the cadence half of the ruling but NOT the alignment half. |
|
||||
| R-86 | **Backup-ALIGNED restore-test scheduling — test a tier ~1 day after ITS OWN backup** | S | **NEXT — UNBLOCKED 2026-08-03 (R-90 closed: ep0 is now a CX33, 8 GB + 4 GiB swap, so endpoint headroom no longer sets the cadence — though 8 GB is comfortable, not unbounded: the original OOM was a 14.46 GB restore)** | R-85 schedules restore-tests on a free-running INTERVAL and rotates oldest-proven-first. That cannot express *"the day after the PBS backup"*: any fixed offset drifts against the backup schedule, so alignment would be luck rather than design. **The ruling:** weekly per tier (daily is overkill), timed **~1 day after that tier produced a new backup** — so the gap between an archive being written and being proven stays small, and a tier is never proven against an archive about to be pruned (offsite retention is 2 weeks). **Shape:** trigger a tier from **its own last successful backup** rather than a clock — test tier X once its newest archive is ~24 h old and not yet proven. Self-aligning when a backup runs late, and it naturally stops testing a tier that has stopped producing backups (today an interval keeps re-testing the same stale archive). Needs the scheduler to consult per-tier backup times (already available: `BackupRunner.NewestArchiveTime`, R-84) alongside `RestoreTestState` (R-85). **Interim in force:** `restore_test_cadence_seconds = 302400` (3.5 d) on both boxes — oldest-first over two tiers lands each tier ~weekly, which matches the cadence half of the ruling but NOT the alignment half. |
|
||||
| R-87 | **The restic (app-data offsite) tier is NEVER restore-tested** | M | idea — surfaced 2026-07-27 while closing R-85 | **R-85 covers whole-guest vzdump tiers only** (`local`, `felhom-pbs`); the agent has no restic surface at all. restic is the CONTROLLER's app-data offsite backup to the Hetzner Storage Box, a separate mechanism — so the tier that is arguably most important to a customer is the one nothing verifies. It is the only tier that survives losing the box **and** carries their actual app data: the whole-guest snapshot deliberately excludes the bind-mounted data drives (`/mnt/felhom-drives`). Restore code exists and has been exercised BY HAND (the immich destroy-and-recover drill, PROVEN-LIVE), but nothing tests it unattended — **exactly the state PBS was in before R-85: it works when someone tries it, and nobody would know if it stopped.** Needs its own design: a restic restore-test is controller-side, has no scratch-guest analogue, and would verify into a scratch dir rather than a booted guest, so R-85's machinery does not transfer. |
|
||||
| R-88 | **An UNREACHABLE backup target reads as "no backup exists" → the agent fires a doomed backup at it** | S | **Part 1 SHIPPED (controller v0.176.0, 2026-07-27); Part 2 OPEN (agent wire change)** — **Part 1** added the failure breaker: consecutive failures tracked per TARGET, backoff `15m→30m→1h→2h→4h` capped, reset on success, gating the QUIESCE rather than the backup so no stack is stopped for a doomed attempt. Never permanent (the cap bounds the retry interval — a latched breaker is a silent backup outage), never global (a broken offsite tier cannot suppress a healthy local one), never applied to `TriggerNow`, and `stillRunning` is not a failure. In-memory on purpose: forgetting a backoff costs one attempt, persisting one could outlive the fix. **Part 2 is agent-side and needs a wire change** — see the P0 finding below. Live on both boxes; the loop reproduction is UNIT-TEST ONLY (reproducing it live would need PBS taken down on ep0, which has no swap). | **The R-81 class (absence-is-not-failure), reappearing one layer down in the AGENT's due-check.** R-84 replaced the in-memory `Store` with storage ground truth (`NewestArchiveTime`), and fails safe on a read error by *falling back to the in-memory record* — "never fake freshness, never suppress". That fallback is correct in isolation but wrong in the one composition where it actually runs: **a cold store plus an unreachable target.** After a restart the in-memory record is empty, so the fallback answers *no backup has ever been recorded* → tier due → the agent fires a backup at the very storage it just failed to read. **Observed:** `felhom-agent` restarted on demo-felhom at 07:02:57 UTC; `local-api: could not read the backup storage for the due-check — falling back to the in-memory record` (vmid=9201 target=felhom-pbs); 11 s later `vzdump 9201 → felhom-pbs` failed with `could not activate storage 'felhom-pbs' … Can't connect to 10.77.0.1:8007 (Connection refused)`. The offsite PBS service was down at the time — so the trigger was real, not synthetic. **NOT bounded, and it also DISABLES the business-hours protection — an availability fault, not a logging nuisance.** The first reading ("fails in ~0 s, so the cost is one spurious event per restart") was wrong on both halves, and a second reading ("it stopped after three tries, so something limits it") was wrong too — the live run stopped after three only because the storage came back. **Verified in code: there is no limiter.** `internal/quiesce` holds no failure counter, no backoff, no breaker, no attempt budget; `runOnce` keeps no state between ticks, and the driver is a plain 5-minute `time.Ticker` (`quiesce.go:149`, default `quiesce.go:107-109`). Had the target stayed down, the loop would have continued indefinitely.
|
||||
|
||||
@@ -175,7 +175,7 @@ Self-resolves the moment the target answers (the storage read succeeds, sees the
|
||||
| R-162 | **`docker diff` is the gate's only witness and its failure mode is quiet** | XS | WATCHING — 2026-08-02 | A limitation, not a defect. The gate's power is `docker diff` excluding mounted paths; on a driver where it is unsupported or lies, the gate degrades to mount-occupancy + writability **and would not say so**. It fails closed (the canary self-test stops reporting BROKEN and the gate then refuses to report), but the message blames the prober rather than the driver. Revisit only if a non-overlay driver ships |
|
||||
| R-163 | **`mp1` is RETENTION, not staging — and it is sized as if it were neither** | M | **WAITING-ON-OPERATOR** — the ratio is a tier-sizing ruling | Settled by reading: a unit is the KEPT copy on the app's **own** drive (`backup.go:245-255`), falling back to the system SSD for driveless apps — *"the SSD-only system-data fallback"* (`appbackup/paths.go:26-27`); **no post-copy deletion**, the only prune is F5 residue-on-old-drives (`backup.go:1053-1112`). So `mp1` **20 G** retains every driveless app's unit while `mp0` permits **50 G** of volumes, and a DB app's unit is up to **~2×** its data (measured 21.1 GB → 40.2 GB). `--sysdata-grow` defaults to 0 (`felhom-agent/.../main.go:178`), not derived from the drive; demo-hp's real guest ships 50G/20G. **`mp1` gates the whole chain** — Tier-2 mirrors the unit *"(always)"* (`tier2.go:302,368`) and Tier-3 carries it. Bounded: a unit is volume tars + DB dumps only, **never `mp8` userdata** (`recovery_unit.go:20-25`). **No number proposed.** → `07-backup-architecture.md` §7.5 |
|
||||
| R-164 | **C2's chain — the DB volume tar cannot be dropped until a sound dump predicate exists** | S | BLOCKED — on the predicate (2026-08-02) | The unit holds a volume tar **and** a SQL dump and the restore uses both: the dump is authoritative and replayed after the tar so it WINS (F17), with only the DB service up (R-47) — `restore_unit.go:262-266`. Dropping the DB tar would halve DB-app units and close R-127(b)'s initdb-skip trap. **The obvious gate is dead, measured:** `ValidateDump`'s empty-`accounts` warning was **correct** (the DB truly had 0 rows; seeding one stopped the warning and put the row in the dump) — but **a fresh appliance legitimately has zero accounts**, so gating on it blocks every new customer's first backup. Order: sound predicate (dump vs **live** per-table counts) → warn→gate → tar-drop. Pairs with **R-127** |
|
||||
| R-90 | **ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86** | S | BLOCKED on Hetzner CX33 availability (operator, 2026-07-27) | Verified read-only: `Swap: 0B`, nothing in `/etc/fstab`, no swap unit, no `/swapfile` — **swap was never added.** On 2026-07-27 06:58:12 a global OOM fired while `proxmox-backup-proxy` held a 3.2 GB peak serving a 14.46 GB restore-test read and the datastore-migration `rsync` held 1.9 GB; the kernel killed the rsync and PBS stopped serving for ~15 minutes. That outage is what exposed **R-88** downstream. Still exposed today: 3.8 GB, no swap, one kill on record. **This gates R-86** — backup-aligned restore-testing puts a multi-GB read on this box *on a schedule*, so the headroom question must be settled before that lands. Interim lever if CX33 stays unavailable: a swapfile, which needs no console |
|
||||
| R-90 | ~~**ep0 has 3.8 GB and NO swap — it OOMs under a restore-test, and that gates R-86**~~ | S | **CLOSED 2026-08-03 — the operator rescaled ep0 to a CX33.** MEASURED on the box, not read from an invoice: `Mem: 7757` MB (**8 GB**, was 3.8), `nproc` **4**, and the 4 GiB swapfile added 2026-07-27 **survived the resize** and is active (checked because a resize is a stop/start, so "still there" was an assumption until measured). **The 40 GB local disk is UNCHANGED** — a CPU/RAM resize only — so no disk figure in any runbook needed correcting. Unblocks R-86 | Runbooks corrected; SPIKE/evidence documents ANNOTATED not revised. `target-selection.md`'s "D-d did not name ep0 either way" stays open — it is the operator's |
|
||||
| R-91 | **The old 13 GB datastore copy is still on ep0's root disk** | XS | WATCHING — gated on demo-felhom's first post-migration PBS backup | The datastore moved to a Hetzner Cloud Volume on 2026-07-27 (`/dev/sdb`, 100 GiB, attached 06:29:40 UTC, now `/mnt/pbs-datastore`, 13 G used of 98 G). The pre-migration copy survives at **`/srv/pbs-felhom`, 13 G**, on `/` (38 G total, 16 G used, 21 G free). **Do not delete yet:** demo-hp has landed two post-migration snapshots (07-27 08:25:47Z, 09:37:29Z) but **demo-felhom's newest is 2026-07-26T12:21:48Z — before the migration**, so the new volume has not yet proven a write for that namespace. Delete once it has. **Doc drift to fix in the same commit:** `CONTEXT.md:1018` still records the datastore at `/srv/pbs-felhom` |
|
||||
| R-92 | **The hub's PBS-DR gauge is 0.1 GB-granular, so small deltas are unverifiable** | XS | idea — 2026-07-27 | The PBS-DR box card rounds to 0.1 GB, which is coarser than the changes an operator wants to confirm after a prune or a GC — a successful prune of a small namespace moves the number by less than one displayed digit, so the UI cannot distinguish "it worked" from "nothing happened". Cosmetic today; it becomes load-bearing the moment retention (R-89) is customer-visible and someone needs to see that a policy change took effect |
|
||||
| R-93 | **`drill-r50` is both a blocked customer and the only drift fixture** | XS | idea — 2026-07-27 | The drill customer is blocked in the hub (so it stops alarming) yet it is also the only record exercising the endpoint-drift path R-77 added. Blocking hides it from `GetActiveCustomerIDs`, so the fixture it provides is silently inert — a monitor with no live subject reads exactly like a monitor that passes. Decide: retire it and build a synthetic fixture, or unblock it and silence per-customer instead (the operator has a per-alert silencing feature planned). Related to the R-50 drill VM, now shut down |
|
||||
|
||||
@@ -3,7 +3,8 @@
|
||||
**Class:** supervised operational run. **No repo version bump** — the only commits are this record
|
||||
and the capacity note. **Nothing was deleted.**
|
||||
|
||||
**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner CX23, Nuremberg.
|
||||
**Host:** `ep0` / `felhom-hetzner`, `167.233.158.164`, Hetzner **CX33 (4 vCPU / 8 GB RAM)**, Nuremberg.
|
||||
> **Rescaled 2026-08-03** from the CX23 (2 vCPU / 3.8 GB) this runbook was written against. **The 40 GB local disk did NOT change** — this was a CPU/RAM resize — so every disk figure below still stands. The 4 GiB swapfile added on 2026-07-27 survived the resize.
|
||||
**Datastore moved:** `felhom-offsite`, `/srv/pbs-felhom` → **`/mnt/pbs-datastore`** (name unchanged).
|
||||
**Window:** 06:58 → 07:19 UTC. PBS down 07:00 → 07:17 UTC.
|
||||
|
||||
|
||||
@@ -229,7 +229,9 @@ as the hub 400ing an unknown event type. `verify-new` verifies each snapshot as
|
||||
`keep-last 2` that covers essentially the whole datastore and turns a dead check live, for a few
|
||||
minutes of ep0 CPU per weekly backup.
|
||||
|
||||
> Watch item: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
|
||||
> Watch item (**superseded 2026-08-03**: ep0 is now a **CX33, 8 GB RAM**, and it HAS a 4 GiB swapfile
|
||||
> which survived the resize — so the pressure below is much reduced, though the shape of the concern
|
||||
> stands). As written: ep0 is a 3.7 GB CX23 with **no swap**, and inline verification runs within the backup
|
||||
> window. Today's full forced verify completed fine (~250 MiB/s, 0 errors), but see
|
||||
> `RUNBOOK-ep0-datastore-volume-2026-07-27.md` for the rsync OOM on this same box.
|
||||
|
||||
@@ -307,7 +309,7 @@ Untouched. Rollback remains a two-line `datastore.cfg` revert. Volume: 98 G, 13
|
||||
watching: it is the only thing that reclaims chunks, and nothing has ever exercised it here.
|
||||
4. **Hub PBS-DR gauge granularity** — 0.1 GB steps mean routine incremental backups are invisible to
|
||||
it. Not a fault, but it cannot be used as write-proof evidence for small deltas.
|
||||
5. **ep0 has no swap** (3.7 GB CX23) — see the volume runbook's OOM.
|
||||
5. ~~**ep0 has no swap** (3.7 GB CX23)~~ — **corrected 2026-08-03: ep0 is a CX33 with 8 GB RAM and an active 4 GiB swapfile.** See the volume runbook's OOM for the original incident.
|
||||
|
||||
## 11. Observations
|
||||
|
||||
|
||||
@@ -5,7 +5,8 @@
|
||||
> firewall, and the hub-driven `felhom-peersync` reconcile surface. Re-running it on a fresh VM
|
||||
> re-creates the endpoint from nothing (that is the DR story, step 8).
|
||||
>
|
||||
> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23, Debian 13,
|
||||
> **Validated:** 2026-07-03 on the dev/test endpoint `felhom-hetzner` (Hetzner CX23 **at the time — rescaled
|
||||
> to a CX33, 4 vCPU / 8 GB RAM, on 2026-08-03; the 40 GB local disk is unchanged**, Debian 13,
|
||||
> `167.233.158.164` / `2a01:4f8:1c16:7aa1::1`) with hub v0.32.0. The production endpoint is a
|
||||
> later re-run of this runbook on a production VM.
|
||||
>
|
||||
@@ -31,7 +32,7 @@ Parameters used throughout (adjust for a new endpoint):
|
||||
points at nothing (live-run finding). Home-resolver propagation can lag public DNS by
|
||||
minutes — a client-side `wg-quick up` that fails to resolve right after record creation
|
||||
just needs a retry.
|
||||
- [ ] Sanity: `ssh root@167.233.158.164 hostname` → `felhom-hetzner` (the throwaway CX23), not
|
||||
- [ ] Sanity: `ssh root@167.233.158.164 hostname` → `felhom-hetzner` (the throwaway box, **CX33 since 2026-08-03**), not
|
||||
any production box.
|
||||
|
||||
## 1. Base (on the box, as root)
|
||||
|
||||
@@ -98,7 +98,8 @@ enrolled host. No access route from DooPlex, and nothing here needs one.
|
||||
### `ep0` (`felhom-hetzner`, `ep0.felhom.eu`) + the Hetzner Storage Boxes — **not protected by D-d; not scratch either**
|
||||
|
||||
Reads are fine. It is the **offsite of last resort** (PBS-DR datastore, WireGuard hub, operator OOB
|
||||
path) and RAM-constrained (3.8 GB, R-90) so a large restore can OOM it. Do not delete datastores, prune
|
||||
path) and — until 2026-08-03 — RAM-constrained (3.8 GB, R-90); it is now a **CX33 with 8 GB RAM and a
|
||||
4 GiB swapfile**, which is what closed R-90. A very large restore is still worth watching. Do not delete datastores, prune
|
||||
jobs, tunnel config or nftables rules; never a drill target. The Storage Boxes hold the restic copy —
|
||||
customer documents and photos, on a credential that can still delete (R-95).
|
||||
**Access: `ssh root@167.233.158.164` from DooPlex** — *not* `felhom-pve → 10.77.0.1`, the route that
|
||||
|
||||
@@ -139,7 +139,7 @@ is not reachable, for two reasons that are each **already-recorded deliberate po
|
||||
`endpoint_id` only; per-endpoint allocation is an explicitly deferred arc (`hub/README.md:260`).
|
||||
|
||||
So the only two configurations are: *DR tier on* → the campaign's PBS traffic lands on **ep0**, which is
|
||||
Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90) and fenced by §3 — or *DR tier off* →
|
||||
Tier 2, the offsite of last resort, RAM-constrained (3.8 GB, R-90 — **note added 2026-08-03: ep0 has since been rescaled to a CX33 with 8 GB RAM; this journal records what was true when it was written and is deliberately not revised**) and fenced by §3 — or *DR tier off* →
|
||||
no Tier 3 at all. **Chosen: DR tier OFF, offsite OFF**, which is the only option §3 permits.
|
||||
|
||||
Consequence, stated plainly rather than discovered later: the campaign touches **neither ep0 nor the
|
||||
|
||||
Reference in new issue
Block a user