Compare commits
2
Commits
c2c1fb48dc
...
36f8630020
| Author | SHA1 | Date | |
|---|---|---|---|
  adminandClaude Opus 5
|
36f8630020 | ||
|
adminandClaude Opus 5
|
f5c9411e5e |
No files matched your search
@@ -1,186 +1,150 @@
|
||||
# REPORT — five process-domain skills + check_skills.py (2026-08-25)
|
||||
# REPORT — R-331: the operator Backup card said every customer had no backups
|
||||
|
||||
Documentation and agent-configuration only. No Go code written or changed, nothing built, nothing
|
||||
deployed. Only DooPlex was touched, and on it only this repo's working tree and `~/.claude/skills/`.
|
||||
**Hub v0.109.0 (with controller v0.225.0) · 2026-08-30**
|
||||
|
||||
## 1. Confirmed baseline
|
||||
---
|
||||
|
||||
`felhom.eu` @ `ebdc04601d37db4c732245a6a9df777bd28fe6df` (2026-08-23T14:12:23+02:00). Clean tree,
|
||||
`HEAD == origin/main`, verified before any edit. Matches the baseline in the task file.
|
||||
## 1. What was wrong
|
||||
|
||||
## 2. Files created / modified
|
||||
|
||||
Created:
|
||||
- `/mnt/5_hdd/felhom.eu/git/felhom.eu/skills/felhom-evidence/SKILL.md`
|
||||
- `/mnt/5_hdd/felhom.eu/git/felhom.eu/skills/felhom-diagnosis/SKILL.md`
|
||||
- `/mnt/5_hdd/felhom.eu/git/felhom.eu/skills/felhom-plain-language/SKILL.md`
|
||||
- `/mnt/5_hdd/felhom.eu/git/felhom.eu/skills/felhom-handoff/SKILL.md`
|
||||
- `/mnt/5_hdd/felhom.eu/git/felhom.eu/skills/felhom-doc-authoring/SKILL.md`
|
||||
- `/mnt/5_hdd/felhom.eu/git/felhom.eu/skills/SOURCES.md`
|
||||
- `/mnt/5_hdd/felhom.eu/git/felhom.eu/scripts/check_skills.py`
|
||||
|
||||
Modified:
|
||||
- `/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/runbooks/workspace-CLAUDE.md` (one line: stale
|
||||
count "the four Felhom skills" → "the Felhom skills"; validator named)
|
||||
- `/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/backlog/OPEN-ITEMS.md` (three rows)
|
||||
- `/mnt/5_hdd/felhom.eu/git/felhom.eu/scripts/CHANGELOG.md`
|
||||
- `/mnt/5_hdd/felhom.eu/git/felhom.eu/CONTEXT.md`
|
||||
- `/mnt/5_hdd/felhom.eu/git/felhom.eu/REPORT.md` (this file)
|
||||
|
||||
**DEVIATION FROM THE TASK FILE §6.3.** It asked for an entry in `felhom.eu/CHANGELOG.md`. **That file
|
||||
does not exist** — this repo keeps per-area changelogs (`scripts/`, `website/`, `hub/`), as
|
||||
`CONTEXT.md`'s own header states. The entry went to `scripts/CHANGELOG.md`, the closest owning area,
|
||||
since the checker lives there. `skills/` has no changelog of its own and none was created.
|
||||
|
||||
## 3. Commit hashes pushed to `main`
|
||||
|
||||
`c30430c530a2fcf3b0aca03c6ef5d266e30b8314` — *skills: five process-domain skills + check_skills.py*
|
||||
|
||||
Pushed with **`git push --no-verify`**; the reason is stated in §12 item 6, as the hook requires.
|
||||
Baseline `ebdc046` → `c30430c`, one commit, no branch.
|
||||
|
||||
## 4. Check-script results and the red-proof
|
||||
|
||||
`python3 scripts/check_skills.py` → **exit 0**, nine skills:
|
||||
The hub customer page's **Backup** card read, for **every customer, indefinitely**:
|
||||
|
||||
```
|
||||
OK felhom-app-catalog 126 lines live-linked
|
||||
OK felhom-build-deploy 179 lines live-linked
|
||||
OK felhom-diagnosis 90 lines live-linked
|
||||
OK felhom-doc-authoring 86 lines live-linked
|
||||
OK felhom-evidence 90 lines live-linked
|
||||
OK felhom-handoff 69 lines live-linked
|
||||
OK felhom-plain-language 55 lines live-linked
|
||||
OK felhom-testing 99 lines live-linked
|
||||
OK felhom-ui-design 71 lines live-linked
|
||||
WARN felhom-build-deploy: 179 lines, OVER the 150-line limit — grandfathered (R-394 — 179 lines
|
||||
when the limit was introduced; trim is a scoped session)
|
||||
|
||||
PASS: 9 skill(s) well formed.
|
||||
Enabled Yes Snapshots 0
|
||||
Repo Size 0 MB Integrity Unknown
|
||||
```
|
||||
|
||||
**RED-PROOF — RUN, AND SEEN FAILING.** It did not pass without having been seen to fail.
|
||||
Measured on `demo-hp` 2026-08-30, at which moment the truth was:
|
||||
|
||||
1. `grep -v '^description:'` removed the `description` line from `skills/felhom-evidence/SKILL.md`.
|
||||
2. Re-ran → **exit 1**. Exact failure message seen:
|
||||
`- /mnt/5_hdd/felhom.eu/git/felhom.eu/skills/felhom-evidence/SKILL.md: frontmatter field 'description' is missing or empty`
|
||||
3. Restored from a scratchpad copy → exit 0; `git status --porcelain` showed no modified tracked
|
||||
file, only the intended new untracked paths.
|
||||
|
||||
This is a **positive control**: the checker was shown finding a planted fault before its silence on
|
||||
the other eight was read as "all well formed".
|
||||
|
||||
## 5. Install output, both runs, and the samefile confirmation
|
||||
|
||||
Run 1 — five newly linked, four already installed, no FAIL, no copy-mode note, **exit 0**:
|
||||
```
|
||||
OK felhom-app-catalog symlink (already installed, live-linked to repo)
|
||||
OK felhom-build-deploy symlink (already installed, live-linked to repo)
|
||||
OK felhom-diagnosis symlink -> /mnt/5_hdd/felhom.eu/git/felhom.eu/skills/felhom-diagnosis
|
||||
OK felhom-doc-authoring symlink -> /mnt/5_hdd/felhom.eu/git/felhom.eu/skills/felhom-doc-authoring
|
||||
OK felhom-evidence symlink -> /mnt/5_hdd/felhom.eu/git/felhom.eu/skills/felhom-evidence
|
||||
OK felhom-handoff symlink -> /mnt/5_hdd/felhom.eu/git/felhom.eu/skills/felhom-handoff
|
||||
OK felhom-plain-language symlink -> /mnt/5_hdd/felhom.eu/git/felhom.eu/skills/felhom-plain-language
|
||||
OK felhom-testing symlink (already installed, live-linked to repo)
|
||||
OK felhom-ui-design symlink (already installed, live-linked to repo)
|
||||
```
|
||||
Run 2 — all nine `(already installed, live-linked to repo)`, **exit 0**. Idempotent.
|
||||
|
||||
`os.path.samefile` between `~/.claude/skills/<name>/SKILL.md` and the repo file, proving a symlink
|
||||
and not a copy:
|
||||
```
|
||||
felhom-diagnosis samefile=True islink(dir)=True
|
||||
felhom-doc-authoring samefile=True islink(dir)=True
|
||||
felhom-evidence samefile=True islink(dir)=True
|
||||
felhom-handoff samefile=True islink(dir)=True
|
||||
felhom-plain-language samefile=True islink(dir)=True
|
||||
```
|
||||
|
||||
## 6. Line count of each new SKILL.md (limit: under 150)
|
||||
|
||||
| Skill | Lines |
|
||||
| source | value |
|
||||
|---|---|
|
||||
| `felhom-evidence` | 90 |
|
||||
| `felhom-diagnosis` | 90 |
|
||||
| `felhom-doc-authoring` | 86 |
|
||||
| `felhom-handoff` | 69 |
|
||||
| `felhom-plain-language` | 55 |
|
||||
| the box's own `settings.json` | `snapshot_count: 67, repo_size_bytes: 140829678, stats_known: true` |
|
||||
| that night's controller log | `[offbox] backup OK: 8 app(s) backed up, 67 snapshot(s), 2m14s` |
|
||||
| **this hub's own Offsite page** | `0.1 GB` used of a `50 GB` quota — read from the same stored report |
|
||||
|
||||
All five under the limit. None had to be split.
|
||||
**A card that reads "no backups" over a working backup is worse than no card.** It is the R-88
|
||||
direction of failure — degrading to *no backup* rather than to *unknown* — on the one screen an
|
||||
operator consults to answer "is this customer protected?".
|
||||
|
||||
## 7. NOT YET VALIDATED — awaiting the operator
|
||||
## 2. Root cause
|
||||
|
||||
**Whether each skill FIRES is behavioural and was NOT proven here.** The checker proves a skill is
|
||||
loadable; it cannot prove the model reaches for it. That is decided by the `description` wording and
|
||||
no mechanical check settles it.
|
||||
The card rendered the report's **`backup`** object. Its `snapshot_count`, `repo_size_mb` and
|
||||
`integrity_ok` fields have had **no producer** since disk-tier restic moved to the host agent (slice
|
||||
8C) — the controller's `buildBackupReport` leaves them zero *deliberately* and says so in a comment.
|
||||
The zeros were correct values for dead fields, rendered as if live.
|
||||
|
||||
One observable WAS obtained and is stated for what it is, not more: after installation all five
|
||||
appeared in this session's own skill listing with their descriptions intact. That proves the harness
|
||||
discovered and parsed them. **It does not prove any of them fires on a real prompt.**
|
||||
**The data was never missing.** The live numbers ride in the report's **`offsite`** object, which this
|
||||
package **already** reads for the Offsite page (`offsiteUsageBytes`) and which `monitor.OffsiteChecker`
|
||||
**already** drives fill and staleness alarms from. That the Offsite page rendered demo-hp's real usage
|
||||
from the same stored report, at the same moment the Backup card said `0 MB`, is the proof the bytes
|
||||
were arriving. This is a **render fix over an existing feed**, not a new pipeline.
|
||||
|
||||
The operator's check, in a **fresh** Claude Code session (one probe phrase each):
|
||||
## 3. Why it was not a one-line template swap
|
||||
|
||||
| Skill | Probe phrase to type | Pass looks like |
|
||||
|---|---|---|
|
||||
| `felhom-evidence` | *"Are we sure the stopped stack is intentional?"* | the reply grades the claim into one of the five tiers; **fails** if it answers ungraded, or uses "clearly" / "obviously" |
|
||||
| `felhom-diagnosis` | *"The controller stopped working after a restart, why?"* | it asks for or builds a failing command FIRST and refuses to theorise; **fails** if it starts reading code and proposing causes |
|
||||
| `felhom-plain-language` | *"wait, what"* after any technical answer | it rewrites the whole answer shorter; **fails** if it defends the original or asks which part was unclear |
|
||||
| `felhom-handoff` | *"I need to stop, pick this up later"* | it writes a note to `/tmp/felhom-handoff-<slug>.md`; **fails** if the plan is only in the conversation |
|
||||
| `felhom-doc-authoring` | *"write a skill for X"* | it treats the `description` as the pointer that decides reachability, and holds under 150 lines; **fails** if it writes the body first and the frontmatter as an afterthought |
|
||||
`snapshot_count: 0` means two opposite things — *this repository holds nothing* and *nobody has ever
|
||||
measured this repository*. **R-225 measured that confusion one layer down**: a rebuilt box rendered
|
||||
„Tarolo meret · 0 pillanatkep" over a store that really held snapshot `f3d9cd67`, and the controller's
|
||||
`StatsKnown` fixed it there. It was never on the wire, so rendering the count without it would have
|
||||
**moved R-225 up to the hub instead of fixing anything**. Controller v0.225.0 now forwards
|
||||
`stats_known`.
|
||||
|
||||
## 8. Evidence
|
||||
## 4. What changed
|
||||
|
||||
N/A. No phase ran on any machine other than DooPlex, nothing was reverted, and no snapshot was
|
||||
restored. The one temporary mutation (the red-proof) was restored in the same step that made it, and
|
||||
its output is quoted in §4 rather than left on disk.
|
||||
`hub/internal/web/backup_card.go` builds a typed `backupCardView` — resolved in Go, because the card's
|
||||
whole subject is a distinction a template `{{if}}` chain over `map[string]interface{}` float64s cannot
|
||||
keep:
|
||||
|
||||
## 9. Teardown
|
||||
|
||||
N/A — this run provisioned nothing. No guest, no VM, no rig, no scratch host.
|
||||
|
||||
## 10. Register
|
||||
|
||||
| Row | Title |
|
||||
| report state | card shows |
|
||||
|---|---|
|
||||
| **R-392** | No architecture document covers the two-AI workflow |
|
||||
| **R-393** | Decision-log skill for unattended runs — deferred, with the reason |
|
||||
| **R-394** | `felhom-build-deploy/SKILL.md` is 179 lines, over the limit its own repo now enforces |
|
||||
| no `offsite` object at all | "No off-site data reported" — **and says explicitly this is not the same as "no backups"** |
|
||||
| `enabled:false` + declared `state` | the blocker by name (`needs_credential`) — a different operator action from "not enabled" |
|
||||
| enabled, `stats_known:false` | **—**, plus "never been measured". Never `0` |
|
||||
| enabled, `stats_known:true` | the real count and size, **including a real `0`** — measured empty is knowledge |
|
||||
|
||||
`documentation/backlog/OPEN-ITEMS.md` — **335,212 bytes** after the three rows (331,024 before).
|
||||
No rows were closed this session, so nothing was compressed or rehomed.
|
||||
**A pre-v0.225.0 controller sends no `stats_known`, which unmarshals to false → "unknown".** That is
|
||||
the fail-safe direction: upgrading the hub ahead of the fleet must not tell the operator that every
|
||||
un-upgraded customer has zero backups. Pinned by a test.
|
||||
|
||||
## 11. Capability map
|
||||
**The Integrity row is deleted, not re-sourced.** Nothing produces it: the controller runs no integrity
|
||||
check, and `NotifyIntegrityOK` / `NotifyIntegrityFailed` exist and are **called from nowhere**. A row
|
||||
that can only ever read "Unknown" is not information, and one that could read "OK" from an unwritten
|
||||
field would be a lie.
|
||||
|
||||
**No product capability changed and no row in `documentation/architecture/00-capability-map.md` was
|
||||
touched.** This task changes no product behaviour: no binary, no endpoint, no template, no guest, no
|
||||
host. `documentation/backlog/ROADMAP.md` is likewise unchanged — this is not product work.
|
||||
`fmtBytesAuto` is new rather than reusing `fmtBytesGB`: that one is fixed at GB because it renders
|
||||
against GB quotas, and it turns demo-hp's real 140 829 678 bytes into `0.1 GB` — which on a card whose
|
||||
entire defect was under-reporting a real backup reads as "nearly nothing".
|
||||
|
||||
## 12. Observations
|
||||
## 5. Tests and the red-proof
|
||||
|
||||
1. **`felhom-build-deploy/SKILL.md` is 179 lines, over the 150-line limit this task introduced.**
|
||||
Found by the new checker on its first run. Not edited — the task scoped the four pre-existing
|
||||
skills out — and made a named single-entry `GRANDFATHERED` exception printed as a WARN on every
|
||||
run so it cannot fade. **FILED: R-394**
|
||||
2. **The task file specified a check on every `skills/*/SKILL.md` that its own "do not edit the
|
||||
existing skills" rule made unsatisfiable.** The two constraints met on `felhom-build-deploy`.
|
||||
Resolved by naming the exception rather than weakening the rule or editing the file; both
|
||||
alternatives would have hidden a real finding. **FILED: R-394** (same row — it is the same fact,
|
||||
and a second row would be the duplicate the one-register ruling exists to prevent).
|
||||
3. **No architecture document covers the agent-tooling layer.** Found by trying to fill the task
|
||||
template's owning-document field and being unable to. **FILED: R-392**
|
||||
4. **A sixth skill — a decision log for unattended runs — was considered and held back**, because it
|
||||
needs a helper script and a storage convention rather than a text file. **FILED: R-393**
|
||||
5. **`felhom.eu` has no root `CHANGELOG.md`**, though the task file and the workspace `CLAUDE.md`
|
||||
both refer to one for this repo. This repo deliberately keeps per-area changelogs, which
|
||||
`CONTEXT.md`'s header states. **NOT-A-FINDING: the per-area convention is deliberate and
|
||||
documented in `CONTEXT.md`; the entry went to `scripts/CHANGELOG.md` and the deviation is stated
|
||||
in §2. Nothing is missing — only the task file's assumption was wrong.**
|
||||
6. **The push used `git push --no-verify`, and this is the required statement of that.** The
|
||||
pre-push hook's `--fast` gate run convicted on **due-checks**, not on anything this session
|
||||
changed: **R-341's dated check came due 2026-08-25**, the day of this session. The DUE-CHECKS
|
||||
block was not touched here (`git diff` on it is empty), and the other eleven gates — including
|
||||
`observations`, `one-register` and `instructions` — all passed. Taking R-341's measurement is a
|
||||
live-host systemd uptime reading, a different task with its own preconditions, and moving its date
|
||||
to clear the gate would have silently deferred someone else's check to make this push convenient.
|
||||
**FILED: R-341** — the row already exists and is the correct home; a new row would be the
|
||||
duplicate the one-register ruling exists to prevent.
|
||||
`r331_backup_card_test.go` asserts the **rendered page**, using demo-hp's real reported values, so a
|
||||
regression fails against the same numbers the defect was measured against. **The defect lived in the
|
||||
template's choice of source object, so a test one layer below it would have been green against the
|
||||
shipped bug** — which is why these drive `handleCustomerUnified` and grep the HTML.
|
||||
|
||||
**RED-PROOF (run 2026-08-30):** restoring the pre-fix card markup fails all four tests —
|
||||
`the rendered Backup card does not contain demo-hp's real snapshot count (67)`,
|
||||
`the card does not carry the real repository size (134.3 MB ...)`,
|
||||
`the card still shows an Integrity row`, plus every branch of the three-way ruling. Restored
|
||||
immediately; `git diff` clean.
|
||||
|
||||
**Green gate:** `go build ./... && go vet ./... && go test ./...` in `hub/` — 18 packages, rc 0.
|
||||
|
||||
## 6. Deployment
|
||||
|
||||
PENDING at the time of writing — see the follow-up commit. The two halves ship independently and in
|
||||
either order: the hub renders "unknown" for any box still on controller 0.224.0, which is correct
|
||||
rather than wrong.
|
||||
|
||||
## 7. The push bypassed a gate, deliberately, and here is the declaration
|
||||
|
||||
**`git push --no-verify` was used for this change.** `repo_gates.py`'s `golden-currency` gate was
|
||||
CONVICTED and it was RIGHT: controller **v0.224.0** and **v0.225.0** are released and the newest golden
|
||||
bake carries **0.223.0**, so a machine installed right now receives neither fix.
|
||||
|
||||
**This is a BYPASS, not a waiver.** The gate offers a waiver only for a release that *deliberately needs
|
||||
no golden*; these need one. **The operator was asked and ruled bypass-now-bake-later**, on the stated
|
||||
ground that neither fix bites a day-0 box — R-330 is a nightly false alarm about apps a new box has not
|
||||
installed yet, R-331 is a hub-side display over backups a new box has not taken yet — and both arrive by
|
||||
self-update afterwards. That ground is recorded on R-242 precisely because it is the thing to re-check:
|
||||
**it does not extend to a release that changes first-boot behaviour.**
|
||||
|
||||
**A golden carrying 0.225.0 is OWED** (`RUNBOOK-manual-build.md` §4.1; the vouch is a three-field change,
|
||||
`MinAgent 0.129.0`). This is the **fourth** bypass of this gate, and the gap it names is now two releases
|
||||
wide rather than one.
|
||||
|
||||
**The other failing gate was fixed, not bypassed.** `due-checks` was red on R-341's `+7 d` measurement,
|
||||
five days overdue. It was **taken** during this session — see §8.
|
||||
|
||||
## 8. R-341's overdue check was taken, and its premise did not survive
|
||||
|
||||
Unrelated to R-331; it blocked the same push, so it was done rather than deferred. Evidence:
|
||||
`documentation/audits/evidence-r341-plus7d-2026-08-30/step1-fd-and-sockets.txt`.
|
||||
|
||||
**Precondition passed**, which is what makes the reading interpretable: `proxmox-backup-proxy` still
|
||||
`MainPID 551655`, `ps -o lstart=` still `2026-08-18 09:51:04`, `NRestarts=0` — the same proxy generation
|
||||
as t0, so nothing restarted and re-based the count. (The anchor is `ps`, not `ActiveEnterTimestamp`,
|
||||
which reads 03:54:54Z here — R-346's trap, avoided.)
|
||||
|
||||
**Result: fd = 17.** Not 17 more — seventeen total, exactly the documented baseline, against **405** at
|
||||
the first check on 2026-08-20. The socket histogram holds **one LISTEN and nothing else**: ESTAB 0,
|
||||
CLOSE-WAIT 0.
|
||||
|
||||
**The verdict is "unanswerable", not "the upgrade fixed it".** R-341 asks whether the PBS 4.2.5-1
|
||||
upgrade changed the fd slope. Inside this interval **we removed the leak ourselves** (R-344, agent
|
||||
0.130.0, now live on both boxes). A slope of ~0 measures our fix, not the upgrade — and reading it the
|
||||
other way would credit a changelog that was read in advance and found to contain no such mechanism. The
|
||||
perturbation pre-registered for this window was Phase C at ~3%; the actual perturbation was the removal
|
||||
of the entire phenomenon. The question is now **moot**, and the row is closed as such.
|
||||
|
||||
**What it does establish, which is worth more than the original question:** twelve days after the R-344
|
||||
fix, on the same proxy generation with no restart to hide behind, ep0 sits at baseline with zero
|
||||
established connections. The 388-descriptor accumulation has not returned, and R-336's ~323-day runway
|
||||
concern retires with it.
|
||||
|
||||
## 9. Not done, and why
|
||||
|
||||
- **No staleness verdict on the card.** `monitor.OffsiteChecker` already owns that and alarms on it. A
|
||||
second verdict over the same data is two things that can disagree — a shape this codebase has already
|
||||
paid for (`LastRun` vs `LastSuccess`, R-100).
|
||||
- **The dead `backup` fields were not removed from the controller's wire format.** Removing them would
|
||||
stop historical reports already in this hub's store from parsing, for no gain — nothing renders them
|
||||
now, and a controller-side test fails if anything starts producing them.
|
||||
@@ -65,6 +65,7 @@
|
||||
| `(*Server).hostStatus` + `hostStatusClass`/`hostStatusLabel` | hub/internal/web/hosts.go (~L16/34/48) | `(lastReport *time.Time) string` | Host liveness badge | Uses the SAME threshold as HostStalenessChecker (down = 2× stale) — never invent a second definition. |
|
||||
| `parseSQLiteTime` | hub/internal/store/store.go (~L1160) | `(s string) time.Time` | Parsing ANY timestamp read from SQLite | modernc/sqlite returns multiple formats; raw `time.Parse` will intermittently zero out. Always use this. |
|
||||
| `compareVersions` | hub/internal/web/server.go (~L571) | `(a, b string) int` | X.Y.Z comparisons in web (floor checks, update-available) | Returns 0 on parse error — unparseable compares as "equal" (see §3). |
|
||||
| `reportBackupCard` + `backupCardView` + `fmtBytesAuto` (R-331, v0.109.0) | hub/internal/web/backup_card.go | `(reportJSON string) backupCardView` / `(int64) string` | THE customer-page Backup card — everything it shows about a customer's backups | **Reads the report's `offsite` object, NEVER `backup`.** The `backup` object's `snapshot_count`/`repo_size_mb`/`integrity_ok` have had no producer since slice 8C and rendering them told every operator every customer had `Snapshots 0` (measured on demo-hp over a repo holding 67). **Always consult `StatsKnown` before believing a zero** — absent/false means "never measured", NOT "empty", and those are opposite news (R-225 measured the same confusion one layer down). Resolved in Go, not the template, because a `{{if}}` chain over `Report`'s `map[string]interface{}` float64s cannot keep the absent/zero distinction the card is entirely about. `fmtBytesAuto` scales MB/GB/TB — do NOT swap in `fmtBytesGB`, which renders demo-hp's real 140 829 678 B as `0.1 GB`. |
|
||||
|
||||
### Host views & lifecycle / offsite endpoints (v0.47.0, hub/internal/web + store)
|
||||
|
||||
@@ -174,6 +175,7 @@
|
||||
| Inline `stringData` secrets à la manifests/felhom.secret.yaml | Commits real credentials to git (healthchecks superuser pw, umami APP_SECRET/POSTGRES_PASSWORD, gitea-creds admin password still live there). | Out-of-band `kubectl create secret` + `secretKeyRef` (hub.yaml resend-api pattern; runbook documentation/runbooks/secrets.md) |
|
||||
| `kubectl apply` / `kubectl set image` on manifests/ | ArgoCD app `felhom` reverts drift on next sync; live state lies about git. | Edit manifest in git → push → ArgoCD sync (CLAUDE.md steps 3–5) |
|
||||
| `:latest` image tag in manifests | Re-push doesn't change the manifest → no redeploy; Synced/Rollback misreport. | Pinned version tag, bumped per deploy |
|
||||
| The report's `backup` object for snapshots / repo size / integrity (`snapshot_count`, `repo_size_mb`, `integrity_ok`) | **No producer since slice 8C** — the controller's `buildBackupReport` leaves all four zero deliberately and says so. Rendering them gave every customer `Snapshots 0 · Repo Size 0 MB · Integrity Unknown` indefinitely (R-331, measured on demo-hp over a repository holding 67 snapshots). `integrity_ok` is worse than stale: the controller runs no integrity check at all, so it can only ever be "Unknown" or a lie. | The report's `offsite` object (`backup.OffboxReportStatus`) via `reportBackupCard` — and check `stats_known` before trusting a zero |
|
||||
| grep/regex hunting emoji in website HTML | Windows grep false-negatives multibyte emoji (proven in D0). | `python scripts/site_gates.py` (codepoint-range check) |
|
||||
| Adding a website page without touching site_gates.py | `PAGES` list (scripts/site_gates.py ~L22) is explicit — an unlisted page is silently ungated (BOM/nav/emoji drift undetected). | Add the filename to `PAGES` in the same commit |
|
||||
|
||||
|
||||
@@ -0,0 +1,14 @@
|
||||
=== R-341 +7d check (taken late: 2026-08-30, due 2026-08-25) ===
|
||||
MainPID=551655
|
||||
ps lstart: Tue Aug 18 09:51:04 2026
|
||||
NRestarts=0
|
||||
ActiveEnterTimestamp=Tue 2026-08-18 03:54:54 UTC # R-346: NOT the anchor
|
||||
fd_count=17
|
||||
now_utc=2026-08-30T16:40:43Z
|
||||
--- ss -lnt sport :8007 ---
|
||||
State Recv-Q Send-Q Local Address:Port Peer Address:Port
|
||||
LISTEN 0 1024 *:8007 *:*
|
||||
--- socket state histogram ---
|
||||
1 LISTEN
|
||||
--- agent-side check: who holds connections now ---
|
||||
Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
|
||||
@@ -156,7 +156,7 @@ the fault was real. Full observables: `tests/campaign11-evidence-2026-08-05/jour
|
||||
| **R-240** | **A backup that covered nothing calls itself „Sikeres".** On a configured box with no app selected for off-site backup, a run reports status `ok` with the warning „Sikeres — nincs mentésre jelölt alkalmazás" — *successful* immediately beside *nothing is selected*. Measured as T4 on the final walk, 2026-08-07; flagged once before (2026-08-06) and deliberately not touched then, because the task that noticed it forbade changing that path. **It is the same rhetorical shape the project has spent a fortnight removing** — R-203's *a warning beside a success is read as a success*, R-234's *„✓ Rendben" over an app that was skipped*, R-225's *unknown rendered as zero* — one notch weaker each time, and this is the weakest and last of them. The state itself is honest and must stay `ok`: an unconfigured box reporting `incomplete` forever is its own defect, pinned by a test. **The defect is the word „Sikeres", not the verdict.** Wording such as „Nincs mentésre jelölt alkalmazás — ez a futás semmit nem mentett" says the same thing without congratulating the customer on it. | **READY** — owner Viktor |
|
||||
|
||||
|
||||
| **R-242** | **A controller release that changes customer-visible behaviour is not delivered until a golden carries it — and nothing enforces that.** R-239 is the symptom; this is the mechanism, recorded **2026-08-07** and deliberately **NOT built** (the task that found it scoped it as a record-only item). **Two releases went out without a golden and the gap was invisible until a walk measured it from the customer's side**: v0.204.0 (R-237) and v0.205.0 (R-234) were written, tested, pushed, and CHANGELOG'd, and every one of those steps passed while a machine installed that night received neither. The register said CLOSED; the fleet said otherwise. **Nothing in the release path knows a golden exists.** The version bump, the image push, the CHANGELOG entry and the register closure are all repo-local; the manifest's `golden_version` is edited by a separate operator act, in a different repo, with no link back. **Proposed shapes, cheapest first — the choice is the operator's and is not taken here.** (a) **A release-path checklist step** — one line in the controller's end-of-session checklist: *a release that changes customer-visible behaviour is not finished until a golden carries it or a register row says why not.* Costs nothing, catches nothing mechanically. (b) **A gate in `repo_gates.py`** comparing the manifest's `golden_version` against the newest released controller and FAILING (or warning) past a tolerance of one minor. Mechanical, runs on every push, and would have fired the morning after v0.204.0. (c) **A hub-side checker** — the hub already knows every box's running controller version from `/hosts` and the vouched golden from the manifest; a periodic comparison against the newest published image would catch drift the repo cannot see, including a vouch that was made and then rolled back. **Earliest catch: (b).** It fires on the push that creates the gap, before any box is installed, and it needs no live fleet. **(c) catches strictly more but only after boxes exist.** (a) is worth doing regardless because it is free. **Not built. No gate was written this session.** **⚠ IT RECURRED WITHIN A DAY, WHICH IS THE ARGUMENT FOR BUILDING IT.** Controller **v0.206.0** shipped the R-241 fixes on 2026-08-07 while the vouched golden still carried **0.205.0** — so a machine installed on the morning of 2026-08-08 would have received neither. Third occurrence of the shape in three days (R-111/R-115/R-120 are the older family). **✅ SHAPE (b) BUILT 2026-08-08 — `scripts/golden_currency_gate.py`, registered in `repo_gates.py` as gate 7.** **It was shown FAILING against that exact state before anything was baked**, which is its red-proof and the reason its own introducing push needed `--no-verify` (stated in the session report rather than worked around): `newest released controller : 0.206.0 / newest golden baked : 0.205.0 → CONVICTED`. **IT IS `--fast`, AND THAT FORCED ITS DESIGN:** both `.githooks/pre-push` AND CI run `repo_gates.py --fast`, so a non-fast gate would run in NEITHER — the R-29 census failure this runner exists to end. **THEREFORE IT CHECKS THE BAKE, NOT THE VOUCH**, because the vouched version lives only in the hub's `hub_settings` with no copy in git, and putting a copy there would create a second source of truth that can drift — a green gate over a false claim being the worst outcome available. **A bake without a vouch still passes: that half is NOT closed and stays on this row.** It also compares versions rather than behaviour, so a release changing nothing customer-visible trips it too — accepted deliberately, because judging that by hand is what failed three times and the cost of a false trip is one bake; a waiver belongs here, never in a habit of bypassing. **VOUCHED 2026-08-08 with the operator's approval** — golden `0.206.0` / sha `c85230b4…108e`; `agent_version` and `min_agent` both stayed `0.127.0`, and `wrapper_sha256` was carried through explicitly because the handler clears it when omitted. **The gate was CONVICTED before the bake and OK after it** — red→green on the same command, which is its proof that it measures something real. **⚠ THE GATE FIRED FOR REAL, 2026-08-08 — and it was right.** Controller **v0.207.0** (R-249/R-252/R-253) is released, tested and pushed, and **no golden carries it** — the newest bake is 0.206.0 — so `golden_currency_gate.py` FAILED, saying exactly the true thing: *a machine installed right now would receive v0.206.0*. **The `felhom.eu` push therefore used `git push --no-verify`, declared here, in the commit message and in the session report.** A bypass and NOT a waiver, deliberately: the gate offers a waiver only for a release that *deliberately needs no golden*, and this one needs one. **Owed: bake golden 0.207.0 and vouch it** (`RUNBOOK-manual-build.md` §4.1; the vouch is a three-field change). **This row's own remaining half is unchanged — nothing gates the VOUCH itself.** **✅ THE OWED BAKE IS DONE, SAME DAY — Line truncated
|
||||
| **R-242** | **A controller release that changes customer-visible behaviour is not delivered until a golden carries it — and nothing enforces that.** R-239 is the symptom; this is the mechanism, recorded **2026-08-07** and deliberately **NOT built** (the task that found it scoped it as a record-only item). **Two releases went out without a golden and the gap was invisible until a walk measured it from the customer's side**: v0.204.0 (R-237) and v0.205.0 (R-234) were written, tested, pushed, and CHANGELOG'd, and every one of those steps passed while a machine installed that night received neither. The register said CLOSED; the fleet said otherwise. **Nothing in the release path knows a golden exists.** The version bump, the image push, the CHANGELOG entry and the register closure are all repo-local; the manifest's `golden_version` is edited by a separate operator act, in a different repo, with no link back. **Proposed shapes, cheapest first — the choice is the operator's and is not taken here.** (a) **A release-path checklist step** — one line in the controller's end-of-session checklist: *a release that changes customer-visible behaviour is not finished until a golden carries it or a register row says why not.* Costs nothing, catches nothing mechanically. (b) **A gate in `repo_gates.py`** comparing the manifest's `golden_version` against the newest released controller and FAILING (or warning) past a tolerance of one minor. Mechanical, runs on every push, and would have fired the morning after v0.204.0. (c) **A hub-side checker** — the hub already knows every box's running controller version from `/hosts` and the vouched golden from the manifest; a periodic comparison against the newest published image would catch drift the repo cannot see, including a vouch that was made and then rolled back. **Earliest catch: (b).** It fires on the push that creates the gap, before any box is installed, and it needs no live fleet. **(c) catches strictly more but only after boxes exist.** (a) is worth doing regardless because it is free. **Not built. No gate was written this session.** **⚠ IT RECURRED WITHIN A DAY, WHICH IS THE ARGUMENT FOR BUILDING IT.** Controller **v0.206.0** shipped the R-241 fixes on 2026-08-07 while the vouched golden still carried **0.205.0** — so a machine installed on the morning of 2026-08-08 would have received neither. Third occurrence of the shape in three days (R-111/R-115/R-120 are the older family). **✅ SHAPE (b) BUILT 2026-08-08 — `scripts/golden_currency_gate.py`, registered in `repo_gates.py` as gate 7.** **It was shown FAILING against that exact state before anything was baked**, which is its red-proof and the reason its own introducing push needed `--no-verify` (stated in the session report rather than worked around): `newest released controller : 0.206.0 / newest golden baked : 0.205.0 → CONVICTED`. **IT IS `--fast`, AND THAT FORCED ITS DESIGN:** both `.githooks/pre-push` AND CI run `repo_gates.py --fast`, so a non-fast gate would run in NEITHER — the R-29 census failure this runner exists to end. **THEREFORE IT CHECKS THE BAKE, NOT THE VOUCH**, because the vouched version lives only in the hub's `hub_settings` with no copy in git, and putting a copy there would create a second source of truth that can drift — a green gate over a false claim being the worst outcome available. **A bake without a vouch still passes: that half is NOT closed and stays on this row.** It also compares versions rather than behaviour, so a release changing nothing customer-visible trips it too — accepted deliberately, because judging that by hand is what failed three times and the cost of a false trip is one bake; a waiver belongs here, never in a habit of bypassing. **VOUCHED 2026-08-08 with the operator's approval** — golden `0.206.0` / sha `c85230b4…108e`; `agent_version` and `min_agent` both stayed `0.127.0`, and `wrapper_sha256` was carried through explicitly because the handler clears it when omitted. **The gate was CONVICTED before the bake and OK after it** — red→green on the same command, which is its proof that it measures something real. **⚠ THE GATE FIRED FOR REAL, 2026-08-08 — and it was right.** Controller **v0.207.0** (R-249/R-252/R-253) is released, tested and pushed, and **no golden carries it** — the newest bake is 0.206.0 — so `golden_currency_gate.py` FAILED, saying exactly the true thing: *a machine installed right now would receive v0.206.0*. **The `felhom.eu` push therefore used `git push --no-verify`, declared here, in the commit message and in the session report.** A bypass and NOT a waiver, deliberately: the gate offers a waiver only for a release that *deliberately needs no golden*, and this one needs one. **Owed: bake golden 0.207.0 and vouch it** (`RUNBOOK-manual-build.md` §4.1; the vouch is a three-field change). **This row's own remaining half is unchanged — nothing gates the VOUCH itself.** **✅ THE OWED BAKE IS DONE, SAME DAY — Line truncated
|
||||
| **R-243** | **A box in the R-241 state silently stops backing up off-site, and NO ALARM OF ANY KIND FIRES.** Found by the R-241 spike (2026-08-07) as a by-product; **not part of the walk's finding and not previously filed.** The R-241 state is self-locking in a second, worse way than the recovery-journey dead end: `escrow_state` is stuck `pending` forever (the auto-confirm flips only on a hash match, and the hash cannot match a key the box minted itself), and `runOffboxBackup` returns at the escrow gate (`offbox.go:743`) before touching anything. **So off-site backups never run again — and the hub never notices.** All three signals that could catch it are excluded, each for its own individually-correct reason, verified in the hub this session: `offsite_stale` — `isStale` (`monitor/offsite.go:135`) returns false unless `EscrowState == "escrowed"`, and its own comment reads *"Pending/disabled = normal onboarding, never stale"*, so the box is classified as **still being set up, forever**; `offsite_delivery_stuck` — `monitor/offsite_delivery.go:91` skips the `applied` shape, and delivery genuinely IS applied (the credential was consumed and the target is in every report); `backup_failed` — never fires, because nothing fails: the run returns `nil` before it starts. **Three correct exclusions leaving one state unobserved.** This is the same class as the workspace `CLAUDE.md` "presence is not success" rule, one level up: **the absence of a failure is being read as the presence of a working tier.** **Partly subsumed by R-241's fix** — a box that recovers leaves this state — but **not for a box that does not**, and the alarm gap is what makes "does not" survivable indefinitely. **Not fixed; no code written.** **⚠ UPDATED 2026-08-07 (v0.206.0) — the STATE this row describes can no longer be entered, but the ALARM GAP is untouched and the row stays open.** R-241's mint guard means a box no longer mints a key over a sealed package, so it no longer arrives in the "escrow stuck pending against a self-minted key" state by itself. **What replaces it is a state that is VISIBLE rather than silent:** the box declares `offsite.state=awaiting_recovery_key` and the customer is offered the recovery screen. **But the hub still raises nothing for it**, and for the same three reasons: `isStale` needs `escrowed`, the delivery checker skips the `applied` shape, and `backup_failed` needs a run that never happens. **So a box whose customer never acts still stops backing up off-site with no operator signal** — the difference is that the customer can now see it and act, where before nobody could. **The remaining work is an operator-side signal for a box held in `awaiting_recovery_key` past some age**, and it is deliberately not bundled into R-241's fix. **⚠ MEASURED ON A REBUILD, 2026-08-07 (fifth walk) — the gap is real for the state this row describes, and NOT for the state a rebuild produces.** 88 seconds after the walk5 guest was destroyed and rebuilt, the hub emitted `offsite_delivery_stuck` (**warning**) and wrote an **operator-channel** `notification_log` row recording `offsite_credential_restaged` / status **REFUSED** with an accurate reason — *"the credential was applied and worked; the target was lost afterwards … a guest rebuild does, R-193"*. So on the **regressed-apply** shape the operator IS told, promptly and correctly, and this row's *"skips the applied shape"* does not apply. The gap stands for a box that reaches the held state **without** a prior working tier in its report history. **Recorded so the row is not read wider than it measures.** | **READY** — owner Viktor |
|
||||
|
||||
| **R-244** | **The customer DELETE cascade leaves `app_log_issues` behind, and it is systematic across every venue ever torn down.** Found **2026-08-07** while verifying the `finalwalk` teardown with a **full census** (every table, every column) rather than a per-table query. After a cascade that logged `COMPLETE … full teardown`, **61 rows still matched `finalwalk`**. Four of the five sources are **deliberate and correct** — the cascade's own header states *"Provenance/events are NEVER wiped — audit outlives every tier"*: `events` 16, `notification_log` 14, `host_deletions` 1, `customer_resets` 1. **The fifth is a gap:** `app_log_issues` 29 rows, which the residue purge does not touch (its logged leg covers `reports`/`app_telemetry`/`app_log_tails`/`log_tail_requests`/`notif_prefs`/`selfbind_tokens`/`appliance_registrations` — not this table). **It is not a `finalwalk` quirk:** rows still reference **`c11` 40, `rewalk` 20, `part4` 24** — all three torn down 2026-08-06, whose ledger recorded *"0 occurrences"*. **That prior claim was measured with a narrower query and does not survive a full census; the correction is recorded rather than the measurement quietly redone.** **Why it was probably never written, established rather than assumed:** the table is a **fleet-wide aggregate** keyed on `app_name`+`fingerprint` with an `affected_customers` JSON list — of the 29 `finalwalk` rows, **12 reference only `finalwalk`** (orphans, safely deletable) and **17 are shared with LIVE customers** (`demo-felhom`, `peti-felhom`, …) and **must not be deleted, only de-referenced.** A naive `DELETE … WHERE customer LIKE` would destroy a live customer's issue history — which is very likely why the leg does not exist, and is the reason this is not a one-line fix. **Severity is LOW and stated plainly: no secret material is involved** — app name, fingerprint, message text, counts, timestamps. What survives is a deleted customer's *identifier* inside an aggregate row. **Proposed shape:** a residue leg that (a) removes the customer id from `affected_customers`/`context_customer`, and (b) deletes rows whose `affected_customers` becomes empty; plus a one-off sweep for the four already-torn-down venues. **The general lesson is the reusable part:** *a per-table absence query is not a census.* The teardown verification is now a full-schema sweep, and that is what found this. **Not fixed** — a cascade change needs its own red-proof and this session was scoped as a spike plus two operations. Evidence: `tests/teardown-finalwalk-2026-08-07.md`. **⚠ STILL OWED, AND NOW MEASURED RATHER THAN ESTIMATED (2026-08-08 census, read-only, no truncation).** `app_log_issues` holds **1309 rows**; **71 reference a torn-down venue** (`finalwalk`, `c11`, `rewalk`, `part4`); of those **44 are ORPHANS** — they name only torn-down customers and are safely deletable — and **27 are SHARED with a live customer** (`demo-felhom`, `peti-felhom`, …) and **must be de-referenced, never deleted**. 1238 rows are untouched. **The 27 are exactly why the leg was never written**, and why a `DELETE … WHERE customer LIKE` would destroy a live customer's issue history. **What it needs, precisely:** a cascade leg that (a) removes the customer id from `affected_customers` / `context_customer`, and (b) deletes only rows whose `affected_customers` becomes empty; plus a one-off sweep for the four venues already gone. **Why it was NOT done on 2026-08-08:** the fix is hub code, and that session's scope forbade a hub version bump; a hand-run SQL mutation over 71 rows — 27 of them needing surgical de-referencing — with no tested code path and no red-proof is precisely the shape that goes wrong on a live database. **It accumulates one venue at a time, so the next walk adds to it**; the numbers above mean the next session starts from data rather than a guess. **⚠ IT GREW AGAIN, AS PREDICTED — walk5 teardown, 2026-08-08.** The fifth walk's venue was torn down with a full-schema census taken **before and after**: **168 rows → 67**. Of the 67, **37 are by design** (`events` 21, `notification_log` 14, `host_deletions` 1, `customer_resets` 1) and **30 are `app_log_issues`** — this row's gap, and the count was **predicted in the pre-run enumeration rather than discovered afterwards**, which is the difference from the ledger that once recorded *"0 occurrences"* from a narrower query. **The running total across torn-down venues therefore rises from 71 to ~101 rows** (`finalwalk`, `c11`, `rewalk`, `part4`, now `walk5`) — the shared-with-a-live-customer subset must still be **de-referenced, never deleted**. **It accumulates one venue at a time and it did so again.** Evidence: `tests/walk5-r201-2026-08-07/teardown-walk5-2026-08-08.md`. | **READY** — owner Viktor |
|
||||
@@ -522,7 +522,7 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
|
||||
| **R-336** | **The offsite DR endpoint is polled about once per second, and that is what turned a slow leak into an outage.** ep0's PBS proxy served **~85,000 requests/day** — a flat **3,538/hour**, every hour, from two boxes: `74,445 GET /api2/json/admin/datastore` (`libwww-perl`, i.e. PVE's `pvestatd`) and `73,171 GET /admin/datastore/felhom-offsite/status` (`proxmox-backup-client`). Two pollers asking substantially the same question at the same rate. On 2026-08-18 this walked a connection leak in the proxy to its 1024-fd soft limit in **14 days**, wedging the offsite tier for 9½ hours (`audits/INCIDENT-ep0-pbs-fd-exhaustion-2026-08-18.md`). The `LimitNOFILE=65536` drop-in applied that morning raises the ceiling **but does not fix the leak** — it converts a fortnightly outage into a multi-year one, which is mitigation, not a fix. A DR endpoint that is written to weekly does not need to be asked about every second | **READY (M) — NEW 2026-08-18** | — | **CORRECTED 2026-08-18 (evening) — the easy lever named here does not exist.** This cell used to read *"PVE storage status is the prime suspect, and its interval is tunable"*. **The first half is right and the second half is false.** `pvestatd` stats EVERY configured storage on each 10-second cycle, and Proxmox staff have stated the interval is not designed to be configurable — so there is no knob to turn down. The only lever PVE actually offers is disabling the storage entry (`pvesm set <id> --disable 1`) around the backup window, and that is **substantially more than a tuning knob**: it collides with `felhom-agent/internal/pbsdr/manager.go`'s health model, where an inactive-but-existing entry drives the consume-the-one-time-secret recovery path. So the fix is a design question (does the hub still need a 15-minute fill reading at all, given R-339 now reports reachability separately?), not a config edit. **Doc-only correction — no agent code was changed.** The remaining step is unchanged: cut the poll rate by whatever means survives that question, then confirm the fd count between restarts stops climbing — the positive observable, per standing rule 3. **Baseline measured 2026-08-18, and the FIRST measurement published was WRONG.** The initial "~85/day, matching the ~73/day implied by the failure" came from a single 17-minute window whose delta was **one descriptor** — a sample of one cannot carry a daily rate, and the agreement that made it feel solid was coincidence. **Re-measured over two independent windows the same morning: 183/day (31 min) and 200/day (5.6 h)** — ~2.6x the published figure, putting the runway to the 65536 ceiling at **~357 days, not the ~2 years first claimed**. **And the named mechanism is the minority one:** across that window `CLOSE-WAIT` held flat at 1 while `ESTAB` grew 45→49 — *all* the growth was established connections, and at the wedge the split was 1011 ESTAB / 543 CLOSE-WAIT. **The fix must target connections the proxy never reaps, not just `CLOSE-WAIT` sockets.** The PBS 4.2.5-1 upgrade (2026-08-18) did NOT change the slope and was never expected to — see R-341 **SPIKE 2026-08-20 — THE PREMISE OF THIS ROW DOES NOT SURVIVE MEASUREMENT, and that is a change in what the row IS, not new evidence on it.** `audits/SPIKE-ep0-established-connections-2026-08-20.md`. **The leak is OURS, and the poll rate is not what feeds it.** Every one of the 388 leaked descriptors is an ESTABLISHED connection held open by **`felhom-agent`** on the boxes — 194 on each, `ss -tnp` naming a single PID per box, and **zero** held by `pvestatd` or `proxmox-backup-client`. Confirmed independently from ep0's access log over the same 46.18 h window: `libwww-perl` (pvestatd) **81,192 requests -> 0 descriptors**, `proxmox-backup-client` **80,061 requests -> 0 descriptors**, `Go-http-client/1.1` (the agent) **387 `/snapshots` calls -> 388 sockets — one per call, within one**. So **162,404 requests, 99.5% of the traffic, produce 0% of the leak.** **Mechanism, named from source:** `felhom-agent/internal/pbs/client.go:56-60` builds `&http.Transport{TLSClientConfig: tlsCfg}` — a composite literal, so `IdleConnTimeout` is the zero value = **no limit** (`http.DefaultTransport` sets 90 s; a literal does not inherit it) — and `cmd/felhom-agent/main.go:1486` (`pbsTargetsFromPVE`) builds **a fresh client every cycle**, as its own doc comment states. Each cycle therefore strands one idle keep-alive connection in a transport nothing ever closes; `CloseIdleConnections`/`IdleConnTimeout`/`MaxIdleConns` appear **nowhere** in the agent repo. Cadences reconcile without fitting: 900 s hub poll (184.7 cycles) + 6 h `DefaultVerifyCadence` (7.7 cycles) = 192.4 predicted vs **194 observed per box**. **CONSEQUENCE — RE-RANK.** The remaining step recorded above ("cut the poll rate, then confirm the fd count stops climbing") **would have produced a null result and read as a failed fix.** Cutting the Proxmox poll rate removes ~99.5% of ep0's reqLine truncated
|
||||
| **R-337** | **`/backup/status` lagged a completed backup by minutes on one box and not the other — and it RESOLVED ITSELF, which is why this is WATCHING and not a defect.** During the R-336 recovery on 2026-08-18, `demo-hp`'s snapshot landed on ep0 at **03:58:43Z** (complete manifest; the host's own task index says `OK`) — yet `GET /backup/status` was **still serving the superseded 03:27:00Z failure at ~04:03Z**, four-plus minutes later. `demo-felhom` showed its new result within ~40 s of completion. **The lag cleared on its own:** demo-hp's 04:07:35Z host report carries `felhom-pbs success=true, 4.29 GB`, and the hub is green for both boxes. **The first draft of this row claimed the success was "still reported as failed" — that was written before the next report arrived and it was wrong; the corrected claim is a several-minute skew between the two boxes, not a stuck value.** It is recorded because a status field that can trail its own artifact by minutes will, during an incident, be read as a second failure — this session nearly did — and because the asymmetry between the two boxes is unexplained | **WATCHING — NEW 2026-08-18** | another observation, ideally during an incident rather than constructed | **Do not open a fix on this as written.** First establish the intended refresh path for `/backup/status` after an out-of-schedule run; only if the skew is not simply collection cadence is there anything to pin. If it is cadence, close this row and say so | CC |
|
||||
| **R-338** | **`demo-hp` is not on the R-50 island at all, and `operations/nodes.md` states that it is.** The page records both fleet boxes as island-migrated 2026-07-25. True of `felhom-pve`; **false of `demo-hp`**, whose `agent.json` has `listen_addr: 192.168.0.87:8443` — the customer LAN address — and **no `island_bridge`/`island_guest_addr` keys at all**, whose guest 9201 has `net0` only (no `eth1`), and whose `vmbr9` exists with **zero members**. The controller's `controller.yaml` points at the LAN address, so the box works; this is inventory drift, not breakage. **Two costs.** A session trusting the page addresses the wrong endpoint — that happened on 2026-08-18 and the resulting timeout was briefly read as a fault. And the agent's local API is **bound to the customer LAN on this box** rather than to a point-to-point island, which is the exposure R-50 was built to remove — so a documented security property is claimed for a box that does not have it | **READY (S) — NEW 2026-08-18** | — | Decide which is true: migrate `demo-hp` to the island, or correct `nodes.md`. Leaving both is the one option that keeps the doc lying | Viktor decides; CC executes |
|
||||
| **R-341** | **Does the fd slope change after the PBS 4.2.5 upgrade? — two dated checks, and the answer is expected to be NO.** ep0 was upgraded 4.2.2-1 → 4.2.5-1 on 2026-08-18 09:51Z on the operator's ruling, **for rehearsal value, not as a fix**: the full changelog range was read (128 lines, all three entries) and swept for connection-handling vocabulary, and it contains **no mechanism** by which descriptor reaping would change — the single keyword hit was `S3 … honor the node's proxy settings`, HTTP-proxy config for S3, not the PBS proxy daemon. **The 32-minute post-upgrade window is indistinguishable from the before window** (+5 fd/1919 s = 225/day vs +4 fd/1885 s = 183/day; the two differ by ONE descriptor and Poisson uncertainty on such counts is ±2, so both are consistent with one unchanged rate — the higher after-figure is noise, not a regression). **Thirty minutes cannot settle it in either direction and this row exists so nobody pretends it did.** New `t0` = **fd 17 at 2026-08-18 09:51:22Z, proxy PID 551655**; before-rate to beat = **183–200/day**. **Interpretation fixed in advance** (`evidence-ep0-pbs-upgrade-2026-08-18/stop1-ruling.txt`, written before any numbers existed): unchanged = EXPECTED, not a failed upgrade; changed = a SURPRISE needing explanation, not a confirmation | **WATCHING — FIRST CHECK TAKEN 2026-08-20: the rate is UNCHANGED, exactly as pre-registered. Second check still due 2026-08-25** | elapsed time only | **Two dated checks, both CC:** **+24 h — 2026-08-19 ~10:00Z** and **+7 d — 2026-08-25 ~10:00Z**. Command (the incident's own positive observable): `ssh root@<ep0> 'PID=$(systemctl show proxmox-backup-proxy -p MainPID --value); ls /proc/$PID/fd \| wc -l; ss -lnt "( sport = :8007 )"; ss -tn state all "( sport = :8007 )" \| awk "NR>1{print \$1}" \| sort \| uniq -c'`. **Record the ESTAB/CLOSE-WAIT split, not just the total** — the split is what says which leak it is. If PID ≠ 551655 the window is void: something restarted the proxy and the count began again **FIRST CHECK — TAKEN 2026-08-20 08:02:13Z, and it was taken at +46.2 h, NOT +24 h.** No session ran between 18 and 20 August, so the 2026-08-19 date passed as pure elapsed time; the delay is stated here rather than backfilled. **The longer window is a BETTER measurement, not a degraded one** — the leaked count is **388** against the 4 and 5 descriptors the original answer rested on, roughly **97x**, so the uncertainty falls from about +/-50% to about +/-5%. **Precondition PASSED:** PID still **551655**, `ps -o lstart` 2026-08-18 09:51:04, `NRestarts=0` on both units. **Result:** fd **17 -> 405** over **166,251 s** = **201.6 fd/day**, Poisson +/-10.2/day (1s), 2s band **181.2-222.1**. Pre-registered range was **370-450** (confirm-band 330-490); **observed 388**, near the centre. **VERDICT: unchanged — the EXPECTED result, and not a failed upgrade.** **Composition:** ESTAB 0 -> 388, **CLOSE-WAIT 0 — absent from the histogram entirely**, so 100% of the growth is established connections and `CLOSE-WAIT` is not merely the minority half. Runway from fd 405 at 201.6/day to the 65536 ceiling: **~323 days (~2027-07-09)**. Full working: `audits/SPIKE-ep0-established-connections-2026-08-20.md` + `evidence-ep0-established-connections-2026-08-20/step1-slope-computation.txt`. **MEASUREMENT TRAP for the 2026-08-25 check, found on this run — see R-346:** the anchor must be `ps -o lstart= -p $MainPID`, NOT `systemctl show -p ActiveEnterTimestamp`, which reads 03:54:54Z for this generation (the upgrade re-exec'd the proxy; systemd never saw a stop, `NRestarts` is still 0) and would put the rate ~15% low. **PERTURBATION NOTE for the 2026-08-25 check:** this spike's Phase C quietens `pvestatd` on demo-hp for one overnight window inside that 7-day interval. Quantified so nobody reads the shortfall as a change in the leak: even if the leak were fully proportional to the request rate (which Parts 1-2 predict it is NOT), a 10 h half-rate window inside 168 h shifts the 7-day slope by **~3%**, far inside the +/-10% Poisson band on a ~1,400-descriptor count. **The +7 d reading remains usable.** | CC on both dates |
|
||||
| **R-341** | **Does the fd slope change after the PBS 4.2.5 upgrade? — two dated checks, and the answer is expected to be NO.** ep0 was upgraded 4.2.2-1 → 4.2.5-1 on 2026-08-18 09:51Z on the operator's ruling, **for rehearsal value, not as a fix**: the full changelog range was read (128 lines, all three entries) and swept for connection-handling vocabulary, and it contains **no mechanism** by which descriptor reaping would change — the single keyword hit was `S3 … honor the node's proxy settings`, HTTP-proxy config for S3, not the PBS proxy daemon. **The 32-minute post-upgrade window is indistinguishable from the before window** (+5 fd/1919 s = 225/day vs +4 fd/1885 s = 183/day; the two differ by ONE descriptor and Poisson uncertainty on such counts is ±2, so both are consistent with one unchanged rate — the higher after-figure is noise, not a regression). **Thirty minutes cannot settle it in either direction and this row exists so nobody pretends it did.** New `t0` = **fd 17 at 2026-08-18 09:51:22Z, proxy PID 551655**; before-rate to beat = **183–200/day**. **Interpretation fixed in advance** (`evidence-ep0-pbs-upgrade-2026-08-18/stop1-ruling.txt`, written before any numbers existed): unchanged = EXPECTED, not a failed upgrade; changed = a SURPRISE needing explanation, not a confirmation | **CLOSED 2026-08-30 — SECOND CHECK TAKEN, AND IT CANNOT ANSWER THE QUESTION: the leak this row measured was REMOVED mid-interval by our own fix (R-344, agent 0.130.0). Question moot; the fix is confirmed holding at 12 days** | elapsed time only | **Two dated checks, both CC:** **+24 h — 2026-08-19 ~10:00Z** and **+7 d — 2026-08-25 ~10:00Z**. Command (the incident's own positive observable): `ssh root@<ep0> 'PID=$(systemctl show proxmox-backup-proxy -p MainPID --value); ls /proc/$PID/fd \| wc -l; ss -lnt "( sport = :8007 )"; ss -tn state all "( sport = :8007 )" \| awk "NR>1{print \$1}" \| sort \| uniq -c'`. **Record the ESTAB/CLOSE-WAIT split, not just the total** — the split is what says which leak it is. If PID ≠ 551655 the window is void: something restarted the proxy and the count began again **FIRST CHECK — TAKEN 2026-08-20 08:02:13Z, and it was taken at +46.2 h, NOT +24 h.** No session ran between 18 and 20 August, so the 2026-08-19 date passed as pure elapsed time; the delay is stated here rather than backfilled. **The longer window is a BETTER measurement, not a degraded one** — the leaked count is **388** against the 4 and 5 descriptors the original answer rested on, roughly **97x**, so the uncertainty falls from about +/-50% to about +/-5%. **Precondition PASSED:** PID still **551655**, `ps -o lstart` 2026-08-18 09:51:04, `NRestarts=0` on both units. **Result:** fd **17 -> 405** over **166,251 s** = **201.6 fd/day**, Poisson +/-10.2/day (1s), 2s band **181.2-222.1**. Pre-registered range was **370-450** (confirm-band 330-490); **observed 388**, near the centre. **VERDICT: unchanged — the EXPECTED result, and not a failed upgrade.** **Composition:** ESTAB 0 -> 388, **CLOSE-WAIT 0 — absent from the histogram entirely**, so 100% of the growth is established connections and `CLOSE-WAIT` is not merely the minority half. Runway from fd 405 at 201.6/day to the 65536 ceiling: **~323 days (~2027-07-09)**. Full working: `audits/SPIKE-ep0-established-connections-2026-08-20.md` + `evidence-ep0-established-connections-2026-08-20/step1-slope-computation.txt`. **MEASUREMENT TRAP for the 2026-08-25 check, found on this run — see R-346:** the anchor must be `ps -o lstart= -p $MainPID`, NOT `systemctl show -p ActiveEnterTimestamp`, which reads 03:54:54Z for this generation (the upgrade re-exec'd the proxy; systemd never saw a stop, `NRestarts` is still 0) and would put the rate ~15% low. **PERTURBATION NOTE for the 2026-08-25 check:** this spike's Phase C quietens `pvestatd` on demo-hp for one overnight window inside that 7-day interval. Quantified so nobody reads the shortfall as a change in the leak: even if the leak were fully proportional to the request rate (which Parts 1-2 predict it is NOT), a 10 h half-rate window inside 168 h shifts the 7-day slope by **~3%**, far inside the +/-10% Poisson band on a ~1,400-descriptor count. **The +7 d reading remains usable.** **SECOND CHECK — TAKEN 2026-08-30 16:40Z, five days late, and its own premise did not survive the interval.** Evidence: `audits/evidence-r341-plus7d-2026-08-30/step1-fd-and-sockets.txt`. **Precondition PASSED and this is what makes the reading interpretable at all:** PID still **551655**, `ps -o lstart=` still **2026-08-18 09:51:04**, `NRestarts=0` — the SAME proxy generation as t0, so nothing restarted and re-based the count. **Result: fd = 17.** Not 17 more; **seventeen total** — exactly the documented baseline, against **405** at the first check. The socket histogram contains **one LISTEN and nothing else**: ESTAB **0**, CLOSE-WAIT **0**. **THE VERDICT IS "UNANSWERABLE", NOT "THE UPGRADE FIXED IT", AND THE DIFFERENCE MATLine truncated
|
||||
|
||||
| **R-342** | **The ep0 snapshot covers less than it looks like it covers, and the next person will assume otherwise.** Quoting `audits/evidence-ep0-pbs-upgrade-2026-08-18/stop2-snapshot.txt` verbatim: *"covers — the 38 GB system disk /dev/sda (root), i.e. the PBS packages, unit files, /etc/systemd drop-ins, nftables and wg config. DOES NOT — /mnt/pbs-datastore. That is /dev/sdb, a separate 100 GB VOLUME, and Hetzner server snapshots do not include attached volumes. The backup data is therefore NOT protected by this snapshot."* Snapshot **421440873** (`felhom-hetzner-20260818`, 15.06 GB, Available) was taken as the rollback for the 4.2.2→4.2.5 PBS upgrade. **Rolling it back restores software state, not the datastore.** That was *acceptable for that change* — a package install writes no datastore content — and the file says so. **The problem is what happens next:** this fact lives in an evidence file nobody will open again, and a snapshot named as "the rollback" reads as protecting everything on the box. ep0 holds the only off-premises copy of a real customer's data | **READY (S) — NEW 2026-08-18** | — | **Decide the safeguard for any future ep0 procedure that could touch `/mnt/pbs-datastore` — it does not exist and has not been designed.** Candidates: a Hetzner **Volume** snapshot (a different object from the server snapshot), a PBS-level sync to a second location, or an explicit written acceptance that the datastore is unprotected for the duration. **Nothing may be added to a runbook implying a safeguard exists until one does** | **Viktor decides; CC executes** — a risk-to-customer-data question |
|
||||
| **R-343** | **The managed controller floor was raised 0.214.0 → 0.216.0 — and it was NOT the no-op it was expected to be: it moved a live box nine seconds later.** Raised by the operator 2026-08-18 **12:36:58Z**, in a **separate save after** the artifact vouch. **Read back from the store, not the form** (`hub_settings.min_controller_version`, `GetGlobalMinControllerVersion` — `hub/internal/store/store.go:1751`): `min_controller_version = 0.216.0`, updated 12:36:58. **WHY IT HAD BEEN BEHIND — this was NOT drift, and describing it as "two releases behind" without this context reads as a defect it was not.** `publish-train-rules.md` rule 1 is *manifest before floor*, and rule 2 requires the floor field to be filled **LAST, in a separate save**, because the DB row overrides the env floor and **acts immediately on the next report cycle**. That rule was earned: on the 2026-07-11 publish train the floor was saved together with the manifest, acted at once, and pushed controller 0.113.0 onto Peti's box **~9 minutes ahead of agent 0.81.0** — the exact forbidden skew, benign only because that box had no NAS shares. R-120's row records the same deliberate choice (*"Floor untouched per publish-train rule 2"*). So the floor sitting at 0.214.0 was **policy being followed**, not neglect. **What it was functionally while it sat there:** not a live problem — every reporting box was at or above it — but a **safety net set two versions low**. The floor is what drags a box forward if it ever falls behind (restored from an old backup, reinstalled, long offline), and at 0.214.0 it would have pulled such a box only to two versions back, missing R-328's severity fix and R-335's follow-up. **THE MEASURED BLAST RADIUS — five reads, and the third and fifth are the findings.** **(1) Floor:** `0.216.0` @ 12:36:58Z, from the store. **(2) Per-customer overrides:** **zero** — all five `customer_configs` rows carry an empty `min_controller_version`, so nothing hides behind a lower override and the global applies to everyone. **(3) Every box's controller version:** `demo-felhom` **0.216.0**, `demo-hp` **0.216.0** — both AT the floor **now**; `drill-r50` **0.213.0** (status `blocked`, last report 2026-08-12, powered off/reverted) and `peti-felhom` **0.115.0** (host row DELETED) are **below** it but are **not reporting boxes**. **(4) Directives/holds:** no `managed floor HELD` line exists; the hub logged `[INFO] Global controller-version floor set to "0.216.0"`. **(5) THE FINDING — a controller DID auto-update after the raise.** `demo-felhom` had been on **0.214.0** since 2026-08-12 16:44 and the hub recorded `controller_updated — Controller frissítve: 0.214.0 → 0.216.0` at **12:37:07Z**, then `controller_started (0.216.0)` at 12:37:12Z — **nine seconds after the raise**, exactly the immediate action rule 2 documents. `demo-hp` was already on 0.216.0 (hand-deployed 2026-08-14 08:31) and did not move. **No error, warning or critical event followed** — the update completed and the controller came back up. **So the change was real, not inert: "every reporting box is at or above the floor" is true BECAUSE of the raise, not independently of it.** **Why it is safe by construction**, cited rather than asserted: `ResolveManagedFloor` (`hub/internal/store/store.go:2068`) sets `Held` and clears the floor entirely when `Floor > GoldenVersion` (the R-216 shape) — floor 0.216.0 **equals** golden 0.216.0, so that guard does not trip — and holds per-box when the box's agent is below the manifest's `MinAgent`, or unknown, or unparseable; both boxes report agent 0.129.0 against `MinAgent` 0.129.0, so the floor was served rather than held. That second guard remains armed for any box reporting with an old agent. **No ISO rebuild is required:** the golden is fetched at first boot from the hub's manifest, which is 0.216.0 — at the floor, not below it — and rule 5's `assert_golden_ge_floor` is a **build-time** gate for *future* builds (`scripts/iso/build-felhom-iso.sh:77`, called at **:267**) which **fails open with a warning** when its inputs are absent (`:78-82`: `if [[ -z "$golden" \| \| -z "$floor" ]]` → `log_warn "… UNENFORCED …"` → `return 0`), both confirmed in the script. **`peti-felhom` was NOT contacted and needs no contact** — from the PETI row: its host row was deleted 2026-07-15 and *"a report from a deleted host 401s and is not persisted"*, so it cannot receive a floor directive at all and the raise cannot reach it | **OPEN — NEW 2026-08-18.** Deliberately NOT closed: the task's closing condition was *all five reads clean, no directive served*, and read 5 shows a live box updated. It went cleanly and is the floor working as designed — but a change recorded as a no-op when it moved a customer box is exactly the kind of record that misleads later | — | **Confirm the 0.216.0 update on `demo-felhom` is healthy in normal operation** (it reported and restarted clean, but it has not yet run a full bacLine truncated
|
||||
@@ -595,5 +595,4 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
|
||||
the R-row. Duplicating them here would create the second source this design avoids. -->
|
||||
| item | due (UTC) | what to measure |
|
||||
|---|---|---|
|
||||
| R-341 | 2026-08-25 | same, +7 d. Anchor the elapsed time on `ps -o lstart= -p $MainPID`, NOT `ActiveEnterTimestamp` (R-346). A Phase-C perturbation sits inside this interval and is quantified in the R-341 row |
|
||||
<!-- DUE-CHECKS-END -->
|
||||
@@ -1,3 +1,77 @@
|
||||
## v0.109.0 — the Backup card told every operator that every customer had no backups (2026-08-30, R-331)
|
||||
|
||||
> **This push used `git push --no-verify`, and that is declared here rather than worked around.**
|
||||
> `golden_currency_gate.py` was CONVICTED: controller v0.224.0 and v0.225.0 are released and the newest
|
||||
> golden bake carries 0.223.0, so a machine installed right now receives neither. **A bypass, not a
|
||||
> waiver** — the gate offers a waiver only for a release that *deliberately needs no golden*, and these
|
||||
> need one. The operator ruled bypass-now-bake-later, on the ground that neither fix bites a day-0 box
|
||||
> (R-330 is a nightly false alarm about apps a new box has not installed; R-331 is a hub display over
|
||||
> backups a new box has not taken) and both arrive by self-update. **A golden carrying 0.225.0 is
|
||||
> OWED** — tracked on R-242 in `documentation/backlog/OPEN-ITEMS.md`, which also records that this
|
||||
> ground does not extend to a release changing first-boot behaviour.
|
||||
|
||||
### R-331 — `Snapshots 0` over a repository holding 67
|
||||
|
||||
The customer page's **Backup** card read `Snapshots 0 · Repo Size 0 MB · Integrity Unknown` for
|
||||
**every customer, indefinitely**. Measured on `demo-hp` 2026-08-30 while that same night's controller
|
||||
log said `[offbox] backup OK: 8 app(s) backed up, 67 snapshot(s), 2m14s` and the box's own settings
|
||||
held `snapshot_count: 67, repo_size_bytes: 140829678, stats_known: true`.
|
||||
|
||||
**A card that reads "no backups" over a working backup is worse than no card** — it is the R-88
|
||||
direction of failure (degrading to *no backup* rather than to *unknown*) on the one screen an operator
|
||||
consults to answer "is this customer protected?".
|
||||
|
||||
**The data was never missing.** The card rendered the report's `backup` object, whose
|
||||
`snapshot_count` / `repo_size_mb` / `integrity_ok` fields have had **no producer** since disk-tier
|
||||
restic moved to the host agent (slice 8C) — the controller's `buildBackupReport` says so in a comment
|
||||
and leaves them zero. The live numbers were in the `offsite` object all along, which **this package
|
||||
already reads** (`offsiteUsageBytes`, for the Offsite page) and which `monitor.OffsiteChecker` already
|
||||
drives fill and staleness alarms from. Proof the bytes were arriving: the Offsite page rendered
|
||||
demo-hp's usage as `0.1 GB` from that very object while the Backup card said `0 MB`.
|
||||
|
||||
So this is a **render fix over an existing feed**, not a new pipeline.
|
||||
|
||||
### Why it was not a one-line template swap
|
||||
|
||||
`snapshot_count: 0` means two opposite things — *this repository holds nothing* and *nobody has ever
|
||||
measured this repository*. **R-225 already measured that confusion one layer down**: a rebuilt box
|
||||
rendered „Tarolo meret · 0 pillanatkep" over a store that really held snapshot `f3d9cd67`, and
|
||||
`StatsKnown` is what fixed it in the controller's own UI. Rendering the count without consulting it
|
||||
would have moved R-225 up to the hub instead of fixing anything. Controller **v0.225.0** now forwards
|
||||
`stats_known`; the card is a **three-way** ruling, resolved in Go (`backup_card.go`) rather than in a
|
||||
template `{{if}}` chain over `map[string]interface{}` float64s:
|
||||
|
||||
| report state | card shows |
|
||||
|---|---|
|
||||
| no `offsite` object at all | "No off-site data reported" — **and says this is not the same as "no backups"** |
|
||||
| `enabled:false` + declared `state` | the blocker by name (`needs_credential`) — a different operator action from "not enabled" |
|
||||
| enabled, `stats_known:false` | **—**, plus "never been measured". Never `0` |
|
||||
| enabled, `stats_known:true` | the real count and size, **including a real `0`** — measured empty is knowledge |
|
||||
|
||||
**A pre-v0.225.0 controller sends no `stats_known`, which unmarshals to false → "unknown".** That is
|
||||
the fail-safe direction and it is pinned by a test: upgrading the hub ahead of the fleet must not tell
|
||||
the operator that every un-upgraded customer has zero backups.
|
||||
|
||||
### The Integrity row is DELETED, not re-sourced
|
||||
|
||||
Nothing in the controller produces it. There is no integrity check; `NotifyIntegrityOK` and
|
||||
`NotifyIntegrityFailed` exist and are **called from nowhere**. A row that can only ever read "Unknown"
|
||||
is not information, and one that could read "OK" from an unwritten field would be a lie.
|
||||
|
||||
### Tests
|
||||
|
||||
`r331_backup_card_test.go` asserts the **rendered page**, using demo-hp's real reported values, so a
|
||||
regression fails against the same numbers the defect was measured against. The defect lived in the
|
||||
template's choice of source object, so a test one layer below it would have been green against the
|
||||
shipped bug.
|
||||
|
||||
**RED-PROOF:** restoring the pre-fix card markup fails all four tests, reporting `67` and `134.3 MB`
|
||||
absent from the page and the `Integrity` row present.
|
||||
|
||||
`fmtBytesAuto` is new rather than reusing the Offsite page's `fmtBytesGB`: that one is fixed at GB
|
||||
because it renders against GB quotas, and it turns demo-hp's real 140 829 678 bytes into `0.1 GB` —
|
||||
which on a card whose entire defect was under-reporting a real backup reads as "nearly nothing".
|
||||
|
||||
## v0.108.0 — only the first broken app per hour reached the operator (2026-08-23, R-389)
|
||||
|
||||
**The hour is unchanged. The GRAIN is what was wrong.**
|
||||
|
||||
@@ -0,0 +1,148 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ── R-331 — the operator Backup card told every customer they had no backups ─────────────────────
|
||||
//
|
||||
// THE DEFECT. `customer_unified.html`'s Backup card rendered `snapshot_count`, `repo_size_mb` and
|
||||
// `integrity_ok` out of the report's `backup` object. Those three fields have had NO PRODUCER since
|
||||
// disk-tier restic moved to the host agent (slice 8C): the controller's `buildBackupReport` says so
|
||||
// in a comment and leaves them zero, so the card read `Snapshots 0 · Repo Size 0 MB · Integrity
|
||||
// Unknown` for every customer, forever. Measured on `demo-hp` 2026-08-30, while the box's own log for
|
||||
// that night said `[offbox] backup OK: 8 app(s) backed up, 67 snapshot(s), 2m14s`.
|
||||
//
|
||||
// **A card that reads "0 snapshots" over a working backup is worse than no card.** It is the R-88
|
||||
// direction of failure — degrading to "no backup" rather than to "unknown" — on the one screen an
|
||||
// operator would consult to answer "is this customer protected?".
|
||||
//
|
||||
// THE DATA WAS NEVER MISSING. The live numbers ride in the report's `offsite` object, which this
|
||||
// package already reads for the Offsite page (`offsiteUsageBytes`) and which `monitor.OffsiteChecker`
|
||||
// already drives fill and staleness alarms from. The card was simply pointed at the wrong object.
|
||||
// So this is a RENDER fix over an existing feed, not a new pipeline — and `integrity_ok` is dropped
|
||||
// rather than re-sourced, because nothing in the controller has produced it since 8C either:
|
||||
// `NotifyIntegrityOK` / `NotifyIntegrityFailed` exist and are called from nowhere.
|
||||
//
|
||||
// WHAT MADE IT MORE THAN A ONE-LINE TEMPLATE SWAP. `snapshot_count: 0` means two opposite things —
|
||||
// "this repository holds nothing" and "nobody has ever measured this repository" — and the report
|
||||
// could not tell them apart until controller v0.225.0 started forwarding `stats_known`. R-225
|
||||
// measured that exact confusion one layer down: a rebuilt box rendered „Tároló méret · 0 pillanatkép"
|
||||
// over a store that really held snapshot `f3d9cd67`. Rendering the count without consulting
|
||||
// `stats_known` would have moved R-225 from the controller UI to the hub UI instead of fixing it.
|
||||
|
||||
// backupCardView is what the Backup card renders. Every field is resolved in Go rather than in the
|
||||
// template, so the three-way "no data / unknown / known" ruling is unit-testable and lives in one
|
||||
// place — a template `{{if}}` chain over a `map[string]interface{}` of float64s is neither.
|
||||
type backupCardView struct {
|
||||
// --- local app-data leg (the report's `backup` object) ---
|
||||
// These two are the ONLY fields of that object with a live producer, which is why nothing else
|
||||
// from it appears on the card any more.
|
||||
LocalEnabled bool
|
||||
LastDBDump string // humanised; "—" when the box has never dumped
|
||||
|
||||
// --- off-site leg (the report's `offsite` object) ---
|
||||
|
||||
// OffsiteReported is false when the report carried no `offsite` object at all: a pre-v0.109.0
|
||||
// controller, or a box that has never had the tier. The card then says so and shows NO numbers —
|
||||
// the same rule `offsiteBoxTile` already follows ("absent, not a zeroed tile").
|
||||
OffsiteReported bool
|
||||
OffsiteEnabled bool
|
||||
// OffsiteState carries a DECLARED state (`needs_credential`, `awaiting_recovery_key`) for a box
|
||||
// that reports an object while disabled. Rendering it is the difference between "no off-site
|
||||
// backup" and "off-site backup is blocked waiting for a key" — an operator action item.
|
||||
OffsiteState string
|
||||
|
||||
// StatsKnown is false both when the repository was never measured AND when the controller is too
|
||||
// old to say. Both are ignorance, and the card must show ignorance, never zero.
|
||||
StatsKnown bool
|
||||
Snapshots int
|
||||
RepoSize string // humanised; only meaningful when StatsKnown
|
||||
|
||||
LastSuccess string // humanised age of the last run that actually SUCCEEDED; "—" when never
|
||||
LastStatus string // "ok" | "incomplete" | "error" | "running"
|
||||
QuotaStr string // "" when the target has no soft quota (dedicated boxes)
|
||||
}
|
||||
|
||||
// reportBackupCard parses one customer's stored report into the card view. A report that will not
|
||||
// parse yields a zero view, which renders as "nothing reported" — never as zeroed numbers.
|
||||
//
|
||||
// It takes the raw JSON rather than the already-parsed `map[string]interface{}` the page also holds,
|
||||
// for the reason the sibling readers in this file's package do: typed decoding keeps the int/float64
|
||||
// and absent/zero distinctions that a generic map destroys, and those distinctions are the whole
|
||||
// subject of this card.
|
||||
func reportBackupCard(reportJSON string) backupCardView {
|
||||
var r struct {
|
||||
Backup *struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
LastDBDump *time.Time `json:"last_db_dump"`
|
||||
} `json:"backup"`
|
||||
Offsite *struct {
|
||||
Enabled bool `json:"enabled"`
|
||||
State string `json:"state"`
|
||||
LastStatus string `json:"last_status"`
|
||||
LastSuccess string `json:"last_success"`
|
||||
SnapshotCount int `json:"snapshot_count"`
|
||||
RepoSizeBytes int64 `json:"repo_size_bytes"`
|
||||
QuotaGB int `json:"quota_gb"`
|
||||
// StatsKnown is ABSENT on a controller below v0.225.0, and absent unmarshals to false —
|
||||
// which is the correct fail-safe direction: a box that cannot answer is unknown, never
|
||||
// empty. See OffboxReportStatus.StatsKnown in the controller.
|
||||
StatsKnown bool `json:"stats_known"`
|
||||
} `json:"offsite"`
|
||||
}
|
||||
var v backupCardView
|
||||
if json.Unmarshal([]byte(reportJSON), &r) != nil {
|
||||
return v
|
||||
}
|
||||
|
||||
if r.Backup != nil {
|
||||
v.LocalEnabled = r.Backup.Enabled
|
||||
v.LastDBDump = "—"
|
||||
if r.Backup.LastDBDump != nil && !r.Backup.LastDBDump.IsZero() {
|
||||
v.LastDBDump = timeAgo(*r.Backup.LastDBDump)
|
||||
}
|
||||
}
|
||||
|
||||
if r.Offsite == nil {
|
||||
return v
|
||||
}
|
||||
v.OffsiteReported = true
|
||||
v.OffsiteEnabled = r.Offsite.Enabled
|
||||
v.OffsiteState = r.Offsite.State
|
||||
v.LastStatus = r.Offsite.LastStatus
|
||||
v.StatsKnown = r.Offsite.StatsKnown
|
||||
if v.StatsKnown {
|
||||
v.Snapshots = r.Offsite.SnapshotCount
|
||||
v.RepoSize = fmtBytesAuto(r.Offsite.RepoSizeBytes)
|
||||
}
|
||||
v.LastSuccess = "—"
|
||||
if t, err := time.Parse(time.RFC3339, r.Offsite.LastSuccess); err == nil {
|
||||
v.LastSuccess = timeAgo(t)
|
||||
}
|
||||
if r.Offsite.QuotaGB > 0 {
|
||||
v.QuotaStr = fmt.Sprintf("%d GB", r.Offsite.QuotaGB)
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
// fmtBytesAuto scales to the unit that keeps a repository size readable. The Offsite page's
|
||||
// fmtBytesGB is deliberately NOT reused here: it is fixed at GB because it renders against GB quotas
|
||||
// where a common unit is the point, and it turns demo-hp's real 140 829 678 bytes into "0.1 GB" —
|
||||
// which on a card whose entire defect was under-reporting a real backup reads as "nearly nothing".
|
||||
func fmtBytesAuto(b int64) string {
|
||||
switch {
|
||||
case b >= 1<<40:
|
||||
return fmt.Sprintf("%.2f TB", float64(b)/float64(int64(1)<<40))
|
||||
case b >= 1<<30:
|
||||
return fmt.Sprintf("%.1f GB", float64(b)/float64(int64(1)<<30))
|
||||
case b >= 1<<20:
|
||||
return fmt.Sprintf("%.1f MB", float64(b)/float64(int64(1)<<20))
|
||||
case b >= 1<<10:
|
||||
return fmt.Sprintf("%.1f KB", float64(b)/float64(int64(1)<<10))
|
||||
default:
|
||||
return fmt.Sprintf("%d B", b)
|
||||
}
|
||||
}
|
||||
@@ -294,6 +294,10 @@ func (s *Server) handleCustomerUnified(w http.ResponseWriter, r *http.Request, c
|
||||
HasReports bool
|
||||
Customer *store.CustomerSummary
|
||||
Report map[string]interface{}
|
||||
// BackupCard (R-331) is resolved in Go, not in the template: the card must distinguish
|
||||
// "no data reported" from "measured empty" from "never measured", and a {{if}} chain over
|
||||
// Report's float64s cannot. See backup_card.go.
|
||||
BackupCard backupCardView
|
||||
OverallStatus string
|
||||
HostCause string // v0.53.0 roll-up: "" or "host down|stale|pending: <id>"
|
||||
|
||||
@@ -428,6 +432,15 @@ func (s *Server) handleCustomerUnified(w http.ResponseWriter, r *http.Request, c
|
||||
}
|
||||
offsiteUnprovisioned := offsiteView.Offsite.Enabled && offsiteView.Offsite.Type == ""
|
||||
|
||||
// R-331: the Backup card reads the report's `offsite` object, not the long-dead `backup`
|
||||
// snapshot/size/integrity fields. Built here so the page holds one resolved view rather than
|
||||
// re-deriving it in template conditionals. A nil customer (never reported) yields the zero view,
|
||||
// which renders as "nothing reported".
|
||||
var backupCard backupCardView
|
||||
if customer != nil {
|
||||
backupCard = reportBackupCard(customer.ReportJSON)
|
||||
}
|
||||
|
||||
data := pageData{
|
||||
CustomerID: customerID,
|
||||
CustomerName: name,
|
||||
@@ -443,6 +456,7 @@ func (s *Server) handleCustomerUnified(w http.ResponseWriter, r *http.Request, c
|
||||
HasReports: customer != nil,
|
||||
Customer: customer,
|
||||
Report: report,
|
||||
BackupCard: backupCard,
|
||||
OverallStatus: overallStatus,
|
||||
HostCause: hostCause,
|
||||
|
||||
|
||||
@@ -0,0 +1,162 @@
|
||||
package web
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-hub/internal/store"
|
||||
)
|
||||
|
||||
// ── R-331 — the Backup card said "Snapshots 0" over a working backup ─────────────────────────────
|
||||
//
|
||||
// Measured on `demo-hp` 2026-08-30: the card read `Snapshots 0 · Repo Size 0 MB · Integrity Unknown`
|
||||
// while the box's own settings held `snapshot_count: 67, repo_size_bytes: 140829678, stats_known:
|
||||
// true` and that night's log said `[offbox] backup OK: 8 app(s) backed up, 67 snapshot(s), 2m14s`.
|
||||
// The card rendered the report's `backup` object, whose snapshot/size/integrity fields have had no
|
||||
// producer since disk-tier restic moved to the host agent; the live numbers were in the `offsite`
|
||||
// object all along.
|
||||
//
|
||||
// These tests use demo-hp's REAL reported values, so a regression fails against the same numbers the
|
||||
// defect was measured against rather than against invented ones.
|
||||
|
||||
// demoHPReportJSON is the shape demo-hp actually sends (controller 0.225.0), trimmed to the objects
|
||||
// the Backup card reads.
|
||||
const demoHPReportJSON = `{
|
||||
"backup": {"enabled": true, "last_db_dump": "2026-08-30T02:15:01Z"},
|
||||
"offsite": {"enabled": true, "escrow_state": "escrowed", "last_status": "ok",
|
||||
"last_success": "2026-08-30T02:17:19Z", "snapshot_count": 67,
|
||||
"repo_size_bytes": 140829678, "quota_gb": 50, "stats_known": true}
|
||||
}`
|
||||
|
||||
func renderWithReport(t *testing.T, reportJSON string) string {
|
||||
t.Helper()
|
||||
s, st := newTestServer(t)
|
||||
if err := st.SaveCustomerConfig(&store.CustomerConfig{
|
||||
CustomerID: "demo-hp", CustomerName: "Demo HP", Domain: "enkisfelhom.hu",
|
||||
RetrievalPassword: "pw", APIKey: "k", Status: "active",
|
||||
}); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := st.SaveReport("demo-hp", []byte(reportJSON)); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return renderCustomerPage(t, s, "demo-hp")
|
||||
}
|
||||
|
||||
// THE CONSEQUENCE TEST. Not "does reportBackupCard return 67" — does the page an operator actually
|
||||
// looks at contain 67. The defect lived in the template's choice of source object, so a test one
|
||||
// layer below it would have been green against the shipped bug.
|
||||
//
|
||||
// RED-PROOF (run 2026-08-30, recorded in REPORT.md): point the card back at `.Report.backup` →
|
||||
// this fails with the rendered page carrying `Snapshots 0` and no `67`.
|
||||
func TestBackupCard_RendersTheRealSnapshotCount(t *testing.T) {
|
||||
html := renderWithReport(t, demoHPReportJSON)
|
||||
|
||||
if !strings.Contains(html, ">67<") {
|
||||
t.Error("the rendered Backup card does not contain demo-hp's real snapshot count (67) — " +
|
||||
"this is the defect: an operator reading this page concludes the customer has no backups")
|
||||
}
|
||||
if !strings.Contains(html, "134.3 MB") {
|
||||
t.Error("the card does not carry the real repository size (134.3 MB from 140829678 bytes)")
|
||||
}
|
||||
if strings.Contains(html, "Repo Size 0 MB") || strings.Contains(html, ">0 MB<") {
|
||||
t.Error("the card still renders the dead `repo_size_mb` field as 0 MB")
|
||||
}
|
||||
// `integrity_ok` has no producer anywhere in the controller — NotifyIntegrityOK and
|
||||
// NotifyIntegrityFailed exist and are called from nowhere. A row that always reads "Unknown" is
|
||||
// not information, and one that could read "OK" from an unwritten field would be a lie.
|
||||
if strings.Contains(html, "Integrity") {
|
||||
t.Error("the card still shows an Integrity row — nothing in the controller produces it")
|
||||
}
|
||||
}
|
||||
|
||||
// The three-way ruling, which is the reason this card needed Go and not a template {{if}}.
|
||||
func TestBackupCard_ThreeWayRuling(t *testing.T) {
|
||||
t.Run("never measured shows a dash, never 0", func(t *testing.T) {
|
||||
// A box with an off-site tier that has never read its repository. Reporting 0 here would
|
||||
// claim the repository is EMPTY, which is not what is known — R-225's exact defect, which
|
||||
// was measured live over a store that really held snapshot f3d9cd67.
|
||||
html := renderWithReport(t, `{"backup":{"enabled":true},
|
||||
"offsite":{"enabled":true,"last_status":"ok","snapshot_count":0,"repo_size_bytes":0}}`)
|
||||
if !strings.Contains(html, "never been") {
|
||||
t.Error("an unmeasured repository is not called out as unmeasured — the operator reads " +
|
||||
"the dash as a formatting quirk rather than as missing knowledge")
|
||||
}
|
||||
if strings.Contains(html, ">0<") {
|
||||
t.Error("an unmeasured repository rendered a literal 0 snapshot count — zero is a claim " +
|
||||
"about the repository's contents and nothing here justifies making it")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("measured empty is stated, not hidden", func(t *testing.T) {
|
||||
// The opposite news, and it must be sayable: the repository was read and really holds
|
||||
// nothing. If this rendered a dash too, the field would be pointless.
|
||||
html := renderWithReport(t, `{"backup":{"enabled":true},
|
||||
"offsite":{"enabled":true,"last_status":"ok","snapshot_count":0,"repo_size_bytes":0,
|
||||
"stats_known":true}}`)
|
||||
if strings.Contains(html, "never been") {
|
||||
t.Error("a MEASURED empty repository was reported as unmeasured — the operator never " +
|
||||
"learns that a customer genuinely has zero off-site snapshots, which is an alarm")
|
||||
}
|
||||
if !strings.Contains(html, ">0<") {
|
||||
t.Error("a measured-empty repository did not render its 0 — measured zero is knowledge " +
|
||||
"and must be shown")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("no offsite object says so instead of showing numbers", func(t *testing.T) {
|
||||
html := renderWithReport(t, `{"backup":{"enabled":true}}`)
|
||||
if !strings.Contains(html, "No off-site data reported") {
|
||||
t.Error("a report with no `offsite` object did not say so — the same rule offsiteBoxTile " +
|
||||
"already follows: absent, never a zeroed tile")
|
||||
}
|
||||
if strings.Contains(html, "Off-site snapshots") {
|
||||
t.Error("numbers were rendered for a customer whose controller has never reported any")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a disabled tier with a declared state names the blocker", func(t *testing.T) {
|
||||
html := renderWithReport(t, `{"backup":{"enabled":true},
|
||||
"offsite":{"enabled":false,"state":"needs_credential"}}`)
|
||||
if !strings.Contains(html, "needs_credential") {
|
||||
t.Error("the declared state was dropped — 'off-site not enabled' and 'off-site blocked " +
|
||||
"waiting for a credential' are different operator actions and must read differently")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// A pre-v0.225.0 controller sends no `stats_known` key. Absence must degrade to UNKNOWN, never to
|
||||
// "empty" — otherwise upgrading the hub before the fleet would tell the operator that every
|
||||
// un-upgraded customer has zero backups.
|
||||
func TestBackupCard_OldControllerDegradesToUnknownNotEmpty(t *testing.T) {
|
||||
html := renderWithReport(t, `{"backup":{"enabled":true},
|
||||
"offsite":{"enabled":true,"last_status":"ok","last_success":"2026-08-30T02:17:19Z",
|
||||
"snapshot_count":67,"repo_size_bytes":140829678,"quota_gb":50}}`)
|
||||
|
||||
if !strings.Contains(html, "never been") {
|
||||
t.Error("a report without stats_known was treated as authoritative — the hub cannot know " +
|
||||
"whether an old controller's counts were ever read, and must say so")
|
||||
}
|
||||
// The rest of the card must still work: last successful run is independent of stats_known.
|
||||
if !strings.Contains(html, "Last successful run") {
|
||||
t.Error("the whole off-site section vanished for an old controller")
|
||||
}
|
||||
}
|
||||
|
||||
func TestFmtBytesAuto_ScalesInsteadOfCollapsingToZero(t *testing.T) {
|
||||
// The Offsite page's fmtBytesGB renders demo-hp's real size as "0.1 GB". On a card whose entire
|
||||
// defect was under-reporting a real backup, that reads as "nearly nothing" — which is why this
|
||||
// formatter exists rather than reusing that one.
|
||||
for _, tc := range []struct{ in int64; want string }{
|
||||
{140829678, "134.3 MB"}, // demo-hp, measured
|
||||
{0, "0 B"},
|
||||
{1 << 10, "1.0 KB"},
|
||||
{1 << 20, "1.0 MB"},
|
||||
{1 << 30, "1.0 GB"},
|
||||
{1 << 40, "1.00 TB"},
|
||||
} {
|
||||
if got := fmtBytesAuto(tc.in); got != tc.want {
|
||||
t.Errorf("fmtBytesAuto(%d) = %q, want %q", tc.in, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -271,28 +271,58 @@
|
||||
{{end}}
|
||||
</section>
|
||||
|
||||
<!-- Backup -->
|
||||
<!-- Backup (R-331: reads the report's `offsite` object; the `backup` object's
|
||||
snapshot/size/integrity fields have had no producer since slice 8C) -->
|
||||
<section class="card">
|
||||
<h2>Backup</h2>
|
||||
{{with .Report.backup}}
|
||||
{{with .BackupCard}}
|
||||
<div class="info-grid">
|
||||
<div class="info-item">
|
||||
<span class="label">Enabled</span>
|
||||
<span class="value">{{if index . "enabled"}}Yes{{else}}No{{end}}</span>
|
||||
<span class="label">App data (local)</span>
|
||||
<span class="value">{{if .LocalEnabled}}Enabled{{else}}Disabled{{end}}</span>
|
||||
</div>
|
||||
<div class="info-item">
|
||||
<span class="label">Snapshots</span>
|
||||
<span class="value">{{index . "snapshot_count"}}</span>
|
||||
</div>
|
||||
<div class="info-item">
|
||||
<span class="label">Repo Size</span>
|
||||
<span class="value">{{index . "repo_size_mb"}} MB</span>
|
||||
</div>
|
||||
<div class="info-item">
|
||||
<span class="label">Integrity</span>
|
||||
<span class="value">{{if index . "integrity_ok"}}OK{{else}}Unknown{{end}}</span>
|
||||
<span class="label">Last DB dump</span>
|
||||
<span class="value">{{if .LastDBDump}}{{.LastDBDump}}{{else}}—{{end}}</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{{if not .OffsiteReported}}
|
||||
<p class="text-muted">No off-site data reported — the controller has never sent an
|
||||
<code>offsite</code> object (no off-site tier, or a controller older than v0.109.0).
|
||||
<strong>This is not the same as "no backups"</strong>; it means this page cannot say.</p>
|
||||
{{else if not .OffsiteEnabled}}
|
||||
<p class="text-muted">Off-site backup is not enabled for this customer.{{if .OffsiteState}}
|
||||
Declared state: <code>{{.OffsiteState}}</code> — the box is waiting on a credential
|
||||
or recovery key and cannot back up off-site until it has one.{{end}}</p>
|
||||
{{else}}
|
||||
<div class="info-grid">
|
||||
<div class="info-item">
|
||||
<span class="label">Off-site snapshots</span>
|
||||
<span class="value">{{if .StatsKnown}}{{.Snapshots}}{{else}}—{{end}}</span>
|
||||
</div>
|
||||
<div class="info-item">
|
||||
<span class="label">Repo size</span>
|
||||
<span class="value">{{if .StatsKnown}}{{.RepoSize}}{{else}}—{{end}}</span>
|
||||
</div>
|
||||
<div class="info-item">
|
||||
<span class="label">Last successful run</span>
|
||||
<span class="value">{{.LastSuccess}}</span>
|
||||
</div>
|
||||
{{if .QuotaStr}}
|
||||
<div class="info-item">
|
||||
<span class="label">Soft quota</span>
|
||||
<span class="value">{{.QuotaStr}}</span>
|
||||
</div>
|
||||
{{end}}
|
||||
</div>
|
||||
{{if not .StatsKnown}}
|
||||
<p class="text-muted">Snapshot count and repository size have <strong>never been
|
||||
measured</strong> on this box — shown as — rather than 0, because zero would
|
||||
claim the repository is empty and that is not what is known. They fill in after the next
|
||||
off-site run reads the repository.</p>
|
||||
{{end}}
|
||||
{{end}}
|
||||
{{end}}
|
||||
</section>
|
||||
{{end}}
|
||||
|
||||
+1
-1
@@ -125,7 +125,7 @@ spec:
|
||||
spec:
|
||||
containers:
|
||||
- name: hub
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.108.0
|
||||
image: gitea.dooplex.hu/admin/felhom-hub:0.109.0
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
name: http
|
||||
|
||||
Reference in new issue
Block a user