admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Record the guards, the narrowing, and the two things I could not do
gates / gates (push) Successful in 30s
Details

Browse Source 
R-273's owed guards are both built and closed. R-291 records what CI stopped
covering and why, so it can be widened deliberately rather than discovered.
R-292 is new and was found by a test failing for the wrong reason:
artifact_sha_invalid conflates "version missing", "registry unreachable" and
"bad sha" into one message. v0.102.0 works around it by ORDERING -- the probes
run first, so an unreachable registry is reported as unreachable -- but the
message itself is untouched.

CONTEXT gains the rule this session is about: a check and the policy it enforces
must read the same number from the same place, or they drift and the drift looks
like a defect in something else. Two corollaries, both of which cost something:
a bounded check must print what it stopped covering on every run, and an
unreadable policy is INCONCLUSIVE rather than unbounded.

Stated in the report rather than glossed: Part 4 (finding receipts for the twelve
downgraded claims) was NOT done and is a shortfall, not a decision -- splitting
it would have produced exactly the half-checked green the exercise exists to
prevent. Part 5 was droppable and dropped. The tag-push green is not re-proved
tonight and is not claimed; the evidence offered is runs 190 and 216.
This commit is contained in:
authentik Default Admin
2026-08-09 19:28:45 +02:00
parent 36bcd12543
commit 985f0ba63c
4 changed files with 184 additions and 167 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT.md
+127 -166
View File
@@ -1,191 +1,152 @@
# REPORT — making the picture true (2026-08-09, unattended)
# REPORT — the guards, and one thing I could not establish (2026-08-09 evening)
Read-only against all live infrastructure. **Both demo machines were powered off and in transit; no
box was probed, woken or waited on.** Claims that only a running box could settle are marked
`needs-hardware`, which is a verdict, not a gap.
No machine touched. Both demo boxes stayed offline and in transit; nothing here needed them.
**Parts 1, 2 and 3 are done. Part 4 and Part 5 are NOT done** — see §6 and §8. Part 5 was marked
droppable and was dropped; **Part 4 was not marked droppable and I ran out of session before it**,
which is a shortfall rather than a decision.
---
## 1. The verdict table
## 1. Part 1.1 — the deleter: **NOT ESTABLISHED**
**55 claims. Statuses moved on 12 of them — all downwards.** `walked 32 → 20`, `built 5 → 17`;
`partial 14` and `missing 4` unchanged. Full per-claim detail with sources is in
`documentation/architecture/where-felhom-stands.yaml`.
**This prompt's account is not corroborated by any source I can reach**, and the prompt itself said
not to take its word for it.
| claim | was | now | why |
| claim in the prompt | what I found |
|---|---|
| "the register records … pruned to the newest ten … 33 deletions, all HTTP 204" | **No such row exists.** The only prune-adjacent row is **R-210**, which is `WAITING-ON-OPERATOR`, says in terms *"Nothing was deleted; this is a list, not an action"*, and concerns **local Docker images on DooPlex**, not the Gitea registry |
| "newest ten per package" | **Not visible in the current state.** `felhom-agent` generic holds 10, but `felhom-controller` and `felhom-hub` container packages hold **19 each** |
Everything else I could reach is silent, and silence here is not evidence of anything:
`package_cleanup_rule` is **empty**; `package_version` has **no soft-delete column**, so a deletion
leaves no row; Gitea's `action` feed carries **no package operation at all** across 2026-08-08 →
2026-08-09 and nothing whatever on the evening of 08-08; the Gitea pod has **53 days uptime, 0
restarts**.
**And I withdrew one of my own claims.** R-287 said *"no DELETE on the packages API appears in 48 h of
Gitea router logs"*. Re-checked: `kubectl logs --since=72h` returns nothing older than **2026-08-09
16:35** and contains **zero** `api/packages` lines even for requests I made myself. **The log never
covered the window, so its silence was never evidence.** Withdrawn in the register.
**What remains established, unchanged:** `v0.120.0` downloadable at 2026-08-08 14:29 UTC (run 267
printed `ok v0.120.0`), 404 by 2026-08-09 09:30 UTC (run 284), and `0.128.0` published at 14:47 UTC —
eighteen minutes after the good run. **It may be unestablishable from this side: Gitea keeps no
package-deletion trail.**
Per §8 I halted that part's attribution and carried on; the rest is independent.
## 2. The number, and the two places that read it
`felhom-agent/scripts/retention-policy.json` → `generic_versions_kept: 10`.
Read by **`scripts/check-published-versions.py`** (bounds its assertion) and referenced by the prune
procedure. The file states, in its own header, that ten is an **observed state and not a located
ruling**, and that the principled bound is the hub's vouched `min_agent` floor — nothing can install
below it — which needs network the gate does not have.
**What CI no longer covers:** *a released version older than the retention window is no longer
asserted downloadable.* Its **git tag and config tree are still asserted**; only the binary's presence
is dropped. The check **prints the dropped versions every run**:
```
11 released version(s); retention policy keeps the newest 10
NOT ASSERTED (older than the retention window …): 0.120.0
^ these versions still have git TAGS … what is no longer asserted is the BINARY's presence.
```
Three controls: green at 10 naming what it dropped (**exit 0**); widened to 11 the evicted version
re-enters and convicts (**exit 1**, `FAIL v0.120.0`); the policy file removed → **exit 2 INCONCLUSIVE**,
naming the path it tried — never silently unbounded.
## 3. `felhom-agent` green
```
reuse-refs OK · instructions OK · published OK · release-complete OK · all agent gates OK
```
**At `main`: green**, pushed as `53d047a`, and the pre-push hook ran the same entry point.
**At a tag: not re-proved tonight, and I will not claim it.** The available evidence is that the gate
is ref-independent — it enumerates from the Gitea tags API, and runs **190** (`v0.126.0`) and **216**
(`v0.127.0`) were tag pushes that passed. Minting a tag purely to prove it would have published a
release, which this session forbids.
## 4. Red-proofs
| # | mutation | asserted applied | outcome |
|---|---|---|---|
| `install.installer-by-tag` | walked | **built** | gate 6 asserts the manifest names an installer tag; no walk of a rollback on file |
| `use.lifecycle` | walked | **built** | no walk document cited |
| `drives.enrol` | walked | **built** | the 08-09 walk exercised RE-attach (which failed, R-280); first enrolment of a NEW drive has no walk |
| `drives.migrate` | walked | **built** | no walk document cited |
| `backup.tier1` | walked | **built** | no walk document cited |
| `backup.whole-machine` | walked | **built** | no walk document cited |
| `backup.restore-proof` | walked | **built** | no walk cited, **and the last recorded restore-test on demo-hp FAILED** (2026-08-05) |
| `fault.selfheal` | walked | **built** | no walk document cited |
| `fault.operator-email` | walked | **built** | source-verified as correct, but no run observed delivering |
| `fail.drive-filling` | walked | **built** | no walk document cited |
| `fail.lost-recovery-code` | walked | **built** | by-design refusal; no walk document cited |
| `fail.hub-down` | walked | **built** | no walk document cited |
| 1 | **tag check removed** | `grep -c` → 1 | **scenario A FAILED, reporting `flash = "artifacts_set"`** — Friday's exact defect returned: the manifest saved with no tag |
| 2 | package check removed | `grep -c` → 1 | scenario B FAILED, `flash = "artifacts_set"` |
| 3 | inconclusive branch mapped onto success | `grep -c` → 2 | scenario E FAILED |
**Upgraded: 1.** `install.byo` — the page said *"the first real one has not happened"*. A real
`--mode byo` install completed on demo-hp on 2026-08-09 (`Day-0 provision SUCCESS`, 3 m 49 s). Still
not a customer's own hardware, so not *walked*, but the sentence was false.
All reverted; `grep -c MUTATION` → **0**; all seven tests green again.
**`needs-hardware`: 4** — `use.lan-fallback`, `backup.restore-proof`, `fail.disk-failing`,
`fail.internet-down`. Each needs an observation on a running box; each says which.
**Red-proof 3, stated precisely rather than flatteringly:** with the inconclusive branch mapped onto
success the save did **not** complete — it fell through to `artifact_sha_invalid`. So the mutation
proves the guard is load-bearing for *the message the operator sees*, not for the save itself. That is
the honest reading, and it is exactly the defect scenario E exists to prevent: the operator being told
"missing or invalid" when the truth is "I could not reach the registry".
**Confirmed: 38.** Seven of those were re-confirmed against live source or the live hub tonight rather
than against paperwork: the tripwire, the off-site repository, the claim path, the catalogue, the
tunnel, the reset code and the operator-email digest.
## 5. The five scenarios, as the operator sees them
## 2. Every downgrade, with the coupling that broke
| | outcome | the message |
|---|---|---|
| **A** tag missing | REFUSED | *"Refused: that version has no usable git tag. The installer fetches an agent's config files from `raw/tag/v<version>/configs/`, so a version published without its tag makes every fresh install and reinstall fail at step 5 of 8 — as root, on a virgin machine. … Fix it by pushing the tag: `git tag -a v<version> <released-commit> && git push origin v<version>`"* |
| **B** package pruned | REFUSED | *"Refused: that version's artifact is not downloadable. The version is tagged but its package is not in the registry, so a box would 404 fetching the binary itself. … Publish it — `bash scripts/release-agent.sh <version>`"* |
| **C** no checksum | REFUSED | the pre-existing *"Couldn't set the checksum …"* — see the note below |
| **D** both good | **SAVES**, `artifacts_set`, byte-identical behaviour | — |
| **E** registry unreachable | REFUSED | *"Refused: could not verify — this does not mean anything is missing. … It refuses rather than saving with a warning, because a warning beside a success reads as a success. … There is deliberately no override: the registry is on your own server, so if it is unreachable the vouch can wait."* |
The rule is *"a proof is about the code that existed when it ran"*. **It did not fire the way the task
expected.** Not one downgrade came from code moving under an old proof. **All twelve came from step 1
of the same rule — the cited evidence does not exist.**
**Scenario C changed shape because my first draft modelled nothing real,** and that is worth recording.
With a Gitea client configured, `resolveArtifactSHA` fetches the sha **authoritatively and ignores what
was submitted** — so "submit an empty sha" cannot produce an empty stored sha. The genuine shape is
Gitea answering with no `sha256`, and the **existing** refusal already owns it. The test now pins the
guarantee (*the manifest is unchanged*) rather than a mechanism I had invented.
Measured: of the 28 capability-map rows behind the page's claims, **8 carry a `tests/` or `audits/`
path in their evidence column and 20 carry prose only.** The green dots were being drawn from rows
that cite an argument, not a walk. Filed as **R-290**.
**No override was built, and none is wanted.** §8's halt condition did not trigger.
**And the decay ran the other way once.** `fault.operator-email` — *"one mail per run, every failing
app named"* — I first took to be contradicted by R-182 (open, *"tells the operator about ONE app and
silently swallows every other"*). Reading live source: the digest `backup_run_failures` is allowlisted
(`hub/internal/api/handler.go:1837`), operator-only (`notify/dispatcher.go:423`) and templated
(`notify/templates.go:48`); `recovery_unit_capture_failed` is record-only (`dispatcher.go:376`); a
cooldown drop now logs a `suppressed` row (`dispatcher.go:314-330`). **The claim is right and the
register row is stale** — filed as **R-289**. The session went looking for stale proofs and found a
stale defect.
## 6. Part 4 — NOT DONE
## 3. The positive control
The twelve downgraded claims were not re-examined and no receipts were searched for. **The honest-grey
count is therefore still the twelve from last night, unverified in either direction**, and
`where-felhom-stands.*` is untouched. This is the session's shortfall: Parts 1–3 took the budget, and
splitting Part 4 in half would have produced exactly the kind of half-checked green the whole exercise
exists to prevent.
```
1 BASELINE real dataset -> OK, exit 0
2 PLANT scratch copy: use.dlna missing -> walked -> CONVICTED, exit 1
"use.dlna: status 'walked' but NO evidence document cited"
3 REMOVE scratch copy deleted; committed dataset never touched
4 RE-RUN real dataset -> OK, exit 0
```
**Whether the "code moved under the proof" rule can fire at all** is therefore still open from last
night, where it fired **zero** times out of twelve — every downgrade came from missing evidence, not
from decayed evidence. My reading remains that it *can* fire but will stay rare until rows cite
evidence at all, which is R-290.
Plant → convicted → removed → clean. The gate also convicted **51 problems in my own first draft** of
the dataset (bad anchors, register ids that are not in `OPEN-ITEMS.md`, an evidence path that does not
exist) before any of this — which is the more convincing demonstration, because it was not staged.
## 7. Hub deployed
## 4. The two known disagreements — both settled, and neither document was wrong
**v0.102.0**, live and verified: deploy image `gitea.dooplex.hu/admin/felhom-hub:0.102.0`, rollout
complete, and the page footer reads `0.102.0`. Manifest commit **`36bcd12`**; code commit `b55fc17`.
The image was verified **served by the registry before** the manifest was bumped, not after. ArgoCD's
`felhom` app has `automated.enabled: false`, so the sync was triggered explicitly — **no
`kubectl set image` at any point.**
**"A customer restores their own data with no help."** The map says **MISSING (as evidence)**; the
2026-08-07 walk records a customer route completed with no shell. **Not a contradiction.** The map's
row is *"A customer (**not the operator**) performs a restore via UI alone"* — it is about *who*. The
walk proves the *route*. No non-operator has ever done it, which is what the page's own neighbouring
claim already says.
## 8. Part 5 — dropped, as marked
**The reinstall story.** The map's `PROVEN-LIVE (2026-08-04 night drill)` row is scoped in its own text
to *"a controller-data-volume rebuild — NOT a total host loss"*. The 2026-08-09 rehearsal was a
whole-host uninstall and reinstall. **The map has no row for that case at all** — a gap, not a
disagreement.
Not started. It was explicitly droppable and it is the only part that touches customer-facing wording,
which is where a rushed edit does most harm.
**Would anything here have caught either one? No — and it could not have, because neither was false.**
Both are collisions of vocabulary: "customer" meaning *the route* or *a person*, "rebuild" meaning
*the guest* or *the host*. No gate detects an ambiguity that makes two true sentences look
contradictory. They surfaced only when someone tried to state them side by side. **That is the
argument for the dataset** — one id, one scope, one status — and against prose rows.
## 9. Register
## 5. The data file
`documentation/architecture/where-felhom-stands.yaml`, 55 entries. **Every entry cites at least one
source and the gate proves it** (`check_stands.py` rule 1). YAML rather than JSON because statuses move
one line at a time and a YAML diff shows which claim moved; a JSON re-dump reflows.
Rules honoured: it is a **view** (every entry cites map / register / evidence); **no status was raised
in it** — the one upgrade is recorded against evidence and the map is named as the thing that must
change; and it is **regenerated, not hand-edited** for the page.
## 6. The page
- `where-felhom-stands.html` — **generated, 54 KB, zero `<script>` tags**, same palette
(`#0b1220` / `#121b2c` / `#34d399` `#60a5fa` `#fbbf24` `#64748b`), 1600 px, A3 landscape print rules.
- Every moved status carries a visible **`changed 2026-08-09, was walked`** chip plus a *why it moved*
line — 12 of them, no diffing required.
- `Where Felhom Stands.html` → **`where-felhom-stands-2026-08-09-snapshot.html`** (`git mv`, so the
space is out of every shell path), with a line in `documentation/README.md` calling it a dated
snapshot that is not maintained.
- **What the old bundle actually was**, since it aimed the fix: not merely minified — the content sat
**gzip+base64 inside a JS module map**, and the three blobs decompress to the bundler and React, with
the document itself in a JSON-escaped string on line 393. It rendered and nothing else.
## 7. The corrected silence rows
**R-281 is WITHDRAWN. It was wrong in both directions**, and the operator's mailbox is what settled it.
- **The tripwire DID fire**: `escrow_blob_served` at 10:19:41 UTC = **12:19 CEST**, eight minutes before
the verified restore.
- **False alarms fired too**: `host_down` 09:28 UTC and `node_down` 09:30 UTC, both error severity, both
`sent`, for deliberate attended work — eight operator mails in all. → **R-285**, the opposite gap
from the one filed.
**The measurement's cause IS established.** The P7 query copied `/data/hub.db` **without
`hub.db-wal`**; the hub runs SQLite in WAL mode, so everything after the last checkpoint was invisible.
**Signature, exact:** P7 reported *"2 events all day, newest `db_dump_completed` 00:30:07"*, and the
number of rows on 08-09 at or before 00:30:07 is **exactly 2**.
The two obvious alternatives were **tested and refuted**, not waved away: a **timezone offset** — all
nine mailbox stamps equal the hub's UTC + 2 h exactly, so the window was right; and a **wrong key or
wrong store** — the same table and key return the correct rows now. A live re-run **cannot** reproduce
the fault because the WAL has since been checkpointed, and that is stated rather than dressed up as a
reproduction.
**The lesson, filed as R-286:** the control was drawn from the *same stale snapshot* as the
measurement, so it agreed. **A control must come from a different channel.** The independent channel —
the mailbox — was available the whole time. This is also a trap `operations/nodes.md` already
documents, and which I had avoided correctly earlier in the same session.
## 8. Register
**Ceiling moved R-284 → R-290.** Opened: **R-285** (planned reinstall pages the operator), **R-286**
(same-channel control), **R-287** (the CI red is true), **R-288** (the capability map is unreadable),
**R-289** (R-182's row is stale), **R-290** (map rows cite no evidence). Withdrawn: **R-281**.
## 9. Part 4 — and I did not change the workflow, on purpose
**Which gate:** `published` / `scripts/check-published-versions.py`.
**The premise is wrong in every particular. It is not ref-sensitive.** The gate enumerates releases
from the **Gitea tags API** (`main()`, `/api/v1/repos/admin/felhom-agent/tags?limit=200`), so the
checked-out ref is irrelevant — and the two previous tag pushes **passed** (run 190 `v0.126.0`, run 216
`v0.127.0`).
**What is true:** run 267 (main, `28ba8593b8`, 08-08 14:29 UTC) printed `ok v0.120.0: binary
downloadable`. Run 284 (**the same commit**, on the tag, 08-09 09:30 UTC) printed
`FAIL v0.120.0 — HTTP 404`. A published release became uninstallable between the two. The registry now
holds exactly the ten newest versions; `0.128.0` was published **14:47 UTC, eighteen minutes after run
267**.
**Who removed `0.120.0` is NOT established, and I will not guess:** `package_cleanup_rule` is empty
(queried in Postgres), `app.ini` sets no limit, `publish-agent.sh:77` only pre-deletes the version it
is publishing, the Gitea pod has 53 days uptime and 0 restarts, and **no `DELETE` on the packages API
appears in 48 h of router logs**. The internal `[cron.cleanup_packages]` `@midnight` job falls in the
window and would leave no router line — **a leading candidate, not a conclusion.**
**So nothing was silenced and no workflow file was changed.** The red is a **true positive** — a tagged
version that cannot be installed is the exact R-115 defect the gate exists to catch, and muting it
would hide the next one. **What it therefore still does not check: nothing. Nothing was disabled.**
The honest fixes — bound the gate to versions at or above the vouched `min_agent` floor (0.127.0 today;
nothing can install 0.120.0), or retire tags whose packages go — are release decisions, and §8 forbids
fixing findings here. Filed as **R-287**.
**Also measured, and it is good news:** the failure alarm did send —
`RESEND-ACCEPTED id=fa1a7a83-714f-4357-b0ca-d3c4bb7ae73f`.
**Ceiling R-290 → R-292.** Opened **R-291** (the narrowing, with its reason, so it can be widened
deliberately) and **R-292** (`artifact_sha_invalid` conflates three facts — found by scenario E
failing, worked around by ordering, message untouched). Closed: **R-273's owed-guards tail**, both
guards built. Corrected: **R-287**, twice.
## 10. Observations — noticed, not acted on
- **`fail.app-crash` and `fault.operator-email` are the same unobserved thing** seen from two sides:
the digest is wired and correct in source, and no one has watched it arrive.
- **The verification depth is recorded per claim** (`depth: source-read | register+map |
needs-hardware`). 23 of 55 got a source or evidence read; the rest were checked against the register
and map only. That is on the face of the data rather than implied by a green tick.
- **The capability map is the thing that actually needs fixing.** The dataset now disagrees with it for
twelve rows, and the dataset is only a view — **the map still says PROVEN-LIVE for all twelve** (R-290).
- `documentation/audits/` holds 131 files and `tests/` 37; the evidence exists in quantity. The gap is
that the map does not point at it.
- **Out of scope and left alone, as instructed:** every finding above, the capability-map restructure
(R-288), and the two guards owed from yesterday.
- **R-292 is the interesting one.** A test I wrote to check a new guard failed for a reason that had
nothing to do with the guard, and that reason was a real pre-existing defect. The five scenarios
earned their keep before the feature shipped.
- **The golden gets no tag probe.** It is fetched by version and has no config tree, so a tag probe
would assert something the installer never does. Deliberate, and stated in the code.
- **The gate is skipped entirely when no Gitea client is configured**, or a hub without registry
credentials could never vouch anything. Pinned by its own test, and worth knowing: the guard is only
as present as the client is.
- `check-release-complete` runs in `--fast`, so a missing tag is now caught by the **pre-push hook**,
earlier than CI.