admin
a7f897a7dd
gates / gates (push) Successful in 6m24s
Read-only. fetch.py + tables.py (stdlib), TABLE.md (45 charts, % of respondents), FELHOM.md (apps, features, reasons, hardware, trends, ten ranked recommendations), METHOD.md (hashes, counts, matching, MIT licence, limits). Survey files are not committed. One register row (R-989) and one STATUS line ask the operator to rule on the recommendations.
225 KiB
225 KiB
OPEN-ITEMS — the single source of truth for open work
Rebuilt 2026-07-27 by read-only triage. ROADMAP.md keeps the full history and reasoning; this
page keeps only what is open, and it is the file to read first. Root REPORT.md is per-session
and overwritten — nothing durable may live only there; a session that must not clobber it writes
a non-overwritten REPORT-<topic>.md sibling instead (CLAUDE.md:82-87), of which 14 now exist.
How a row is filed (2026-10-03 — the triage; operator may reverse the scale and the list)
One table shape: | ID | Category | Sev | What | State | Blocked on | Next action | Owner |.
scripts/register_shape_gate.py refuses a row that does not split into these eight cells (a literal |
belongs inside backticks), whose Category is not one of the eleven below, or whose Sev is not
P1–P4. A new row goes into its category's section, in severity order.
Fix small, do not file (the size rule — operator brief 2026-10-05, the burn-down). The register grew because every
session closed a few rows and filed a few small new ones. So: a finding that is cosmetic or small — fixable in the
session in about 30 minutes, in a repo the session may change — is FIXED in that session, with a test where it changes
behaviour, and NOT filed. It is recorded in that repo's CHANGELOG.md and in the session report under „fixed without
a row". Only a finding that needs a decision, a design, a larger build, or a change the session may not make (a
protected machine, a repo out of scope, a release budget already spent) becomes a row. This narrows — it does not
repeal — „an enumerated gap becomes a row": a small gap leaves the session fixed, which is a record too.
Every session report states four numbers: rows before, rows after, rows opened, rows closed — counted the way
register_shape_gate.py counts (| **R-n** | lines in this file).
Sev — one scale for this file and ROADMAP.md:
- P1 — now. A household can lose or leak data, a box can stop or be taken over, or a promise we make is false — today. A P1 row says in one sentence what the harm is.
- P2 — before the first paying customer. Must be done before anyone pays (business and legal work too).
- P3 — during the first customers. A real defect or gap a customer may meet, with a workaround or a low chance.
- P4 — later / nice to have. Ideas, polish, tooling comfort, post-alpha features.
Older tags stay readable in the row text: P2-HIGH/P2-MEDIUM → P2, P3-LOW → P3. The Sev cell is the
rank. A row whose rank moved says so in one dated sentence at the end of its State.
Category — eleven, following architecture/00-capability-map.md's sections where they fit:
Install & onboarding · Apps & catalog · App updates · Backup & restore · Storage & devices · Security & access
(logins, gates, the tunnel, secrets) · Box system & updates (host, guest, OS, controller/agent self-update,
golden) · Monitoring & notifications · Hub & operator · Business & legal · Process & tooling (gates, docs,
the workflow).
State leads with one of: READY · OPEN · BLOCKED · WATCHING · WAITING-ON-OPERATOR · NARROWED ·
DEFERRED · VERIFY (= may already be finished; check against live source before closing). Every row has an
Owner. A row whose state LEADS with a finished word (CLOSED, SHIPPED, FIXED, DECIDED, …) does not belong
here — it moves to CLOSED-ITEMS.md in the same commit that finishes it;
scripts/closed_register_gate.py RULE 3 refuses the push otherwise.
2026-10-03 — the triage. 125 finished rows moved to CLOSED-ITEMS.md (and 20 old rows that never had an
id); the campaign write-ups, the 2026-08-04 rulings and the old ranking paragraphs that sat between the tables
moved, word for word, to documentation/archive/OPEN-ITEMS-narratives-2026-10-03.md. The full previous text
of this file: git show 9e2786c:documentation/backlog/OPEN-ITEMS.md. What to work on next:
documentation/audits/backlog-triage-2026-10-03/RECOMMENDATION.md.
DECIDED — the backup and restore arc is CLOSED FOR BETA (2026-09-01)
Written in the voice this file uses for a settled decision: what was decided, and the condition
that reopens it. It exists because nobody ever said the arc was finished, and an arc nobody closed
gets picked up again in a month by someone who reads the blanks in 07 §8 as unfinished work.
CLOSED FOR BETA at controller v0.232.0 / hub v0.111.1 (2026-09-01).
What is finished, and proven live: everything a customer does for themselves — losing files,
losing an app's data, losing a whole app, losing a drive. The restore states what it returned,
refuses without room, cannot be fed a part-copy, and puts the customer's own data back if it fails.
The second drive's copy is a route. A poorer copy cannot delete a richer one. The off-site store is
verified weekly at full depth and proved nightly to still contain something.
Rows 1, 2, 3, 3b, 3c, 6, 7 and 14 of 07 §8 — every one PROVEN.
What is deliberately deferred until after beta — [BETA-DEFERRED], and these are the row numbers,
not a description:
07 §8 row |
the failure | status today |
|---|---|---|
| 4 | primary drive dies — the drive-loss journey (the route is proven; no disk has ever died under it) | PARTIAL |
| 8 | host dies, drives intact — a host rebuilt as itself | IMPLEMENTED, never executed |
| 9 | whole box lost (fire/theft) | IMPLEMENTED / UNPROVEN |
| 10 | ransomware / malicious deletion — a ransomware-shaped recovery | PARTIAL |
| 11 (and 11b) | hub lost — a hub restore | UNPROVEN; it has never been performed |
| 12 | off-site provider lost (Hetzner) | [FACT] only |
These are real, they are recorded, and none of them is a beta blocker. Every one is invocable by the operator, not the customer; every one needs hardware, a provider, or a destructive rehearsal that beta does not.
A NUMBER IN THE BRIEF FOR THIS SECTION WAS WRONG AND IS CORRECTED HERE. It said "six rows of §8 still have no measured time". Six rows are DEFERRED; ELEVEN rows carry a blank RTO — counted, not estimated: 4, 5, 8, 9, 10, 11, 11b, 12, 13, 14, 15. The other five are blank for reasons that are not deferred work, and collapsing them into one number is how a blank stops meaning anything:
- row 5 —
PROVENby construction; the secondary is a derived copy, so there is no recovery to time. - row 13 —
NONE for host-lossby design; R exists in zero system copies, so there is no route. - row 14 —
PROVEN; the break-glass route works and has simply never been stopwatched. - row 15 — an open DEFECT (R-104, the stale-lock path), not a deferred recovery. It is NOT inside this stopping line and must not be read as parked by it.
- row 11b — a consequences note attached to row 11, not a recovery row of its own.
What is NOT deferred and stays open: R-95 and R-433, both BLOCKED-ON-PROVIDER behind
documentation/runbooks/provider-questions-2026-09-01.md; R-435, documentation only; and
R-104, row 15's defect. The alarm text (R-434) was fixed the same day and is closed.
THIS REOPENS IF: a customer-facing recovery path is found broken; or Hetzner's answers change what the snapshots are worth (either answer in Part 3's file can do it — "full restore only" makes row 10 urgent, "append-only enforced" makes R-95's root cause cheap to remove); or a real customer's data is at stake in one of the deferred rows.
THE MARKER. Every deferred row carries the literal string [BETA-DEFERRED] in 07 §8, so
grep -n '\[BETA-DEFERRED\]' documentation/architecture/07-backup-architecture.md returns the set
as a group. It returns EIGHT lines, not seven — the seven tagged rows plus the one line in §8's
own header that defines the marker. That is stated rather than hidden, because a count that does not
match what the reader sees is how an instrument stops being believed (R-421). It is a marker, not a status — the status cells are unchanged, because
nothing was proven on the day this line was drawn and a stopping line that moves a status is a
stopping line that lies.
Install & onboarding — 5 rows (P3 3, P4 2)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| R-980 | Install & onboarding | P3 | A person cannot check that the installer is ours: the ISO's first boot runs the install script from a movable tag with no hash or signature, and the ISO checksum sits on the same website as the download. Fix: a signed manifest (or signature) checked by the ISO before it runs the script, and the public key printed in the guide. Security review 2026-10-12 (audits/security-review-2026-10-12/); detail held off-git for the operator (/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/). |
OPEN — design-proposal | — | Design note, then build | CC |
| R-250 | Install & onboarding | P3 | A customer create can fail fail-closed because the host-key scan ladder is shorter than the DNS/AAAA settle time. Found 2026-08-07 creating the fifth walk's venue. POST /configs/new with off-site enabled provisions the Storage Box sub-account and then scans its SSH host key to pin it — fail-closed by design (offsite.go:111-121, "don't serve a descriptor the controller can't verify"), with defaultScanBackoff = 2+4+8+16+30 ≈ 60 s, sized by its own comment to "the observed DNS propagation lag". Measured, both halves: the first create exhausted the ladder — five no such host, then dial tcp [2a01:4f8:bacc:2:200::d30]:23: connect: network is unreachable (the name had just begun resolving, AAAA-first, into a pod with no IPv6 route) — and the hub logged [ERROR] offsite provision for walk5. An identical second POST succeeded ~70 s later on its own final rung (shared already provisioned … (subaccount 285351)). Total settle ≈ 100 s against a 60 s budget. The endpoint was never the problem: verified afterwards from both the node and the hub pod, by name, SSH-2.0-OpenSSH_9.6p1. Two distinct things are wrong and should not be merged: (1) the budget is sized against DNS existence, but what actually bit is the AAAA-before-A window, a different and longer phenomenon — lengthen the ladder and/or prefer the A record for the scan dial; (2) the operator is told nothing actionable — the create fails with a generic error, the remedy is "press it again", and nothing says so. The retry is genuinely safe (idempotent on the label; confirmed afterwards — one_time_secrets 1, one sub-account, no double-provision) but that safety is invisible to the person deciding whether pressing again will double-charge them. Severity LOW-MEDIUM: self-clearing, no data at risk — but it is the first thing a new customer's provisioning does and it fails looking like an outage. |
VERIFY — 2026-10-12: hub 0.147.0 live (S29, decision 224): the host-key scan dials IPv4 first, waits ~2 min (was ~1), and when it still runs out says the store was created and it is safe to press Save again (TestScannerDialsIPv4First, TestScanOutOfRetriesSaysSafeToPressAgain, red-proved). Known limit: the hub Ingress reads for 60 s, so a very slow first try can show a gateway timeout while provisioning finishes in the background (pressing again is safe). READY — operator ruling 2026-10-12 (S29, 09 §3 decision 224): yes — the scan prefers IPv4, waits longer, and the error says „safe to press again". READY — owner Viktor |
— | The next real off-site setup (a new customer): read the hub log for the scan; success or the new message → close | CC |
| R-516 | Install & onboarding | P3 | [P3-LOW] English a customer meets on a fresh box and its apps' first screens — enumerated by the big night. MEASURED 2026-09-14 (BIGNIGHT, VM 333, ISO 1.27.1, controller 0.242.0). Felhom-owned: (1) the dashboard menu item „Debug"; (2) the dashboard CPU tile „Load: 0.29 / 0.39 / 0.37"; (3) the launcher tile „Filebrowser" opens a login in English with no Felhom text (R-513); (4) the storage page mixes formal „Adjon hozzá / Csatlakoztasson" with the product's „te". App first screens a household meets before any Felhom text helps: (5) Uptime Kuma 2.4 opens on „Which database would you like to use?" (SQLite / Embedded MariaDB, „Next") — the app card's „Első lépések" does not mention it; (6) PrivateBin, Gokapi, AdventureLog and FileBrowser UIs are English (the apps' own). Already rows: the Proxmox installer screens (R-495, answered by the guide), wiki.DOMAIN (R-498). Fix shape: rename „Debug"/„Load" (controller); add the Uptime Kuma database step to its card, or pre-seed db-config.json for SQLite in the template (catalog). Added by F4 (20:04:54Z): (7) the storage page prints the disconnect time as a raw ISO UTC string „Leválasztva: 2026-09-14T19:58:02Z"; (8) the „Meghajtó leválasztva" banner appears twice on every page; (9) „4 telepített alkalmazás nem fut — nézze meg a rendszermonitort" uses the formal form. Added by F7 (20:50–21:00Z, system disk at 95 %): (10) a banner on every page in English, „SSD disk usage high: 90%"; (11) the dashboard tile reads „Rendszer (/) 61.8 GB / 68.7 GB (90%)" while df reports 95 % (reserved blocks ignored), and „(/)" labels the data volume /mnt/sys_drive; the deploy page says nothing about free disk. Added by the i18n spike (2026-09-17, controller v0.247.0): (12) six formal („ön") forms in the converted dashboard copy — „Olvassa be telefonnal" (launcher QR hint), „Biztosan kikapcsolja a megosztást?" (launcher), „Ha újratelepíti … importálnia kell" (the layout's remove-app modal), „Kérjük, vegye fel a kapcsolatot" (backups empty state). They were NOT fixed — a localisation release may not change Hungarian bytes — and are now COUNTED by controller/scripts/i18n_missing_gate.py (HU_FORMAL_CEILING = 6, a ratchet: a new one convicts, fixing one lowers it). The inventory (audits/I18N-INVENTORY-2026-09-17.md) is the list this row closes against in localisation slice 6 (R-561). Extended 2026-09-17 by slice 1 (controller v0.248.0–v0.250.0): the converted copy now counts 16 formal forms (HU_FORMAL_CEILING 6 → 10 → 12 → 16, each raise stated, Hungarian unchanged by rule) — and the count is an UNDER-count: the gate's stem list sees 4 on the release C pages (storage, drive wizards, debug), while a wider list („írja be”, „adja meg”, „adjon hozzá”, „válassza ki”, „biztosan eltávolítja”, „engedélyezze”, „hozzon létre”, „kattintson”) finds at least 22 keys there alone (login „Adja meg a jelszavát”, the NAS guide, the storage confirms). Widen the stems when this row is worked; the ceiling rises with them. NARROWED 2026-09-20 by localisation slice 6's walk, item by item (audits/i18n-slice6-2026-09-20/R-516-item-by-item.md): item 2 FIXED (slice 1), items 3 and 6 APP-OWNED (and for an English household item 6 is an advantage), item 11 is a wrong NUMBER rather than a language and needs its own row, item 1 is correct English for an English reader and still open for a Hungarian one. Items 4, 7, 8, 9 and 10 are NOT DECIDABLE FROM AN ENGLISH WALK - they are about Hungarian copy quality, a disconnected drive and a full disk, none of which this walk had. What this row is now waiting for is a HUNGARIAN walk on a box with a second drive, not another English one; saying it closed would be the kind of closure that makes a register stop meaning anything. -- 2026-10-08: the recovery screen's own answers are Go literals in Hungarian only — 14 renderRecovery messages in controller/internal/web/recovery_handlers.go (seen: the safe default „A művelet nem fejeződött be…" rendered on an English page in a red-proof run). Today's new R-304 sentence is a bundle key in both languages; the 14 old ones are not. |
NARROWED 2026-09-20 - rank P3-LOW; owner: CC; needs a Hungarian walk 2026-10-05 (burn-down night): items 1 and 4 FIXED on controller main (d7efa7b): the menu says „Hibakeresés” (English „Debug”); the storage pages use the te-form (formal-form ceiling 17 → 14; 116 parity fixtures changed by exactly those bytes). Item 2 was already fixed. Left: items 5–6 (apps' own screens) and the rest. 2026-10-06: items 1 and 4 DELIVERED (controller v0.298.0). 2026-10-06 (burn-down night, later): items 7–10 and 12 FIXED on controller main (e4774e6, 2e5ef7e): the disconnect time in local time; one banner per unplugged drive; „nézd meg”; the disk/memory/CPU/temperature banners in the household's language (health.* keys; the wire text to the hub unchanged and pinned); the last eleven counted formal forms are te-form (ceiling 14 → 0; 119 parity fixtures changed by exactly those bytes). Left: items 5–6 (apps' own screens); about 20 formal forms the gate does not count yet („írja be”, „adja meg”, „hozzon létre”, …) on the debug, storage, network-storage and security pages. Item 11 moved to its own row R-889. 2026-10-06 02:40: the bundle has NO formal form left (controller 4c3c203, unreleased): 49 sentences in the te-form; the gate's stem list widened (it counted 61 on the previous bundle, 0 now) with decoys and one listed third-person exception. Left (narrowed): Felhom-owned formal forms in Go and template literals the gate cannot read yet — the network-storage attach errors, escrow and share handlers, the single-copy notice, a settings refusal, the SMART mail, a fill-watch „Kérjük”, and the setup wizard pages; each moves into the bundle in the te-form. Items 5–6 are the apps' own screens (not ours). 2026-10-06: items 7–10, 12 and the bundle's 49 formal forms DELIVERED in controller v0.299.0. 2026-10-06 night: the Go-literal leftover moved (controller main 3d6ba28, unreleased): 22 Go-literal messages — escrow handler (6), share password page (2), export upload (1), network-storage uid/remove/attach failures (13) — are bundle keys (api.escrow.*, api.share.*, api.export.*, api.netstorage.*) in the te-form with English (14 were formal); a detached attach failure renders in the reader's language; the NAS refusal says „Válassz". TestR516_GoLiteralFormalFormsAreGone (source scan + positive control), TestR516_NetAddFailureFollowsTheReader; red-proofs audits/night-burndown-2026-10-06/ctrl/R-516-red*.txt. Left: the setup wizard and recovery-info.txt (R-554); the SMART and fill-watch texts (the hub event's message — wire text, needs the household-copy producer shape); items 5–6 (apps' own screens). |
— | — | CC |
| R-494 | Install & onboarding | P4 | NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (architecture/01-topology-and-trust.md). Original finding, kept: [P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead. MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention I1): the claim mail points at https://felhom.drill0242.felhom.eu; that name has no A and no AAAA record (dig @1.1.1.1, control felhom.enkisfelhom.hu resolves); the hub has no tunnel- or DNS-creation code (hub/internal/cloudflare/ holds only geo-rule removal; cf_tunnel_token is a pasted, optional form field, configs.go:1478) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered google.com but not the dashboard name at 13:27:39Z; the agent applied the record at 13:27:44Z (lanresolver: applied split-horizon record … ip=192.168.0.158, 3 m 46 s after the controller started), so the box CAN answer the name — but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (curl --resolve …:443:192.168.0.158). A volunteer could not have done that. What it needs: an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. |
READY — rank P3-LOW; owner: CC Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook. | — | — | CC |
| R-504 | Install & onboarding | P4 | [P3-LOW] iso.felhom.eu cannot show an index page on its own — its root returns 404, and the download page lives on the website instead. MEASURED 2026-09-14: https://iso.felhom.eu/ and /index.html → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded index.html at / was not measured (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at felhom.eu/letoltes (published with the ISO, after the operator's yes). Remaining: a redirect from iso.felhom.eu/ to that page needs a Cloudflare rule the session has no credential for. |
WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule) Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic. | — | — | operator |
Apps & catalog — 7 rows (P2 1, P3 2, P4 4)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| R-981 | Apps & catalog | P3 | The homepage app gets the Docker socket (that is root in the guest) and has no login, no setup gate and no family gate. Fix: remove the socket mount, or put homepage behind the family gate. Security review 2026-10-12 (audits/security-review-2026-10-12/); detail held off-git for the operator (/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/). |
OPEN | — | Change the catalog template; prove on 9202 | CC |
| R-971 | Apps & catalog | P2 | A change pushed to the catalog's main reaches the installed apps on every box within about 15 minutes, and nothing on the box refuses dangerous container settings (privileged mode, the Docker socket, host mounts, host namespaces); so whoever can push to the catalog repository can take every guest. Design-shaped: installed apps take catalog fixes on purpose. Decision for the operator in STATUS.md. Fix options: a privilege allow-list check on the box before any compose is written (and in catalog CI), and later signed catalog releases. Security review 2026-10-12 (audits/security-review-2026-10-12/); detail held off-git for the operator (/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/). |
WAITING-ON-OPERATOR — confirmed by a second reader | operator decision | Decide; then build the box-side check | operator |
| R-676 | Apps & catalog | P3 | [P3-LOW] Watch: immich's first start restarted 12 times — decision 28's crash-loop stop (6 in 10 min) would stop it. From the 2026-09-17 chaos night (DB connection dropped during the first-start geocoding import on a 6 GB guest; it did not recover that night). No healthy app in any drill evidence restarts on a first start (1831 samples, 40 live containers), so the threshold stands; this row exists so the first immich install under v0.269.x is watched. audits/night-2026-09-24/A3/40-first-start-restarts.txt 2026-09-25 night (read from source, v0.271.0): a DEPLOY's first start is NOT covered by decision 28's suppression — Deploying clears when compose up -d returns (deploy.go "Clear deploying flag"), and ObserveUnhealthy then samples the app; an automatic update's step, verify and undo ARE covered (Updating, pinned by TestD28_NoCrashLoopStopDuringAnAutomaticStep). So a first start that restarts ≥ 6 times in 10 min is stopped — which R-676 already accepts for a broken first start; a healthy slow first start would be stopped too. -- 2026-09-30: the first-start restarts are explained. immich's first-start geodata import OOM-kills its database at 512M on a guest with no swap (R-732, measured: 61–104 kills); the 2026-09-17 chaos-night case (DB connection dropped during the import on a 6 GB guest) fits it. Fixed in the catalog (56c4888, 768M). The watch itself (decision 28 on a DEPLOY's first start) is unchanged. |
OPEN — P3; owner: CC (watch) | — | — | CC |
| R-76 | Apps & catalog | P4 | FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable MIGRATED FROM ROADMAP.md 2026-08-22 (R-369) — originally filed 2026-07-26, size S, roadmap state idea (surfaced by the R-75 spike, 2026-07-26). Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. |
OPEN — migrated from ROADMAP 2026-08-22, rank unchanged 2026-10-06 night: not a catalog fix — the setgid chain is set by the controller's FileBrowser setup; a controller row. | — | Two related findings from audits/SPIKE-catalog-data-paths-2026-07-26.md P3/P5, both pre-existing and deliberately left alone by that spike. (a) FileBrowser Quantum 1.3.3 creates files 0644 and folders 0755 and does not propagate the setgid bit — even though the entrypoint wrapper's umask 002 really is in effect (/proc/1/status Umask: 0002). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's group half holds and only its mode half is lost. The consequence is proven with a control: inside a UI-created 0755 folder a gid-1000 process's file landed group 1000, while the identical write into the 2775 parent landed group 100. So any folder a customer creates through FileBrowser breaks the shared-group chain one level down. Latent today — every userdata-touching catalog app that declares an identity declares uid/gid 1000, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at infra/infra.go:156 is right that the image ignores -e UMASK but does not say t |
CC |
| R-707 | Apps & catalog | P4 | [P2] 37 apps still start with a login a stranger can take (09 §3 decision 45). Audit of all 53 apps: app-catalog-felhom.eu/FIRST-ADMIN.md (class, fix route, status, measured or read). Open: 3 hard-coded defaults — calibre-web (admin / admin123, measured working on demo-hp and 9202; its own cps.py -s route needs a generated password WITH a special character — our generator is letters+digits, a controller change), mealie (changeme@example.com / MyPassword), wger (admin / adminadmin); 34 open first-run screens (the first visitor creates the admin: actualbudget, adventurelog, audiobookshelf, calcom, docmost, emby, ghost, gitea, gramps-web, home-assistant, homebox, immich, jellyfin, komga, n8n, navidrome, opengist, outline, papra, plant-it, radarr, rallly, recipe-importer, romm, seerr, sonarr, sparkyfitness, tandoor, termix, uptime-kuma, vikunja, wanderer, wishlist, zipline). Stale notes: romm's default_creds admin / admin answers 401 on demo-hp (like a wrong password) — the page now warns with a login that does not exist; zipline's looks stale too. Measured on demo-hp 2026-09-28 (read-only): bookstack's default still logs in on the INSTALLED app (the fix is for new installs; the page now warns). Each fix: route (a) env or (b) the app's own CLI/API via after_install:, proven on 9202 with the default failing and the generated password working; route (c) a page sentence. Several sessions (operator, 2026-09-28). 2026-09-29 (controller v0.280.0, catalog d0e7e2e): every class-3 app fixed — mealie, wger, calibre-web by after_install (calibre-web with the new password:24:special), proven on 9202 fresh installs (audits/login-gate-2026-09-29/D/); the setup gate (decision 46, spike PASSED) built and live on immich, n8n, audiobookshelf (probes measured) and uptime-kuma (button) (…/C/); romm's and zipline's stale notes removed. Left: 30 class-4 apps — gate each (probe measured on 9202 where one exists — 11 upstream candidates listed in …/B/B-VERDICT.md §3; the button otherwise). 2026-09-29 afternoon (controller v0.281.0, catalog 6faf432): 28 more class-4 apps gated — 32 of 34 — each proven on 9202 (audits/gate-rollout-2026-09-29/B): stranger → gate page / 401, household reached the first-setup screen, the gate opened (9 by a measured probe, the rest by the press), the app answered after. seerr, outline, rallly: gated, their opening needs a media server / e-mail (not proven). Left: wanderer (R-714); plant-it is not installable. |
NARROWED (2026-10-03 triage: the row's verdict was finished, but it names open work no other row carries — seerr, outline and rallly are gated, but the gate OPENING is not proven (needs a media server / e-mail)) — CLOSED — 2026-09-29 (the rest → R-714) | — | — | CC |
| R-905 | Apps & catalog | P4 | wger's collected style files (283 MB) go into every wger backup, though the app rebuilds them at every start. Since R-762's fix (DJANGO_DEBUG=False + collectstatic, catalog cf1ed43) the static files sit in a named volume that the backup legs copy like data (measured 283 MB, audits/design-build-2026-10-06/); collectstatic regenerates them, so they are rebuildable bytes in every Tier-1 unit, Tier-2 copy and off-site snapshot. wger is lifecycle: hidden and no box runs it (hub read 2026-10-08). Options to weigh: a non-backed-up volume class for regenerable data, or an anonymous volume. Found by the R-762 helpers 2026-10-08. |
OPEN — needs a design before wger is shown | — | Design the „regenerable volume" exclusion (catalog + controller backup legs) | CC |
| R-988 | Apps & catalog | P4 | The Usenet pair the operator asked for on 2026-10-12: SABnzbd and NZBGet. Same shape as the torrent pair published the same day (app-catalog-felhom.eu 9081a47) and the same pipeline: a template each, a bench run on LXC 9401, a box walk on 9202 through the product, a first ladder step written by upgrade-test.py --write-ladder, a complete onboarding record, README + FIRST-ADMIN rows, and the app cards. Two things are known already and decide the shape: SABnzbd publishes no image of its own (so lscr.io/linuxserver/sabnzbd, the same third-party exception Transmission got, which the operator granted on 2026-10-12), while NZBGet has the project's own nzbgetcom/nzbget; and both need a PAID Usenet provider (server, port, username, password), which the household must bring - so the app page has to say so before the install, and a provider's credentials are deploy fields the household types, never generated. Neither can be exercised end to end without an account, so the record will say what was measured without one and what it cost. |
OPEN - operator request, not started | - | Write both templates; bench then box, as for the torrent pair | CC |
App updates — 4 rows (P3 4)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| R-440 | App updates | P3 | [P2-MEDIUM] 23 catalog image pins float, so an update is not reproducible. compose pull on a moving tag fetches whatever upstream published that day. MEASURED 2026-09-01 over app-catalog-felhom.eu @ 29edad9c5bf4: 79 image: lines across 53 apps, 66 distinct; 23 of those lines carry a tag with no patch version. postgres:16-alpine (8 apps), redis:7-alpine (6), mariadb:11.6 (2), plus one each of postgres:15-alpine, postgis/postgis:16-3.5-alpine, mariadb:11.4, mariadb:12.3, ghcr.io/claperco/claper:2.5, ghcr.io/thomiceli/opengist:1.13, wger/server:2.6. A 24th is arguable and is recorded rather than rounded away: ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 pins both extensions exactly but leaves the PostgreSQL patch floating. A customer pressing Frissites can therefore swap their DATABASE ENGINE build with no catalog change and no record; two boxes updated on two days end up different. Severity MEDIUM on its own; it becomes BLOCKING the moment a pre-update copy exists, because "what did we upgrade from and to" must be recordable and today it is not — which is also why R-440 must be read next to the digest discipline in Rule 10 of the spike. MEASURED LIVE 2026-09-01 — the floating pins have ALREADY moved, with a passing control. Running digests on demo-hp compared against what the registry serves for the same tag today: mariadb:11.4 MOVED (sha256:4f1d8d20... -> sha256:611a2fcc...) and mariadb:12.3 MOVED (sha256:a02fe89c... -> sha256:dd9b303a...), while postgres:16-alpine, redis:7-alpine, mariadb:11.6 and opengist:1.13 were SAME — and both fully-pinned CONTROLS (rommapp/romm:5.0.0, privatebin/pdo:2.0.5) were SAME. So on a box with ZERO visible drift by tag, pressing Frissites today silently swaps the DATABASE ENGINE build under romm and bookstack, with no catalog change and no record. Compounding fact found while reading: the recovery unit records ImagePins but the manifest comment says "image NOT stored - re-pulled on restore", so a RESTORE of a floating-pinned app also re-pulls whatever is current — the same non-reproducibility on the recovery path. HALF OF THE ANSWER SHIPPED 2026-09-02 (controller v0.233.0, slice 1): app.yaml.installed_images now records, per compose SERVICE, the reference AND the repo digest each container was actually created from — so "what did we upgrade FROM" is answerable on any box that has taken one lifecycle action since the upgrade. What is still missing is the other half: comparing that digest against what the registry serves for the same tag TODAY, which needs a network call the render path deliberately does not make (see R-446). The row therefore stays OPEN and its rank is unchanged — recording a digest does not make a floating pin reproducible; it makes the drift measurable after the fact. audits/SPIKE-app-update-2026-09-01.md — NIGHT 2026-09-23: every image moved tonight (and the 21 backfilled moves) carries its resolved digest in the catalog's test record; the floating pins themselves still float until the box pulls by digest (09 §6.4 part 6). -- MEASURED 2026-09-30 on demo-hp 9201: a fresh install or a guarded Update renders name:tag@sha256 from the ladder (digest.go RenderWithLadderDigests) — 9 of 20 services there run such a definition (adventurelog, docmost, paperless-ngx). The other 11 run TAG-ONLY definitions: bookstack, kimai, opengist, privatebin, romm were installed before v0.269.0 and not updated since, and CarryDigests (digest.go:141) keeps only a digest the running definition already has; bentopdf and calibre-web have no ladder at all. So a re-pull of those (a restore) takes whatever the tag serves that day. audits/pg-last-six-2026-09-30/E/E2-digests-demo-hp.txt. -- 2026-09-30 (evening): calibre-web now has a ladder entry (53a4a1d), so a new install or a guarded Update renders its tested digest; bentopdf still has none. audits/more-night-apps-2026-09-30/ Merged 2026-10-05 from R-446 (duplicate): the household-visible consequence — for these apps the „Naprakész" badge can never turn „behind" (felhom-controller controller/internal/stacks/updateorder.go:96 returns false with no catalog digests / no test date); R-740 (same-tag re-test) is closed, so floating-tag drift for laddered apps is handled. |
NARROWED 2026-09-30 — floating only for (1) apps with no ladder entry and (2) apps installed before v0.269.0 until their next guarded Update; owner: CC. Re-ranked 2026-10-03: P2→P3: narrowed to apps with no ladder entry and pre-v0.269 installs; each guarded update fixes the latter. | — | — | CC |
| R-462 | App updates | P3 | [P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time. The R-449 harness works and is proven by a red negative control (audits/SPIKE-upgrade-test-2026-09-06.md §1). Costed with this run's REAL numbers rather than an estimate: a successful edge takes 6.4 s – 305.1 s, median 71.8 s; a FAILING edge takes 556 s, roughly 8×, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost 5.07 GB, so 53 apps naively extrapolate to ~90 GB and, at the median, about an hour of harness time for one edge each. THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row. Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). Fixture time scales with apps and does not amortise. The decision this row is really asking for is scope, not schedule: all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. Recommended shape, NOT a design — the operator picks: start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: VIKTOR rules on scope, CC implements. audits/SPIKE-upgrade-test-2026-09-06.md §5 ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was. It read "VIKTOR rules on scope"; the operator ruled on 2026-09-13 (09 §3 decision 6) that the upgrade test goes to ALL apps through the nightly rotation, explicitly not "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in 09 §6.4: legs A–E ≈ 21–34 CC-hours plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL pg_upgrade rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. -- UPDATE NIGHT 2026-09-21: The count moved from 3 apps to 21 EDGES ACROSS 19 APPS. The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: 14 proven, 3 failed, 4 inconclusive. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. Ten of the fourteen printed a verbatim migration line, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at audits/update-night-2026-09-21/fixtures.py, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into app-catalog-felhom.eu/scripts/upgrade_fixtures.py with seven new EDGES (U1-U7) so the same edges can be run on the harness venue with their ABORT step, which the box deliberately does not offer. OWED, stated so it is not mistaken for done: the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). -- 2026-09-23, the RomM lesson is IN THE HARNESS: upgrade-test.py v2 runs every edge that read back under light load for --soak seconds (default 600) and reads the kernel's oom_kill counter host-side; a kill or restart turns proven into failed, a peak over 80 % of the limit adds memory_tight. Red-proof: romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then OOM-killed at +76 s under four light callers, verdict failed (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence audits/update-rulings-2026-09-23/harness/. — NIGHT 2026-09-23: the bench and the box now share ONE fixture set — the box walk's fixtures run on the bench through upgrade_boxport.py (ported verbatim into the catalog), plus a new wishlist fixture and fixes for opengist 1.15, komga (/api/v2/users/me) and nextcloud (wait for occ status). 14 apps / 15 edges tried; 12 proven on both venues and published with their test records (audits/DRILL-night-2026-09-23.md Part C). -- 2026-09-30 (day brief): three new front-door fixtures (sparkyfitness, rallly, outline; catalog e6f3ec2) and five apps moved on both venues — the three PostgreSQL conversions (R-463) plus bookstack 26.05.5 → 26.09.1 and kimai apache-2.57.0 → 2.67.0 (upstream dropped the apache- prefix; the plain tag's digest equals apache's, measured). The catalog currency and the next-apps ranking: audits/catalog-currency-2026-09-30.md — 25 of 53 apps behind upstream inside a major, 19 across one (morning); after the session 31 apps can update themselves at night (+ nextcloud with a whole copy), 32 carry a proven step, 21 have no ladder. -- 2026-09-30 (evening, more-night-apps brief): ten more steps published on both venues — first ladders for calibre-web, gitea, wger, crafty-controller, uptime-kuma, zipline (new front-door fixtures; gitea's installer form), within-major steps for emby, ghost, home-assistant, outline, rallly. Night-updatable by the audit's method (a): 30 + 2 conditional at the session's start → 35 + 3 conditional (calibre-web conditional: files_may_change). zipline got a two-step ladder 4.6.1 → 4.7.0 → 4.8.0 (the direct jump fails, R-742). Stopped: wanderer (the bench cannot run it, R-739). Fixture/harness defects fixed on the way: R-735, gitea's installer, the HTTPS backend on the bench, per-file names for files_may_change. audits/more-night-apps-2026-09-30/ |
READY — rank P2-MEDIUM; owner: CC (scope already ruled, 09 §3 decision 6) Re-ranked 2026-10-03: P2→P3: 35+ apps already update by proven steps; an app without a ladder simply does not move, which is safe. |
— | — | CC |
| R-683 | App updates | P3 | [P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made. 2026-09-24 chaos round 3 (nextcloud, backup_max_age: 1m): no backing-up phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. audits/night-2026-09-24/E/round-03*.json, E/round-11-controller-pre.log |
OPEN — P3; owner: CC (watch) | — | — | CC |
| R-785 | App updates | P3 | [P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24). READ 2026-10-01 (audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt). Needs: an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. |
OPEN — rank P3-LOW; owner: CC (after R-784) | — | — | CC |
Backup & restore — 28 rows (P2 3, P3 15, P4 10)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| R-978 | Backup & restore | P3 | Starting the escrow ceremony again can make the customer's written-down recovery code useless: the new package replaces the old one before the new code is confirmed. It can happen to an ordinary customer who starts and walks away. Fix: keep the previous package until the new code is confirmed. Security review 2026-10-12 (audits/security-review-2026-10-12/); detail held off-git for the operator (/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/). |
OPEN | — | Fix in the agent with a test | CC |
| R-973 | Backup & restore | P3 | A compromised controller can make the whole-guest backup run again and again, and each run prunes older restore points: the local tier directly, and the off-site tier through the store's own nightly keep-last-2. So 07 §8 row 10 (ransomware) is weaker than it reads. The second reader confirmed the local half; the off-site half is inferred. Fix: a cadence or daily cap on the local-API backup route, and off-site snapshots that a nightly prune cannot remove. Security review 2026-10-12 (audits/security-review-2026-10-12/); detail held off-git for the operator (/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/). |
OPEN | — | Add the cap with a test; decide the off-site retention shape | CC |
| R-926 | Backup & restore | P3 | After a household removes an after_install app keeping its backups and then restores it, nobody has checked whether the app page still shows a password that works. A type: password deploy field is NOT in stacks.PortableSecretEnvVars (that list is type: secret only, deploy.go:1199), so it does not travel in the backup unit — the same shape as R-765, where Radicale's restored login was replaced by a value no page showed and every phone lost its calendar. MEASURED on 9202 2026-10-10 for grocy: remove keeping the backups, restore, and the household's own password still signs in and the seeded data reads back — so the DATABASE half is right. What was NOT settled is the PAGE half: app.yaml holds the value encrypted at rest, so reading it proves nothing (a 76-character string that of course did not sign in), and POST /apps/grocy/initial-credentials/reveal answered 404 for this app. Seven apps are in the same class: bookstack, calibre-web, claper, dawarich, grocy, mealie, wger. |
OPEN — the question is unanswered, not the answer bad | — | Ask the controller what the app page renders for a generated type: password after a restore (one endpoint read, one app), and if it differs from the one that works, decide: carry the value in the unit, or re-run after_install after a restore |
CC |
| R-232 | Backup & restore | P2 | DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails. Surveyed read-only 2026-08-06 (audits/RECON-dooplex-backup-2026-08-06.md). What works: five sets, 14/14 successful runs in 14 days; a file was restored from the data repo and matched the live original byte for byte; every set except two is cross-disk; k3s is integrity-checked on every run. What the matrix exposes, ranked: (a) notify_failure is a no-op — NOTIFY_ON_FAILURE=true but NOTIFY_WEBHOOK_URL is commented out, so a failed backup notifies nobody; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) Nothing leaves the box — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is nfs://192.168.0.180: pointing at DooPlex itself, and the only outbound-looking cron pulls inbound from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) The backup tree is a single writable path and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) Two same-disk sets: .claude-memory and the PostgreSQL dumps, whose source directory sits inside the backup tree. (e) Longhorn retain=1 — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) /opt/backup/docs/BACKUP-RESTORE.md does not exist though the systemd unit advertises it. (g) secrets/restic-repo has never held a snapshot — backup-secrets.sh contains no restic call; the secrets are GPG files on sda1 only. (h) No restore has ever been run beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. Not a finding: the restic passphrase. The on-box copy is on sdb1, a different disk from the backups, and the operator holds an offline copy out of band — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. Nothing was changed by the recon. |
NARROWED 2026-10-08 — owner Viktor. (a) DONE with the operator's yes in chat: notify_failure now mails admin@felhom.eu through Resend; proven by one test mail that reached the inbox (audits/day-2026-10-08/r232/; no backup was started). (b) partly: the hub database leaves DooPlex nightly to ep0 (R-173); everything else stays on the box. (c)–(h) unchanged. 2026-10-09 (operator yes in chat): (b) and (h) DONE for Gitea and the secrets. Nightly 00:20 felhom-dooplex-offsite pushes Gitea (614 MB of repositories, the gitea database dump taken first, app.ini) and the nightly GPG secrets export, encrypted with a NEW key, to ep0 operator (host/dooplex-gitea) on the hub-DB write-only token — no change on ep0; first push 57 s; restore test weekly (manifest, git fsck every repo, pg_restore --list), run once: 27 805 files, 10 repos OK; DooplexGiteaOffsiteStale/DooplexGiteaRestoreTestStale in force (absent() included, promtool + 2 red-proofs); every backup unit now mails on failure (OnFailure=, dry run reached the inbox). First real Gitea restore: into a throwaway on the bench, 10/10 repos, product main = live, a file byte for byte, a login (runbooks/gitea-restore.md). The hub came back too (R-173, closed). The registry is left out on purpose: its images rebuild from the code. Correction to the recon §7: the API mirror holds ONE repo (homelab-manifests), not all — Gitea had a single copy, on DooPlex. audits/dooplex-survival-2026-10-09/ LEFT: (c) append-only for the on-box tree, (d)–(g) as written; the paper copy of the new key (operator). READY for the rest |
— | — | operator |
| R-893 | Backup & restore | P3 | After a failed OFF-SITE replay, the rollback pours the NEWER pre-restore copy over the OLDER volume just put back. Read in source 2026-10-06 (R-638 option A, not measured): internal/backup/offbox_reconstitute.go writes the undo copy from the live (newer) database, replaces the volumes with the snapshot's older tars, then — when the replay fails — rollbackSafetyDump loads that newer dump over the older database volume. The loader only drops what the dump knows, so tables the newer migration removed stay; and when the snapshot's older definition was written, the rollback branch does not put the newer definition back, so the older app starts on rolled-back data; non-database volumes stay at the snapshot's state. An order change cannot fix it (the only undo is a logical dump, and its volume was replaced). Known limit in 07 §6.3. -- 2026-10-08 design: audits/day-2026-10-08/design-R-893.md — re-verified; and the screen's „your data is back as it was" (err.backup.db_restore_failed_rolled_back) is false for files, other volumes and the app version. Question D8 on STATUS's decision sheet. -- 2026-10-08 14:16 operator ruling D8 (09 §3 decision 192): yes, both, in that order — first the app stays stopped for support (option C), next „put back exactly as it was" (option A). |
NARROWED — 2026-10-09: the first half DELIVERED in controller 0.304.0. Slice 0 (the live measurement) NOT run: 9202 has no off-site, and a dump cannot be cut by hand inside an encrypted off-site copy on Tester 1; it needs a scratch off-site repo — a plan, not a finding. NARROWED 2026-10-08 — the first half (D8 option C, the app held stopped) is built on controller main and ships tomorrow; what stays open is the second half, „put back exactly as it was” (option A: a pre-restore copy of the app's volumes and placed files, a fit check, disk for one copy; the slice-0 measurement on 9202 first). OPEN — filed 2026-10-06 2026-10-06 night: verified in source, no code — offbox_reconstitute.go:758 (undo dump from the live DB), :778-860 (files and volumes from the snapshot), :804 (the snapshot's definition is written when its version differs), :898 (the rollback loads the newer dump over the older volume; nothing writes the live definition back). Not a reorder fix: it needs R-638 option B (a rebuilding loader) or a pre-restore volume copy (disk cost; R-685's class). Which state a household gets after a failed off-site replay is the operator's call. Next: the 9202 measurement, then the design. |
a design: R-638 option B (a loader that rebuilds instead of overlays) or a pre-restore volume copy | Measure it once on 9202 (a forced replay failure after an off-site restore over a migrated app); then a design for the operator | CC |
| R-895 | Backup & restore | P2 | The hub's clean-up-window check trusts the snapshot counts the box sends, so a broken-into box (or past-dated fakes added through the add-only key) can shrink the real off-site history without an alarm. READ 2026-10-06 night in source (R-822's design): the before/after comparison uses counts the box itself reports (hub/internal/offsitekeys/service.go:284, :343); new fakes keep the count level. Decision 68 already accepts a box-trusted count. |
OPEN — filed 2026-10-06 night 2026-10-07 07:58: kept open for later (09 §3 decision 166). |
a design + one read-only measurement (does the Storage Box shell on port 23 show snapshot file upload times?) | Option B of audits/night-burndown-2026-10-06/design-R-822.md: the hub lists the repo's snapshots/ files over its own login before and after a window and alarms on snapshots no box run explains |
CC |
| R-923 | Backup & restore | P2 | The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies. Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: vaultwarden-system/vaultwarden (1.37.3), PVC vaultwarden-data 7.4 MB (db.sqlite3 + WAL, rsa_key.pem, no attachments); its only backups were Longhorn on DooPlex. A key in a password manager that runs on DooPlex is not off DooPlex — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its ADMIN_TOKEN is plain text (an Argon2 hash is its guidance). |
NARROWED 2026-10-09 (operator yes in chat): Vaultwarden's own backup copy + rsa_key.pem ride the nightly encrypted DooPlex→ep0 job (scripts/dooplex-offsite/), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: runbooks/break-glass-sheet.md, runbooks/total-loss-of-dooplex.md. LEFT: the operator prints the sheet and stores it away from home; the ADMIN_TOKEN hash (a DooPlex change, operator's word). |
operator: print the sheet | Operator: print + store the sheet; then close | operator |
| R-49 | Backup & restore | P3 | [P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup". Measured 2026-07-19: immich_ml_cache.tar 823 660 032 B (~60%) — re-downloadable ML model weights; immich_postgres_data.tar 308 251 136 B (~23%) — a raw tar of the postgres data dir that DUPLICATES the logical .sql dump captured beside it; upload/backups/ 18 MB — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 dccc13fe… tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: 72 MB of originals. MIGRATED FROM ROADMAP.md 2026-08-22 (R-369) — originally filed 2026-07-19, size S–M, roadmap state idea. Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. |
OPEN — migrated from ROADMAP 2026-08-22, rank unchanged Re-ranked 2026-10-03: P2→P3: wasted space and transfer, no data risk; needs a capture-set ruling, not a sale blocker. | — | Evidence: audits/DIAG-immich-restore-round2-2026-07-19.md §4 (full byte breakdown). This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. Recorded, deliberately not changed — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) immich_ml_cache — pure cache, strongest case; (b) the postgres_data volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) upload/backups/. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app |
CC |
| R-127 | Backup & restore | P3 | The catalog's data_key: true flag is UNRELIABLE — at least four data-encrypting keys the catalog itself labels as encryption keys are unflagged; and the O4 restore path can regenerate a DB password that then does not match the restored data directory |
READY (S/M) 2026-10-06: leg (a) PUSHED to the live catalog (c265b37). 2026-10-06 (burn-down night, later): leg (b) NEEDS A DESIGN. 07 §7.4 sets no direction; refusing the restore without the DB password, or ALTER USER after it, each change restore behaviour on customer data. |
— | Found by D5's Part 0, and it is why D5's boundary is type: secret rather than data_key. Two separable legs. (a) The misclassification. Only 5 fields across 4 apps set data_key: true (adventurelog/SECRET_KEY, homebox/HBOX_AUTH_API_KEY_PEPPER, papra/AUTH_SECRET, sparkyfitness/{API_ENCRYPTION_KEY,BETTER_AUTH_SECRET}), yet n8n/N8N_ENCRYPTION_KEY („Titkosítási kulcs"), wanderer/POCKETBASE_ENCRYPTION_KEY („Adatbázis titkosítási kulcs"), calcom/CALENDSO_ENCRYPTION_KEY and bookstack/APP_KEY are unflagged — the catalog's own Hungarian labels contradict the flag. D5 makes this non-urgent but not harmless: everything type: secret now travels, so the keys DO reach the drive; what stays wrong is the fail-closed gate, which only refuses for data_key names — so if one of these is missing from both sources the restore proceeds onto data it cannot decrypt instead of refusing. Fix = flag them (app-catalog-felhom.eu, a catalog-only change) + a gate/test that the flag set and the label set agree. (b) The regenerated-DB-password trap. internal/backup/restore_unit.go O4 generates a replacement for any missing non-data-key secret. Proven on postgres:16-alpine: with PGDATA restored from the volume tar, POSTGRES_PASSWORD is ignored (initdb skipped), so the app fails over the compose network while the dump replay still succeeds through the container's local trust socket — success reported, data unreachable. v0.188.0 corrected the WARN's false claim that "stored data is unaffected" and scoped it, but did not add a guard: D5 shrinks this to the rare case (the secret was empty at capture AND absent from the guest). Real fix = either treat a DB password as fail-closed like a data key, or ALTER USER to the regenerated value after the volume restore. 18 DB/root-password fields are in scope; MariaDB fails loudly instead (Access denied), which is the safer half |
CC |
| R-231 | Backup & restore | P3 | /opt/backup/scripts/ on DooPlex is unversioned host state — found 2026-08-06 while adding the auto-memory store to the backup set. No repository tracks the scripts that protect the recovery chain, so the edit made that day (CLAUDE_MEMORY_DIR in backup-config.sh, multi-path restic call in backup-data.sh) exists only on the box. This is the same class the part-2 session was closing, found inside the fix for it; the change is transcribed in felhom.eu/workspace/README.md so it is at least recorded. Two related facts, both understating current safety: the backup destination (/mnt/5_hdd/backup) is on the same physical disk as the workspace it protects, and the DooPlex backup set has no off-site leg (sync-hetzner-backups.sh is jarrs.eu and pulls from Hetzner to DooPlex). Bringing a root-owned production backup script under version control, and deciding what installs it, is its own scoped change. |
VERIFY — 2026-10-12 evening: the token left the clone URL (operator yes in chat; homelab-manifests 1943a70, installed by the role): backup-manifests.sh sends it as an http.extraHeader through GIT_CONFIG_* env; read-only proof: git ls-remote of the private repo with the header → HEAD, without → refused. The 2026-10-13 03:10 run is the test of both changes. VERIFY — 2026-10-12: in git and installed from it (operator yes in chat, operator in the session): homelab-manifests fdd324b, homelab-ansible/roles/dooplex_backup/ (11 scripts + the timer units, byte-identical to the live ones; no secret in any file; install.sh dry run / --apply). The install changed only the units' mode (744 → 644). Baseline: last night's run Result=success, exit 0 (audits/answers-2026-10-12/s20-backup-scripts.txt). Seen by the commit's security review, NOT changed (a DooPlex behaviour change needs the operator's word): backup-manifests.sh puts the Gitea token into the clone URL for the mirror run (visible in the process list for those seconds) and strips it from the stored remote afterwards; no token VALUE is in any file, and the repo is private (anonymous API read = 404). A header-based fetch (git -c http.extraHeader=… from the token file) would remove it — one line in the decision list. READY — operator ruling 2026-10-12 (S20, 09 §3 decision 215): yes — the scripts into a repo + an install script; the operator present for the install. READY — owner Viktor |
— | 2026-10-13: read dooplex-backup.service Result/ExecMainStatus after the 03:10 run; success → close |
CC |
| R-314 | Backup & restore | P3 | StopAbandon has no web route — a customer who telephones is served by a command line. --abandon-stop exists on the controller binary (cmd/controller/main.go:86) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). An operator who is telephoned instead has to reach a shell on the customer's machine. Used this session on the operator's ruling, container stopped first so the running controller could not overwrite settings.json from memory — a sequencing subtlety that is itself an argument for a route -- 2026-10-08 design: audits/day-2026-10-08/design-R-314-279-177.md — the box's own 14-day countdown has no operator door (the hub's 7-day phase does, decision 74); one door for R-314 + R-279: a fixed list of operator actions carried in the report reply. Question D1 on STATUS's decision sheet. -- 2026-10-08 14:16 operator ruling D1 (09 §3 decision 185): yes — option A, a fixed list of operator actions carried in the box's report reply. |
VERIFY — 2026-10-09: DELIVERED (hub 0.144.0 + controller 0.304.0); run_job fill-watch proven live on the Tester 1 box (hub row „done — ran to its end"; control: the box log „checked 3 filesystem(s)"; an unknown action is refused 400). abandon_stop / abandon_extend NOT yet proven — no box has a deletion countdown; prove them on the first one. VERIFY — built on main 2026-10-08 (D1, decision 185), ships with tomorrow's controller + hub releases; closes after the live proof on 9202 (abandon_stop / abandon_extend through the hub's buttons). READY (S) — NEW 2026-08-12, RANK 3 2026-10-05 (burn-down night): NEEDS A DESIGN. There is no operator door into a RUNNING controller (its HTTP surface is the household's session; the CLI runs in a separate process and cannot safely trigger a job in the running one). Same design as R-177 and R-279. |
R-241, R-307 | An operator-authenticated POST that calls the same StopAbandon, so the telephone path and the code path converge |
CC |
| R-401 | Backup & restore | P3 | Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store. Controller v0.228.0 (R-399) made --read-data-subset=100% the default for every box. The whole justification is a single data point: demo-hp, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% 39.2 s — four seconds. Re-proven live 2026-08-31 at 38.7 s. It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA. A 50 GB store is ~370x the data and this curve says nothing about it. Nothing was invented from that one point — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. readDataSubsetRe already accepts n/m, so a rotating schedule (1/7 on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. THE TRIGGER IS AN EVENT, NOT A DATE: the slow-check WARN from v0.228.0 firing on any box (integritySlowNoticeThreshold, 5 min) — that line names the duration, the depth and this row. WHAT HAPPENS IF NOBODY ACTS: every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. Whoever acts must also revisit integrityCheckTimeout (30 min), which is now the number a large store meets first. |
OPEN — WATCHING | — | When the WARN fires: measure the curve on that store, then choose between a rotation (n/m), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. |
CC |
| R-409 | Backup & restore | P3 | Nothing in the product can vouch for the bytes of a restored recovery unit — the only hash record covers 0.002 % of it. MEASURED on demo-hp 2026-08-31 against kimai's restored unit: manifest.json's checksums object carries sha256 for .felhom.yml (2 235 B), app.yaml (488 B) and docker-compose.yml (2 195 B) — 4 918 bytes of a 213 231 242-byte unit. The database dump (48 217 B) and the two named-volume tars (160 331 776 B + 52 845 056 B) — the recoverable data, 99.998 % of the bytes — have no recorded hash anywhere. And nothing else supplies one: restic 0.14.0's restore --verify is a size-and-mtime reconciliation (a size-and-mtime-preserving one-byte corruption of a 160 MB tar passed clean, red-proofed), and restic ls --json file nodes in 0.14.0 carry name, size, mode, uid/gid and three timestamps and no content hash. So "the restore produced correct files" is currently unanswerable by any automated means. What is NOT claimed here: restic check --read-data-subset=100% already proves the STORE's packs, and the config files that ARE hashed are the ones a wrong-content failure would be hardest to spot in. |
OPEN — MEDIUM | R-87, R-361 | Cheapest fix, and it is already half-built: extend the capture's checksums to cover db_dumps and volume_dumps — R-361 already computes a canonical dump sha256 to prove itself, so the value exists at capture time. Then a restore-test has a real reference and R-87's narrow version becomes a content check rather than a completeness one. Evidence: audits/SPIKE-restic-restore-test-2026-08-31.md §Q2, §Q3. |
CC |
| R-433 | Backup & restore | P3 | A sub-account cannot reach ANY Storage Box snapshot, by any name — so clause (b) of the 2026-09-01 R-95 re-scope ("the rest is recoverable file by file") is NOT SUPPORTED. MEASURED 2026-09-01 on demo-hp over the credential the box already holds, read-only, no delete verb issued. The sweep: a batched stat -c %n over Hetzner's port-23 restricted shell (500–600 paths per round trip, stdout carrying only paths that exist) tried 777,600 names of the vendor form YYYY-MM-DDTHH-MM-SS across nine full days at second granularity, and 126 alternative shapes — zero resolved. The control is what makes the zero mean anything: the identical 600-name batch with one real path appended returned it in 6 of 6 batches. The structural cause: /home (the customer data, u629488-sub3) is st_dev 0,82; /.zfs/snapshot is st_dev 0,276; /home/.zfs does not exist. A snapshot under /.zfs/snapshot belongs to a different dataset than the one holding felhom-repo. What still stands: clause (a) — the box can delete its live repository but cannot WRITE into /.zfs/snapshot — is unchanged and re-confirmed. What is now open again: the only routes to the older copy are a panel rollback of the WHOLE Storage Box (deletes newer snapshots, hits every customer on it) and the provider API (fenced by §11-D, and hub/internal/hetznerapi/hetznerapi.go has no snapshot method at all, so it needs new code regardless). NOT ESTABLISHED, and it is the question that decides whether per-file recovery exists for anyone: whether the MAIN account can see the snapshots. No main-account credential exists in this project. The ranking is Viktor's; I am not re-ranking R-95 — but the argument that moved it down is the argument this drill removed. audits/evidence-drill-r95-recovery-2026-09-01/ BLOCKED-ON-PROVIDER 2026-09-01. The one question that can move this is drafted and ready to send: Question 1 of documentation/runbooks/provider-questions-2026-09-01.md — can the MAIN account retrieve individual files from a snapshot, without a whole-box restore? Neither answer leaves this row where it is: "yes" makes per-file recovery real but permanently operator-only, and closes this; "no, full restore only" means the snapshots do not bound a single customer's exposure at all, because using them costs every other customer on the box their newer snapshots — and R-95 becomes urgent. Nothing here can progress without it, and it is not CC's to send (§11-D). Tracked by a dated check in the DUE-CHECKS block. ⚠ A TRAP ON THE WAY TO THIS ANSWER, found by the operator 2026-09-01 and armed against in the questions file: HETZNER HAS TWO SIMILARLY-NAMED PRODUCTS WHOSE DOCS SAY OPPOSITE THINGS. Storage SHARE (a managed Nextcloud, NOT us) documents "Currently, we only support restores for the full backup ZFS snapshot to a specific point in time" (docs.hetzner.com/storage/storage-share/faq/backup-snapshot/). Storage BOX (ours) documents the opposite — "You can download individual files or entire directories as usual" (docs.hetzner.com/storage/storage-box/snapshots/). A web search for the obvious phrasing surfaces the SHARE page, and it reads like a definitive NO. Tell them apart by the giveaways: the Share page talks about Nextcloud's data cache, a database dump and the konsoleH interface, and never mentions Storage Box. Why this is filed and not left as trivia: a support agent could answer Question 1 from the wrong page, and that answer would push this row to the top of the register for no reason. Question 1 now names the product, quotes the Storage Box line, and says explicitly that we know what the Share FAQ says. If an answer cites full-snapshot-only, check WHICH PRODUCT it is about before acting on it. Nothing in this repository ever leaned on the Share claim — verified by grep at the time; the only vendor line we cite is the Storage Box one. RE-DATED 2026-09-15 → 2026-09-22 (due-checks gate): the operator mailbox read through the Gmail connector ((from:hetzner OR subject:hetzner OR "Storage Box") after:2026/09/01) holds no Hetzner reply — one match, our own offsite_snapshots_dropped alarm. Whether the two tickets were ever sent is not visible to CC; if they were not, sending them is the operator's act. -- ANSWERED. Hetzner replied on ticket #2026090103040671; the operator supplied the thread on 2026-09-22 after this session searched for it and WRONGLY reported no answer (see the instrument note below and R-628). Q1 - can the MAIN account read individual files and directories out of a specific snapshot, without restoring the whole box? Hetzner: "With the main account it should be possible to download files and directories from the snapshot as it is stated in the documentation", citing docs.hetzner.com/storage/storage-box/snapshots#access-to-snapshots; and separately "A restore of a snapshot will revert the Storage Box completely to the state it was in when the snapshot was taken." So the Storage BOX documentation governs, not the Storage Share FAQ - which is exactly the trap provider-questions-2026-09-01.md warned the reader about, and the answer came back on the right side of it. File-level snapshot access is a MAIN-ACCOUNT capability; the sub-account cannot do it, which matches R-433's own measured finding (777,600 names swept from a sub-account, zero hits). HEDGE, kept because it is in the reply: "should be possible" is not "is", and nobody has yet read a file out of a snapshot from the main account on u629488. That is now the measurement this row needs, and it is cheap. Q2 - is --append-only enforced by Hetzner, or taken from what the client sends? Hetzner: "You can use --append-only and it would look like this: command="rclone serve restic --stdio --append-only path/to/repo" <public key>", with fluix.one/blog/hetzner-restic-append-only/. So it is enforced by US, by pinning a FORCED COMMAND to the SSH key in the Storage Box's authorized_keys - not by Hetzner globally, and not by anything the client asks for. A key without that prefix still gets a deleting server. This is the answer R-95 has been blocked on and it says append-only IS expressible on a Storage Box, which R-95 records as impossible over SFTP - see R-95 and R-436. NOT YET MEASURED, and that distinction is the whole of what this row is worth: this is a support answer, not a proof. What is owed is a real forced-command key on u629488, a restic forget --prune through it that is REFUSED, and a backup through it that still succeeds. |
BLOCKED — BLOCKED-ON-PROVIDER — Question 1 of runbooks/provider-questions-2026-09-01.md |
— | — | CC + operator |
| R-540 | Backup & restore | P3 | [P3-LOW] The hub knows exactly ONE off-site pool box, so there is no rule for what happens when it fills. Read from source 2026-09-16 while making off-site the default: HETZNER_POOL_BOX_ID is a single value, and every shared customer becomes a sub-account on that box. With off-site now ON for every new customer (hub v0.116.0) the box fills faster, and the fill warning (80%/90% of the box, monitor/offsite.go) tells the operator it is filling but nothing says which box a new customer should land on. Needs a selection rule (least-full, or explicit per-customer), not a bigger box. No customer is at risk today: the pool box read 0.3% full (2.7 GB of 1 TB), Σ shared quota 150 GB, oversub 0.15x. |
READY — rank P3-LOW; owner: CC (hub) 2026-10-05 (burn-down night): NEEDS A DESIGN (and money). A selection rule needs a multi-box configuration and a second box. Next: an operator pick of the rule and its trigger (e.g. add box 2 at 70 %). | — | — | CC |
| R-548 | Backup & restore | P3 | [P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever. MEASURED 2026-09-17 (chaos night) on tester-1-022354: a whole-guest backup wrote a ~29 GB source (mp0 = local-lvm:vm-9201-disk-1, 70 G provisioned, 40.58 % used, backup=1) into pve-root, which on a 32 GB system disk is 14 GB total with ~4.9 GB free. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — ~16 MB/s, i.e. under four minutes to a full / on the nested PVE. The product’s behaviour is correct and legible throughout: it failed the tier and said which one — whole_guest_backup_failed (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (target_id:"local", success:false, size_bytes:0), and the off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can never succeed, and it keeps retrying on a backoff for ever, burning I/O and risking / each time. Honest caveat: the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. Fix shape: compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: audits/evidence-chaos-night-2026-09-17/round-6.txt. -- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27 („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's local holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). audits/pg-last-six-2026-09-30/C/. |
READY — rank P3-LOW; owner: CC | — | — | CC |
| R-921 | Backup & restore | P3 | When the off-site tier answers BUSY, the household still gets a short stop of every app — with no copy. MEASURED 2026-10-08 night on demo-hp (read-back 2026-10-09, audits/dooplex-survival-2026-10-09/partE/R-518.txt): local tier 20:20–20:25Z (own stop, < 2 min), then at 20:26:14Z the agent refused the off-site request (backup refused — a heavy operation is already in flight, busy=backup:local — the night OS step ran 20:27–20:29Z), yet the controller's metrics show all 15 app containers down at 20:26:19Z and back at 20:27:19Z; the retry ran the off-site tier at 20:34Z with its own short stop. So a two-tier night costs THREE stops, one of them for nothing. This was R-518's noted "unmeasured" risk. Fix shape: ask the agent whether it is free (or reserve the slot) BEFORE quiescing, and resume at once on BUSY. |
NARROWED 2026-10-09 — built; RELEASED 2026-10-10 (controller 0.305.0 on demo-hp, demo-felhom, Tester 1): before stopping apps the controller asks the agent which tiers have a job in flight (GET /backup/status); another tier's job in flight → no stop, the tier stays due (red-proved: 3 apps stopped for a refused tier before); a refusal after the stop still resumes the apps at once (pinned). Also fixes a stop after a long upload or a restart mid-upload. LEFT: the measured case itself — the agent's host-wide busy lock (the night OS step, restore test, fstrim) is served on NO endpoint, so the controller cannot see it before stopping; needs one agent field (e.g. busy on GET /backup/status) + a controller pre-check behind a feature probe. 07 §6.4. |
an agent change | Build the agent busy field + its controller check; read back a two-tier night |
CC |
| R-933 | Backup & restore | P3 | Mirror the images of every catalog app we support into our own registry, before scale. Operator, 2026-10-12 (S8, 09 §3 decision 203): „Will need to mirror the apps we support later, just to be sure." Today a restore re-pulls each image from its maker; a version the maker deleted cannot start (the accepted limit, 07 §6.6, R-698 closed). Size it first: the supported catalog's images (every ladder version a box can hold) × their size; then where the copy lives (the DooPlex registry is Tier 2, precious — the copy must not make DooPlex a single point of failure for every household's restore) and how a restore falls back to it. |
OPEN — NEW 2026-10-12 | the size measurement | Measure the total size of the supported catalog's images; then a one-page design (where, how a restore falls back, cost) | CC (measure + design), operator (the go) |
| R-91 | Backup & restore | P4 | Old 13 GB datastore copy at /srv/pbs-felhom on ep0's root disk Checked from source 2026-10-05 (burn-down round 2): Gate is long past (row waits on demo-felhom's first post-migration PBS backup, migration 2026-07-27). Last positive record of the copy: audits/CAMPAIGN-9-restore-proof-2026-07-28.md:759 'ep0 : /srv/pbs-felhom rollback copy intact (13G)'; CONTEXT.md:3666 still says it is 13 G of dead weight awaiting R-91. No later record of deletion found (grep srv/pbs-felhom across felhom.eu). Deleting is on ep0 (protected) and needs an operator word. |
WATCHING | demo-felhom's first post-migration PBS backup | Delete once it lands; fix CONTEXT.md:1018 same commit |
CC |
| R-164 | Backup & restore | P4 | C2's chain: the DB volume tar cannot be dropped until a SOUND dump predicate exists. The unit carries both a volume tar and a SQL dump; the restore uses both — the dump is authoritative and replayed after the tar so it WINS (F17), with only the DB service up (R-47) — internal/backup/restore_unit.go:262-266. Dropping the DB container's tar would halve DB-app units and close the R-127(b) initdb-skip password trap (restored PGDATA ⇒ POSTGRES_PASSWORD ignored). |
BLOCKED — on the predicate | a dump-validity predicate that is not accounts has rows |
The obvious gate is DEAD, measured: ValidateDump warns when the accounts table is empty, and that warning was correct — the live DB genuinely had 0 accounts, and seeding one stopped the warning and put the row in the dump. But a fresh appliance legitimately has zero accounts, so promoting that predicate to a gate would block every new customer's first backup. Order: (1) a sound predicate — dump vs live per-table counts, not an absolute expectation; (2) warn→gate; (3) tar-drop. Until (1), the tar is load-bearing — not because dumps are bad, but because nothing can yet prove one is good. Pairs with R-127 |
CC |
| R-213 | Backup & restore | P4 | Putting files back in place — the half the recovery screen deliberately does not do. The screen (R-193, controller v0.200.0) unlocks the repository and LISTS what is in it; restoring is per-app and lives in the backups area, and the operator ruled the two separate on 2026-08-05: a screen that unlocks and then offers to overwrite is two decisions wearing one button. What is missing is the step after the listing — a customer who can now SEE their files still has to work out, per app, which restore to choose. The operator named its requirement: a live-versus-backup comparison — the customer must be able to see what would change before anything is overwritten | DEFERRED — operator ruling 2026-10-12 (S18, 09 §3 decision 213): designed after the first customers. Stays P4; re-dated: look again when the first paying household has run 30 days. OPEN — not started, deliberately |
the comparison design (nothing exists for it yet) | After the first customers | Operator + CC |
| R-336 | Backup & restore | P4 | The offsite DR endpoint is polled about once per second, and that is what turned a slow leak into an outage. ep0's PBS proxy served ~85,000 requests/day — a flat 3,538/hour, every hour, from two boxes: 74,445 GET /api2/json/admin/datastore (libwww-perl, i.e. PVE's pvestatd) and 73,171 GET /admin/datastore/felhom-offsite/status (proxmox-backup-client). Two pollers asking substantially the same question at the same rate. On 2026-08-18 this walked a connection leak in the proxy to its 1024-fd soft limit in 14 days, wedging the offsite tier for 9½ hours (audits/INCIDENT-ep0-pbs-fd-exhaustion-2026-08-18.md). The LimitNOFILE=65536 drop-in applied that morning raises the ceiling but does not fix the leak — it converts a fortnightly outage into a multi-year one, which is mitigation, not a fix. A DR endpoint that is written to weekly does not need to be asked about every second |
READY (M) — NEW 2026-08-18 2026-10-06 night: no hub half — the pollers are pvestatd and proxmox-backup-client on the boxes; the lever (disabling the storage entry) collides with the agent's pbsdr health model; still a design question. | — | CORRECTED 2026-08-18 (evening) — the easy lever named here does not exist. This cell used to read "PVE storage status is the prime suspect, and its interval is tunable". The first half is right and the second half is false. pvestatd stats EVERY configured storage on each 10-second cycle, and Proxmox staff have stated the interval is not designed to be configurable — so there is no knob to turn down. The only lever PVE actually offers is disabling the storage entry (pvesm set <id> --disable 1) around the backup window, and that is substantially more than a tuning knob: it collides with felhom-agent/internal/pbsdr/manager.go's health model, where an inactive-but-existing entry drives the consume-the-one-time-secret recovery path. So the fix is a design question (does the hub still need a 15-minute fill reading at all, given R-339 now reports reachability separately?), not a config edit. Doc-only correction — no agent code was changed. The remaining step is unchanged: cut the poll rate by whatever means survives that question, then confirm the fd count between restarts stops climbing — the positive observable, per standing rule 3. Baseline measured 2026-08-18, and the FIRST measurement published was WRONG. The initial "~85/day, matching the ~73/day implied by the failure" came from a single 17-minute window whose delta was one descriptor — a sample of one cannot carry a daily rate, and the agreement that made it feel solid was coincidence. Re-measured over two independent windows the same morning: 183/day (31 min) and 200/day (5.6 h) — ~2.6x the published figure, putting the runway to the 65536 ceiling at ~357 days, not the ~2 years first claimed. And the named mechanism is the minority one: across that window CLOSE-WAIT held flat at 1 while ESTAB grew 45→49 — all the growth was established connections, and at the wedge the split was 1011 ESTAB / 543 CLOSE-WAIT. The fix must target connections the proxy never reaps, not just CLOSE-WAIT sockets. The PBS 4.2.5-1 upgrade (2026-08-18) did NOT change the slope and was never expected to — see R-341 SPIKE 2026-08-20 — THE PREMISE OF THIS ROW DOES NOT SURVIVE MEASUREMENT, and that is a change in what the row IS, not new evidence on it. audits/SPIKE-ep0-established-connections-2026-08-20.md. The leak is OURS, and the poll rate is not what feeds it. Every one of the 388 leaked descriptors is an ESTABLISHED connection held open by felhom-agent on the boxes — 194 on each, ss -tnp naming a single PID per box, and zero held by pvestatd or proxmox-backup-client. Confirmed independently from ep0's access log over the same 46.18 h window: libwww-perl (pvestatd) 81,192 requests -> 0 descriptors, proxmox-backup-client 80,061 requests -> 0 descriptors, Go-http-client/1.1 (the agent) 387 /snapshots calls -> 388 sockets — one per call, within one. So 162,404 requests, 99.5% of the traffic, produce 0% of the leak. Mechanism, named from source: felhom-agent/internal/pbs/client.go:56-60 builds &http.Transport{TLSClientConfig: tlsCfg} — a composite literal, so IdleConnTimeout is the zero value = no limit (http.DefaultTransport sets 90 s; a literal does not inherit it) — and cmd/felhom-agent/main.go:1486 (pbsTargetsFromPVE) builds a fresh client every cycle, as its own doc comment states. Each cycle therefore strands one idle keep-alive connection in a transport nothing ever closes; CloseIdleConnections/IdleConnTimeout/MaxIdleConns appear nowhere in the agent repo. Cadences reconcile without fitting: 900 s hub poll (184.7 cycles) + 6 h DefaultVerifyCadence (7.7 cycles) = 192.4 predicted vs 194 observed per box. CONSEQUENCE — RE-RANK. The remaining step recorded above ("cut the poll rate, then confirm the fd count stops climbing") would have produced a null result and read as a failed fix. Cutting the Proxmox poll rate removes ~99.5% of ep0's request load and zero descriptors. The poll rate is still wrong on its own terms — 85,000 requests/day to a weekly-write DR endpoint — but it is now a scaling/cost item, not the leak fix, and the leak fix is R-344. Q3 (is the leak proportional to the request rate?) is PREDICTED not-proportional and NOT YET MEASURED — Phase C is held at STOP 1 with its prediction pre-registered in evidence-ep0-established-connections-2026-08-20/phaseC-prediction.txt. Do not record a proportionality verdict here until that window has run. RE-SCOPED 2026-08-20 — THIS ROW IS NO LONGER A LEAK FIX, AND ITS RECORDED NEXT-STEP WOULD HAVE "FIXED" NOTHING WHILE LOOKING LIKE A FAILED FIX. That near-miss is the reason the spike-first rule exists and it is kept here deliberately. The old next-step read: cut the poll rate by whatever means survives that question, then confirm the fd count between restarts stops climbing. Had it been executed, the fd count would have kept climbing at the same ~200/day, the poll reduction would have been recorded as ineffective, and the real defect — ours, in felhom-agent, R-344 — would have been further from being found, not closer. Measured 2026-08-20: pvestatd (libwww-perl) and proxmox-backup-client made 162,404 requests in a 46 h window and leaked zero descriptors; the agent made 811 and leaked 388. The fix (agent 0.130.0) took ep0 from 388 accumulated descriptors to its baseline of 17, with the poll rate completely unchanged — 85,000/day before and after. WHAT THIS ROW ACTUALLY IS NOW — a SCALING concern, still worth fixing on its own merits: ~85,000 requests/day to a DR endpoint that is WRITTEN TO WEEKLY, from two boxes. That is ~42,500/box/day, so at fifty customers it is ~2.1 million requests/day — about 25 requests/second, constantly, against a CX33. The design question is unchanged and is still the hard part: does the hub still need a 15-minute fill reading at all, given R-339 reports reachability separately? And the lever remains awkward — pvestatd stats every configured storage on each 10-second cycle with no tunable interval, so the only PVE-side lever is disabling the storage entry, which collides with felhom-agent/internal/pbsdr/manager.go's health model. NEW ACCEPTANCE CRITERION, since the old one is void: the fd count is NOT the observable for this row any more — that belongs to R-344 and is already satisfied. Measure the REQUEST RATE at ep0's access log, and state the projected rate at the target customer count. |
CC |
| R-541 | Backup & restore | P4 | [P3-LOW] There is no path to move a customer between off-site boxes, or from shared to dedicated. Read from source 2026-09-16: provisioning is idempotent-reuse keyed on the customer (shared already provisioned for tester-1 (subaccount 311327)), and a dedicated deprovision destroys the repository — so "move this customer" has no safe route today. It becomes reachable the moment R-540's second pool box exists, or when a customer outgrows the shared model. Needs: a move that copies the repository, re-keys, and only then releases the old sub-account — a new mechanism nobody has measured. |
READY — rank P3-LOW; owner: CC (hub) — design first Re-ranked 2026-10-03: P3->P4: needs a second pool box or an outgrown customer first; operator-only and far off (0.3% full). | — | — | CC |
| R-691 | Backup & restore | P4 | [P3-LOW] Kept data (09 §3 decision 36): two gaps of the first build. (1) The read-only file-browser view cannot open a folder another user owns with mode 0770 — nextcloud's appdata/nextcloud is www-data drwxrwx--- (measured on 9202 2026-09-25), FileBrowser runs as uid 1000, so „Megőrzött adatok" shows the folder and not its files; the files are still listed, sized, loadable and deletable. Fix direction needs a decision (a read-only ACL, or a helper that lists as root) — not a chmod of the household's data. (2) „Use my kept data" / Load looks only at the own unit (Tier 1) and the second-drive mirror (Tier 2); an app whose only database copy is off-site gets "no backup". Controller 43e99d1. audits/night-2026-09-26/E/ -- 2026-09-25 live proof: (1) confirmed on 9202 — the view mounts nextcloud's kept folders :ro but its files are www-data 0770. Also seen: the source's name „Megőrzött adatok" is Hungarian on an English box (the file browser's config holds one name). -- 2026-09-27 (controller v0.275.0): (1) FIXED: the view joins the kept folder's OWNING GROUP when it is group-readable (never root's, never its own), binds stay :ro, nothing on disk changes (CC-unattended decision, 07 §6.5); the source's name follows a language switch (the switch re-syncs the file browser). Red-proofed, audits/version-travel-2026-09-26/D3/. NOT live-proven with a real nextcloud kept folder. STILL OPEN: (2), the Use/Load choice does not look at the off-site copy. -- 2026-09-27 (second session): (2) NOT built on purpose — it composes the unit-only off-site download (RestoreOffboxScratch(full=false)) with the unit restore into a new restore path on household data, and no box CC may touch has an off-site target to prove it on (9202 has none; 9201 on both demo hosts is fenced). Needs: a Tier-0 guest with an off-site target, or an operator word to use one.** -- 2026-09-28 (controller v0.277.0): (2) BUILT — KeptBestCopy offers the off-site copy when it is newer than every local copy or the only one; the page names the copy and its date; LoadKeptOffsite downloads the unit alone, refuses a unit of another drive, with no data, or with no recorded data version (07 §6.5/§6.6), then restores. Red-proofed (audits/kept-offsite-2026-09-28/redproofs/). Tier 1 regression live on 9202 (the choice named „saját mentés, 2026-09-28 10:06”, seed + file back). Floor 0.277.0, both demo boxes. STILL OPEN: the live off-site proof — a throwaway nextcloud on demo-hp 9201 joined the off-site copy 2026-09-28 10:15; its first snapshot runs the night of 09-28/29 (Part E (b)). Tier 2 not runnable live (9202 has one drive). -- 2026-09-28 afternoon: LIVE-PROVEN on demo-hp 9201 (controller 0.278.0), endpoint level. A throwaway nextcloud, seeded through its own front door, joined the off-site copy; the off-site run-now pushed snapshot 6cb379a8. (a) The full off-site restore (prepare → download → reconstitute): 3 volumes + the database replayed, the seed read back, a marker user written after the snapshot read ABSENT, same versions. (b) Remove keeping the data + deleting the local copies → reinstall: the choice and the kept list both named "távoli mentés, 2026-09-28 15:40" / "the off-site copy, 2026-09-28 15:40"; "use my kept data" downloaded the unit alone and loaded 3/3 volumes + 1/1 database in 55 s; the seed and the kept files read back. App removed with its data; demo-hp's app list equals the list before. audits/kept-offsite-2026-09-28/E/. |
NARROWED (2026-10-03 triage: the row's verdict was finished, but it names open work no other row carries — the second-drive (Tier 2) path of Use/Load is not proven live — 9202 has one drive) — CLOSED — controller v0.277.0, live 2026-09-28 | — | — | CC |
| R-816 | Backup & restore | P4 | No off-site failure class has ever been seen live. F-DIAG (controller v0.182.0, 2026-07-28) split off-site failures into six causes — quota, orphaned, no_repo, no_units, transport, unknown — each with its own Hungarian message. None of the six has been exercised by a real failure on a box; the recovery inventory records it only as a known limit (documentation/architecture/_recovery-inventory-2026-07-28.md:955). Filed 2026-10-03 from the F-DIAG row's residue when that row moved to CLOSED-ITEMS.md. |
READY — filed 2026-10-03 (triage); owner: CC. Exercise each class once on a scratch guest (a full quota, a missing repository, a blocked transport) and read the message the household sees. | — | — | CC |
| R-832 | Backup & restore | P4 | ep0's copy in a place outside both Hetzner and the operator's home (roadmap). Today DooPlex (the operator's home) holds it (decision 71). A Hetzner Storage Box would share a provider with ep0 and with every household's file backups, and cannot run PBS, so the copy could not be verified or restored from directly. | DEFERRED — later, if the product grows | — | — | operator |
| R-878 | Backup & restore | P4 | A catch-up (R-871) runs the database-dump leg in the DAY, and that leg stops an app with a volume for its copy — the household may notice the stop, and a large volume makes it longer. MEASURED 2026-10-05 on demo-felhom: the catch-up at 08:25:02 stopped opengist, copied 182.5 KB, started it again — about 1 s, then a few seconds of health: starting; the night does exactly the same, unseen. Nothing measured for a large volume. Fix direction (if it matters): skip the volume copy of a running app in a DAYTIME catch-up and leave it to the next night, or warn. audits/catchup-2026-10-05/partA/live-demo-felhom.txt |
READY — owner: CC 2026-10-05 (burn-down night): NEEDS A LIVE MEASUREMENT — a timed catch-up on 9202 for an app with a multi-GB volume, then the skip-or-warn choice. | — | measure a large volume first | CC |
| R-32 | Backup & restore | P4 | [P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible. The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's "hetzner":"ok" leg destroys the sub-account, but a Hetzner sub-account is an access-control object, not a data object — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. MIGRATED FROM ROADMAP.md 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state idea. Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. |
OPEN — migrated from ROADMAP 2026-08-22, rank unchanged 2026-10-06 night: design written (audits/night-burndown-2026-10-06/design-R-32.md) — pick A: purge the repo + set-aside copies through the sub-account's own password login BEFORE deleting it (no main-account credential); the „repo data destroyed" comment at hub/internal/offsite/offsite.go:276-280 is false for the shared tier; part (3) waits on a read-only measurement (du on port 23). Operator: agree to route A. 2026-10-07 07:58: 09 §3 decision 167 — option A yes, and delete the old RESET leftovers once. 2026-10-07 (Part D step 2, read only): the hub shows bx11 holding 3.3 GB of data while the four live customers account for ~0.6 GB → ~2.7 GB belongs to no live customer; no live customer shows a set-aside folder, so the leftovers are in homes whose sub-account is gone, reachable only by the pool box's MAIN account. The one-time clean-up is STOPPED: no main-account login for bx11 is held on DooPlex (the API token in the credentials file sees no box). Needs: the main-account login, or the operator lists and deletes in the Hetzner console (audits/day-2026-10-07/D/D2-listing-readonly.txt). 2026-10-07: option A DELIVERED (hub 0.142.0): RESET purges the repo and every .orphaned-* through the sub-account's own login before deleting it, and keeps the sub-account if the purge fails (red-proved, audits/day-2026-10-07/D/). Not yet seen live (no scratch RESET today). The one-time clean-up of the ~2.7 GB of leftovers is STOPPED: it needs the pool box's main-account login (see above). 2026-10-07 14:43 — 09 §3 decision 173: leave the ~2.7 GB of old leftovers for now; re-ranked P2 → P4. Option A itself is delivered (hub 0.142.0). What stays open is ONLY the leftovers: they were computed (box total minus live customers), never listed, and deleting them needs the pool box's MAIN login, which reaches every customer's data. Blocker: a safe main-account path. |
— | Ruling from the run (three parts, deliberately separate): (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable BY DESIGN → RESET gains a main-account purge of the customer base dir (the existing operator ack already covers it); (2) the move-aside guard STAYS for reinstall-without-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator Restic tab shows per-customer directory bytes vs attributed snapshot bytes, so dead data cannot hide. Measured on the pool box that night: 49 M attributed (2 snapshots, 48.717 MiB) against 1.4 G + 3.0 M unattributed across TWO .orphaned-* dirs. Evidence restic-and-pool.txt |
CC |
Storage & devices — 3 rows (P3 2, P4 1)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| R-330 | Storage & devices | P3 | Disk health Phase 2 — the three SMART attributes the wire does not carry. The failing drive's most telling counter was 187 Reported_Uncorrect, sitting at normalized 1 against threshold 0 with a raw count of 1001 — one point from failing and structurally unable to get there. Also wanted: 199 UDMA_CRC_Error_Count (cabling) and 188 Command_Timeout. None are on the agent→controller wire today, so v0.215.0's ladder could not use them. Phase 2 also persists periodic SMART samples (the right home is metrics.MetricsStore, NOT the Phase-1 state file, which is one record per disk and must stay that way). This is a declared WIRE change, so under the G-1 gate the hub must model the new fields in the SAME session — that is precisely why it was kept out of Phase 1, where it would have turned a one-word severity fix into a three-repo change |
READY — 2026-10-12: the wire half is CLOSED in this row — released in agent v0.150.0 (boxes run 0.154.0), controller and hub decode 187/188/199. Only open work: the verdict slice (use the three counters in the disk verdict and the household's words) and persisting samples. READY (M) — NEW 2026-08-14 2026-10-06 night: the wire half BUILT on main (unreleased): the agent sends SMART 187 reported_uncorrect, 188 command_timeout (raw, vendor-packed on some drives — carried as reported), 199 udma_crc_errors (pointer + omitempty: unknown is absent, never 0); the controller decodes them and the hub's host-report mirror models them (G-1 wire gate green; it convicted the agent half alone). Carried only — no verdict, banner, mail or alarm reads them (TestR330_CountersChangeNoVerdictYet); using them changes what a household is told and is the next slice, with persisting samples. Red-proofs audits/night-burndown-2026-10-06/r330/. Ships with agent v0.150.0 and the next controller and hub releases. |
R-328 (closed) | The verdict slice: 187/188/199 into the disk verdict with tests; then persist samples | CC |
| R-332 | Storage & devices | P3 | The new Hiba-from-counters path has never fired on real hardware. v0.215.0's whole point is a verdict the product could not previously reach, and it is proven only against the committed fixture's values in unit tests (12 scenario groups, 11 of 12 red-proofs failing as required). The live validation on demo-hp proved the negative — three healthy disks still read Rendben across the deploy, no false alert — and the severity wire end to end, but no live disk has actually reached Hiba. This is the honest gap and it must not be closed by pointing at the fixture tests: the drive that produced the fixture is in DooPlex, which is Tier 2 and never a drill target, and the demo boxes are all-flash and healthy | WATCHING — NEW 2026-08-14, NARROWED same day. One item originally in this gap is now PROVEN LIVE: the persisted state surviving a controller restart. The v0.215.0→v0.216.0 redeploy destroyed and rebuilt the container, and the new one read back a changed_at written by the PREVIOUS version (2026-08-14T07:23:14.640216851Z, still intact at 09:31:35Z) instead of re-baselining — Scenario L on real hardware, not just the production-path unit test. What remains unproven is the verdict itself, plus the stronger restart half: an already-ALERTED disk not re-alerting |
a real degrading disk, or an injection harness | Closing condition: a live disk reaching Hiba from counters, OR a deliberate injection through the REAL pipeline (agent /disks → controller check → hub event), not a hand-set verdict |
CC |
| R-331 | Storage & devices | P4 | Disk health Phase 3 — growth-rate detection, and retiring the static 64. The v0.215.0 count backstop (64 unreadable sectors → Hiba) is a judgement from ONE drive: the observed benign excursion peaked at 16 and cleared inside an hour, and the terminal run passed 64 at 13 Aug 11:28 and never came back. It is deliberately a backstop BEHIND the sustain rule, not the primary signal, but it is still a magic number tuned on a single sample and it will be wrong for some drive. With Phase 2's history the box can ask the question that actually matters — is this count climbing, and how fast — which distinguishes a drive with eight stable aging sectors from one adding forty a day, something no static threshold can do. Revisit 64 when that exists | READY (M) — NEW 2026-08-14 | R-330 | Growth-rate rule over persisted samples; re-derive or delete the static 64 | CC |
Security & access — 16 rows (P1 1, P2 3, P3 8, P4 4)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| R-985 | Security & access | P3 | Our zero-knowledge wording is stronger than the product: on a live box the operator can reach the off-site backup key (through the vaulted root credential and the operator tunnel) without the customer's recovery code. Design-shaped; decision for the operator in STATUS.md. Fix options: change the wording, or remove operator root reach to the key. Security review 2026-10-12 (audits/security-review-2026-10-12/); detail held off-git for the operator (/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/). |
WAITING-ON-OPERATOR — design-proposal | operator decision | Decide the wording or the change | operator |
| R-983 | Security & access | P3 | Seven small dashboard weaknesses: after sign-in the browser can be sent to another site; the server-reboot route acts on a GET, so a link can reboot the guest; no anti-framing headers; parallel sign-in attempts get past the limiter; the settings page allows 8-character passwords; reset-code requests have no limit on the box; logout can be triggered from another site. Fix each with a test, in one controller release. Security review 2026-10-12 (audits/security-review-2026-10-12/); detail held off-git for the operator (/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/). |
OPEN | — | One hardening release | CC |
| R-977 | Security & access | P3 | FileBrowser's default password can come back if its data volume is ever re-created, because the box checks it only once; and the tunnel can start before the box is claimed. Static; needs a live check on 9202. Fix: check at every start, and do not start the tunnel before the claim. Security review 2026-10-12 (audits/security-review-2026-10-12/); detail held off-git for the operator (/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/). |
OPEN | — | Measure on 9202; then fix | CC |
| R-976 | Security & access | P3 | A family member's session can be copied out of a family-gated app, and a dashboard password reset does not end family or gate sessions; so a stolen session keeps working for up to 30 days. Fix: give each app its own derived token instead of the session itself, and end every family and gate session on a reset. Security review 2026-10-12 (audits/security-review-2026-10-12/); detail held off-git for the operator (/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/). |
OPEN | — | Fix with tests in a controller release | CC |
| R-970 | Security & access | P2 | Whoever holds a box's own hub API key — the hub, or anyone with a copy of the hub database — can read the box's settings file with every secret in it and replace that file through the internet; the box does not check that such a change is signed. This contradicts 01 §4 (the box exposes no control endpoint) and §10 (a hub compromise alone cannot forge commands). Nothing in the product calls this route any more. Fix: remove the route, or require an operator-signed op for it. Security review 2026-10-12 (audits/security-review-2026-10-12/); detail held off-git for the operator (/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/). |
OPEN — confirmed by a second reader | — | Remove the route with a test that it is gone; release the controller | CC |
| R-954 | Security & access | P2 | We cannot tell whether anyone used the old hub operator key (R-950) or the open hub read routes (R-951) before 2026-10-12; if someone did, they could have read every box's break-glass root credential and every customer's box report. The exposure began in February 2026; the logs that could show it cover only 2026-10-12 from about 09:54 and one day in June, so about 238 days are not visible. Fix (operator decision): change the break-glass root password of every box (tester-1 holds a real person's data), or accept the risk; either way, keep ingress access logs long enough to answer this next time. | WAITING-ON-OPERATOR — filed 2026-10-12 by the security review; the log search was not completed in-session | operator decision | Decide: rotate every box's break-glass root password now, or accept | operator |
| R-955 | Security & access | P4 | The cross-box check of the hub API is done in source and tests but not fully proven: if a later check masks a missing id check, a box's key could read or change another box's hub record without any test noticing. Read 2026-10-12: every box-facing /api/v1 route takes the box from its own key or refuses a different id; 14 routes already had a refusal test, 5 got one (crossbox_refusal_test.go). Left: red-prove the 4 new tests that were not red-proved (as the restore-directive one was), and the live read-only check with one demo box's key against the other demo box's id. |
OPEN — filed 2026-10-12 by the security review | — | Red-prove the 4 tests; run the live GET check on the two demo boxes | CC |
| R-924 | Security & access | P2 | The operator signing keys exist only on DooPlex, with no passphrase and in no backup — a loss of DooPlex means no box ever accepts a signed agent update, config bundle or OS step again without an on-site re-enrollment. MEASURED 2026-10-09: /mnt/5_hdd/felhom.eu/felhom-op-operational, felhom-rec-recovery, felhom_op_ed25519 (0600 kisfenyo); ssh-keygen -y -P '' opens all three (no passphrase); /mnt/5_hdd/felhom.eu is in no backup set, the off-site job does not carry them. 04 §1 says the operational key is "passphrase-protected" and the recovery key "kept offline" — today neither holds (custody ruling 1 of 2026-09-16 put both on DooPlex for this phase; 04 §3.1). Both keys share one fate, so the cold key cannot do its one job (re-pin after losing the operational key). runbooks/total-loss-of-dooplex.md S6. |
NARROWED 2026-10-09 — operator ruling: option A (print both AND copy them). The three key files and their public halves ride the nightly encrypted DooPlex→ep0 copy (scripts/dooplex-offsite/, a missing key refuses the push); the Sunday restore test proves each private key still derives its own public half (never printed). LEFT: the operator prints S6a/S6b (runbooks/break-glass-sheet.md); the keys still have no passphrase (custody ruling 1 of 2026-09-16 stands until the first sale). |
operator: print | Operator: print the two keys; then close | operator |
| R-132 | Security & access | P3 | curl -w '%{redirect_url}' reconstructs the request URL WITH its basic-auth credential — so a -u ":$HUB_PW" call that never put the password in a URL still printed it Merged 2026-10-05 from R-350 (duplicate): (1) 2026-08-20 occurrence: POST /configuration/artifacts answers 303; leak lives only in the CC transcript under ~/.claude/projects/ on DooPlex, not in git/evidence (checked then). (2) -v and --libcurl also re-render the credential, not only %{redirect_url}; confirm redirects with %{http_code} + follow-up GET. (3) Rotation path: hub /configuration form (current_password/new_password/confirm_password); DB override wins over ConfigMap (break-glass); CC can rotate file-to-file without printing (operator-present-one-time-secrets) if asked. |
WAITING-ON-OPERATOR — ACTION: rotate HUB_PW Folded R-580 2026-10-03 (the same curl -w %{redirect_url} credential echo, seen again 2026-09-18). Note 2026-10-12 (security review): manifests/hub.yaml also carries the hub login's bcrypt hash in this public repository (the break-glass field); when HUB_PW is rotated, decide whether that field should stay empty. |
— | Happened on 2026-07-31 while red-proofing the R-120 gate: the hub operator password was written to the session transcript by the write-out format, not by the request. -u is safe; the reporting was not. Rule: read the redirect from -D - and grep ^Location:, never %{redirect_url}, on any authenticated call. Rotate the hub password (/configuration → Login password; ConfigMap auth.password_hash is the reset path) and update ~/.config/credentials |
Viktor |
| R-137 | Security & access | P3 | Cloudflare geo-WAF rules are zone-scoped and non-namespaced — four cross-tenant faults. globalRuleDesc = "[felhom-geo] Global" (waf.go:18) is one literal description per ZONE; appRuleDescPrefix keys by app name with no customer (waf.go:21); BuildGlobalExpression has no positive hostname scoping (waf.go:241); applyDiff deletes every [felhom-geo] rule not in THIS box's desired set (geosync.go:320) |
READY (M) — blocks shared-zone onboarding | — | With two customers in one zone: they overwrite each other's Global rule forever; one customer's country policy applies zone-wide; per-app rules collide by name; and disabling the feature for one (or the hub's RemoveGeoRules) wipes them all. Interim mitigation, no code: keep geo-restriction OFF for every shared-zone customer. Fix = namespace descriptions by customer_id + add http.host ends_with "<domain>" to both expressions — a TWO-REPO change (controller + hub RemoveGeoRules). Same audit §5.1 |
CC |
| R-775 | Security & access | P3 | [P2-MEDIUM] Grimmory: a stranger's 5 wrong sign-ins lock EVERY visitor out of the web login for 15 minutes — so Grimmory was not published. MEASURED 2026-10-01 on 9202 (drill catalog, v3.4.1, through traefik): after 5 wrong tries for admin every further sign-in answered 429 — the household's right password AND a different name — and stayed 429 for 10+ minutes of retries. Read in the jar: AuthRateLimitService — Caffeine expireAfterWrite(ofMinutes(15)), MAX_ATTEMPTS 5, keys login:ip: and login:user:; Spring forward-headers-strategy: native takes the address from X-Forwarded-For, and behind the tunnel every visitor is the tunnel container's address (R-753) — the wger shape (R-752), with no setting to change it. Everything else in the checklist passed (bench + box step v3.4.1 → v3.5.0, gate by its own probe, OPDS through traefik); two smaller findings for the publishing session: on a reinstall over the first install's kept books, a new upload was saved to the drive but not added to the library (box/grimmory/reinstall-c1.txt, not investigated); and the remove + restore round trip (2.5) cannot be shown on 9202 for a drive app — its backup lives on the scratch drive, which is not a registered drive (R-756). The template waits in audits/new-apps-2026-10-01/wip/grimmory/. Needs (operator): (A) publish with a sentence on the page that wrong guesses by others can lock the login for 15 minutes (MEASURED: during the lock an e-reader's OPDS feed still answered 200 with its own login, wrong 401 — box/grimmory/opds-under-lock.txt), or (B) wait until the box passes each visitor's real address (R-753). audits/new-apps-2026-10-01/box/grimmory/throttle.txt -- 2026-10-01 (evening): option B's precondition SHIPPED (controller v0.286.1, R-753): Grimmory's Tomcat RemoteIpValve walks from the right and counts 172.16.0.0/12 as a proxy (READ in source, Spring Boot 4.1.1 — not yet measured with Grimmory's own lock), so login:ip: becomes per visitor; login:user: still lets a stranger lock the public name admin 15 min. A third route was spiked and passed: Grimmory behind the permanent family gate with its e-reader paths excepted (R-780). Recommendation: publish behind the family gate if R-780 is built; otherwise B with a measured 3.6. UPDATE 2026-10-02 — NARROWED, Grimmory PUBLISHED behind the family gate: a stranger cannot reach Grimmory's web sign-in at all (6 tries through the simulated tunnel: the gate's 401, then the household signs in 200 — audits/family-gate-2026-10-02/A/items.txt), and the e-reader exceptions keep Grimmory's own login. The 2.5 round trip now WORKS on 9202 (audits/family-gate-2026-10-02/B/box/life.txt). What is left: a family member past the gate can still lock a NAME (the admin's) for 15 minutes with 5 wrong tries — hard-coded in Grimmory; and the reinstall-over-kept-books finding (new-apps-2026-10-01/box/grimmory/reinstall-c1.txt) is still not investigated. |
WATCHING — rank P3-LOW; owner: CC Re-ranked 2026-10-03: P2→P3: Grimmory is now behind the family gate; only a family member can still lock a name. | — | — | CC |
| R-908 | Security & access | P3 | An old Resend API key is still in the history of the homelab-manifests repository; it was replaced on 2026-06-29, but whether it is also revoked at Resend is unknown. FOUND 2026-10-08 by the contact-mailer source search (audits/mailer-source-2026-10-08/SEARCH.md, „A side finding"): commit c9648cd (2026-02-05, „added mailer pod") put a key literal in the old felhom-system/contact-mailer.yaml comment; the file was deleted in ee93b50 but the history keeps it. Compared by hash only, never printed: it is NOT today's key (Secret/resend-api, rotated 2026-06-29, felhom.eu feea0606). If the old key still works at Resend, anyone with read access to that repository can send mail as felhom.eu. |
OPEN — owner: operator Note 2026-10-12 (security review): gitleaks 8.24.3 with its default rules did NOT flag this key in the full history scan, so a scanner gate alone would not have caught it (R-986). | — | In the Resend console, check that the old key is revoked (revoke it if not); rewriting the repository's history is NOT needed once it is revoked | operator |
| R-525 | Security & access | P4 | [P3-LOW] FileBrowser has its own login; putting it behind the dashboard session (traefik forwardAuth or Quantum proxy auth) is a new mechanism nobody has measured. Filed 2026-09-15 by the P1-fixes task (B.5). R-513 closed the default-password hole with a generated password; a household still has two logins. What it needs: a spike on a scratch guest — forwardAuth to the controller session, and what FileBrowser Quantum does with a trusted header. | READY — rank P3-LOW; owner: CC (spike) Re-ranked 2026-10-03: P3->P4: comfort feature needing a new unmeasured mechanism; the default password hole is closed. | — | — | CC |
| R-925 | Security & access | P1 | manifests/felhom.secret.yaml carries REAL secret values, and the Gitea repo that holds it is readable by anyone on the internet with no login. FOUND 2026-10-09 by a background security review of the legal-pages commit, which flagged the served <!-- source --> comments; chasing what those comments pointed at found this instead. MEASURED, in this order: (1) scripts/manifest_bearer_gate.py itself prints manifests/felhom.secret.yaml:39 KNOWN-BACKLOG committed secret 65cee3c4…7a86; (2) the file holds live-shaped values, not placeholders — SECRET_KEY (69 chars), SUPERUSER_EMAIL/SUPERUSER_PASSWORD (18), Umami APP_SECRET (66) and POSTGRES_PASSWORD (34), and a username/password pair (18); values were never printed, only measured by length; (3) gitea.dooplex.hu resolves to 37.191.56.193, the same public address as the website; (4) an anonymous curl returns HTTP 200 and 1686 bytes for that exact file, and 200 for hub/internal/store/store.go and manifests/hub.yaml; (5) the off-network control, which is what makes (4) mean anything — fetched from outside the operator's network entirely: a real path returns the file's first line, a nonsense path returns 404, so the 200 is genuine anonymous read from the internet and not a LAN-only ACL. This contradicts the project's own written model: documentation/runbooks/secrets.md:3-4 says “Secret values are never committed to git. The manifests in manifests/ carry only placeholders + comments that point here.” That promise is false today, which is the P1 wording in this register's own scale — see the ranking note below. The same runbook already states the rule that governs the fix: “The git-history copy stays alive until the value is ROTATED — de-git alone kills nothing.” Known neighbours, none of which cover this: R-887 records that an outside crawler walks the public Gitea pages and leaves “Gitea's exposure to the crawler” as the operator's undone item; R-580 still owes a Gitea admin token rotation. Neither says a secret file is world-readable. Why P2 and not P1, stated so the operator can overrule it: the exposed credentials guard analytics and an undeployed healthchecks instance, not household data — measured: umami-db is a ClusterIP service on 5432 with no external IP, so the Postgres password is not reachable from the internet; the realistic harm is session forgery against the public stats.felhom.eu via APP_SECRET, and reuse of those passwords anywhere else. No customer box, hub token or escrow key is in this file. CC did NOT change anything: making the repo private could break the public day-0 path (the installer is fetched from a public tag, R-110), and rotation plus repository visibility are operator decisions on production infrastructure. PARTLY FIXED 2026-10-09, and RE-RANKED P2 -> P1 on what it turned out to be. The exposed value was not an analytics password: it was the Gitea admin account password (is_admin: true; /api/v1/admin/users answered 200), published in a repo gitea.dooplex.hu serves anonymously to the internet. That is push access to every repo — including the one whose website/ is git-synced live and whose scripts/ is published by tag to every new box installer (R-110), i.e. a supply-chain path onto customer hardware. My first ranking said P2 because I had only measured the analytics blast radius; the credential test is what corrected it. DONE (CC, operator-instructed): (1) umami-config APP_SECRET + POSTGRES_PASSWORD rotated — the password changed inside Postgres too (ALTER USER), because the env var is only read at first init; verified by a real beacon returning 200 with a bogus-site-id control still returning 400. (2) healthchecks-config SECRET_KEY + SUPERUSER_PASSWORD rotated (nothing consumes them — there is no healthchecks Deployment). (3) gitea-creds no longer holds the admin password at all: it holds a scoped token (read:package + read:repository). Both scopes are load-bearing and the second was found by breaking it — a package-only token made the hub log Template fetch: unexpected status 403, because the template fetcher reads a raw file out of the felhom-controller repo. (4) The Gitea admin password was changed and gitea-system/gitea-admin updated; verified: new password 200, old published password 401 — the leak is dead. (5) The file is removed from git and .gitignore's *secret* now applies to it; manifest_bearer_gate.py's KNOWN_BACKLOG carve-out is gone, red-proofed with a decoy (exit 1 with, 0 without). AN INCIDENT CAUSED BY THE FIX, recorded because it is the useful part: the rollout restart needed to pick up the new umami secret put umami into CrashLoopBackOff and took stats.felhom.eu down (503). Cause was not the rotation — at memory: 512Mi the pod runs for months but cannot restart, because startup (Prisma + Next.js) peaks over the limit and is OOMKilled (exit 137). Raised to 1Gi, in the manifest, not just live (the .claude/rules/manifests.md rule: a bare kubectl set is reverted by the next sync and the fix silently disappears). Service restored and verified. New values are in a 0600 file on DooPlex (~/rotated-secrets-2026-10-09.txt), never echoed; the operator moves them to the password manager and deletes it. WHAT REMAINS, and it is bigger than what was fixed: that one password is still the admin password in about 12 other namespaces — nextcloud, paperless, bookstack (a database root password), tandoor, calibre, adventurelog, gokapi, qbittorrent (x2), servarr, homepage (x2). Rotating Gitea does not touch them: each is its own login and each is still the string that was published. Also owed: a Gitea token for user kisfenyo sits in plaintext in the origin URL of the local homelab-manifests clone (.git/config); CC printed it to a session transcript while investigating, so it should be rotated regardless — this is R-580's shape (store the remote without credentials). Not a finding: homelab-manifests itself is private (404 anonymously) and does not contain the password; ArgoCD's repo credential is a separate token, so the rotation did not touch it. |
NARROWED 2026-10-09 — the felhom.eu half is DONE and verified (Gitea admin password rotated, old one now 401; umami + healthchecks rotated; file de-gitted; gate carve-out removed). What remains is the ~12 reused logins elsewhere in the homelab and the repo-visibility decision; owner: operator 2026-10-10 (release session), two consequences measured: (a) DooPlex's own Docker login for gitea.dooplex.hu (~/.docker/config.json) still holds the OLD admin password — the registry token endpoint answers 401 for it and 200 for the current one — so every image push from DooPlex fails unauthorized until it is refreshed; the 0.145.0/0.305.0 pushes used a one-off login in a scratch DOCKER_CONFIG, shredded after. (b) The felhom ArgoCD app reads OutOfSync for Secret/gitea-creds, Secret/healthchecks-config, Secret/umami-config and Deployment/umami: a whole-app sync would push git's (de-gitted) view over the rotated live Secrets — the hub deploy synced ONLY Deployment/hub (audits/release-2026-10-10/hub-0.145.0-deploy.txt). Refreshing the DooPlex login is a DooPlex change (operator's word). |
— | OPERATOR RULED 2026-10-09: (a) he rotates the remaining services himself — CC's scope stopped at the Felhom boundary; (b) the repo STAYS anonymously readable for now, because making it private breaks the website git-sync and the installer tag fetch, which both clone with no credentials (R-110). The standing consequence of (b): no secret may ever enter this repo again, which manifest_bearer_gate.py now enforces with no exemption; and if (b) is reversed, the 32 <!-- source --> comments on /adatkezeles must be stripped in the same change. The exact checklist of the 12 remaining secrets (namespace / secret / key, re-measured after the rotation, with the two traps that bite — a DB password is not changed by editing the Secret, and a pod that has run for months may not restart) is in documentation/runbooks/secrets.md. Operator: (1) work that checklist, bookstack-db/root-password first because it is a database root password and every one of these hosts answers on the public internet; (bookstack first — it is a database root password); (2) rotate the kisfenyo Gitea token embedded in the local homelab-manifests remote URL, and store the remote without credentials; (3) decide whether gitea.dooplex.hu should answer anonymously at all — remembering the website git-sync clones it with no credentials and the installer is fetched from a public tag, so making it private breaks both unless they get credentials first, and the 32 <!-- source --> comments on /adatkezeles become a map of it; (4) move ~/rotated-secrets-2026-10-09.txt into the password manager and delete it. If nothing is done: the published password keeps opening a dozen services, even though Gitea itself is now safe |
operator |
| R-779 | Security & access | P4 | [P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box. Measured 2026-10-01 19:51 UTC (audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. Needs: one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. |
WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up. | — | — | CC + operator |
| R-904 | Security & access | P4 | Cloudflare can read every household's app traffic; replacing it with our own relay is a later item. Facts (reviewer discussion 2026-10-08; 01): app traffic and the dashboard reach the box through the Cloudflare Tunnel (01 §5 trust table, rows end-user ↔ apps and customer ↔ controller UI; §7), and the tunnel's public end is Cloudflare's edge, where TLS ends — so Cloudflare can technically read that traffic (the FAQ says so since 2026-10-08, R-900; „TLS ends at the edge" is not written in 01 — add it there). What Cloudflare gives today, free: inbound reach with no router setup, the CGNAT answer (01 §4, §7); certificates (01 §7, the free tier covers one level below a zone); the geo-WAF the hub enforces (01 §5 last row, §7); flood protection (not in 01). The alternative named: our own EU relay over WireGuard with TLS passthrough by SNI, certificates on the box, the geo-block on the relay. Its costs: one more machine the operator keeps up, and a single point of reach for every box; weaker flood protection; about a week of work after a spike. Operator ruling 2026-10-08 09:07 (09 §3 decision 184): a later item. |
DEFERRED — 2026-10-10: the asked sentence is in 01 §7 („TLS ends at Cloudflare's edge …”); the spike stays deferred. DEFERRED — after the first customers (operator ruling 2026-10-08 09:07) |
— | A spike after the first customers (the relay's reach, cost and flood behaviour, measured) | operator |
Box system & updates — 8 rows (P2 1, P3 6, P4 1)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| R-984 | Box system & updates | P4 | Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN. Fix each in its own small change. Security review 2026-10-12 (audits/security-review-2026-10-12/); detail held off-git for the operator (/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/). |
OPEN | — | Batch into hardening work | CC |
| R-982 | Box system & updates | P3 | The Proxmox host accepts root password login over SSH from the LAN (measured on demo-hp), so anyone on the household network can try the root password. Design-shaped: the break-glass password exists for the console. Fix: key-only SSH for root; the password stays for the console. Evidence: audits/security-review-2026-10-12/evidence/demo-hp-host-reads.txt. |
OPEN — design-proposal | — | Decide; then set it in the installer and the config bundle | CC |
| R-979 | Box system & updates | P3 | Controller updates are trusted by registry tag alone (no digest, no signature), and the Update button offers any newer tag, even an unvouched one; the built-in traefik, cloudflared and FileBrowser images are also pinned by tag only. Whoever can push to the registry decides what runs. Fix: pin by digest from the hub's vouched list; offer only vouched versions. Security review 2026-10-12 (audits/security-review-2026-10-12/); detail held off-git for the operator (/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/). |
OPEN — design-proposal | — | Design note, then build | CC |
| R-974 | Box system & updates | P3 | A compromised host agent can likely become root on the host through the root wrapper that applies PBS settings: the wrapper accepts a folder path inside the agent's own writable tree and then acts on it as root. Static (the symlink behaviour was reasoned, not run). Same class as R-861. Fix: a fixed root-owned folder, and refuse symlinks before any root action. Security review 2026-10-12 (audits/security-review-2026-10-12/); detail held off-git for the operator (/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/). |
OPEN | — | Fix the wrapper with a test on a throwaway box | CC |
| R-972 | Box system & updates | P2 | The shipped controller and hub programs are built with Go 1.24.13 and golang.org/x/crypto v0.31.0, and the agent with Go 1.26.0, so dozens of fixed Go security bugs (including html/template escaping fixes) are still in what runs on boxes and the hub. Go 1.24 has had no security fixes since Go 1.26 was released (Go supports the two newest major releases: https://go.dev/doc/devel/release#policy); the scanners name fixes in Go 1.26.9 and 1.27.2. Also: the controller image's Debian 12 base has 12 fixable critical packages, and the Samba image has 48 fixable high ones. Fix: build all three on a supported, patched Go; update x/crypto; rebuild the base images. Scanner output: audits/security-review-2026-10-12/scanners/. |
OPEN — measured on the shipped binaries | — | Bump the toolchain and bases in their own release task | CC |
| R-862 | Box system & updates | P3 | Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its felhom-os-apply (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it. FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; felhom-op cannot become root). The steps: runbooks/config-bundle.md "Tester 2". Then CC signs the bundle and reads it back. |
WAITING-ON-OPERATOR — the operator said (2026-10-04 ~19:05) he will try through his tunnel 2026-10-07 07:58 (09 §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD. 2026-10-07 (Part H, read only): in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (audits/day-2026-10-07/H/). |
the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator |
| R-78 | Box system & updates | P3 | local_api authority ruling — auto-reconcile vs detect-only MIGRATED FROM ROADMAP.md 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state idea (deferred OUT of R-77 on purpose). Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove. |
OPEN — migrated from ROADMAP 2026-08-22, rank unchanged | — | R-77 ships detection because the fix is genuinely undecided, and both directions can lose customer-visible function. Direction 1 (today): controller.yaml wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. Direction 2 (bootstrap.json wins, auto-reconcile on boot): a guest whose controller.yaml is CORRECT and whose bootstrap.json is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a working channel clobbered on the next restart, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — mergeLocalAPI replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. |
CC |
| R-468 | Box system & updates | P3 | [P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13). 25 goldens in 26 days in August, almost one per release, because golden_currency_gate.py trips on every release by design and the only honest ways past it were a bake or a declared --no-verify (thirteen by 2026-09-01, R-404/R-417). The ruling: bake WEEKLY, and always before any drill or fresh install. Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. The mechanism (built 2026-09-13): documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml, four lines (issued, expires, reason, register_row: R-468), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. The 14-day cap is enforced by the gate, not the runbook — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. It never covers a golden that is UNRECORDED (R-385) — that is not a cadence choice. A dated waiver cannot be forgotten; it just expires — the difference from R-242's original rule, which recurred the day after it was written. Tests: scripts/test_golden_currency_gate.py cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only expires — 2). This is a PRE-CUSTOMER arrangement: the first external install retires it (delete the file in that commit). Cadence written into RUNBOOK-manual-build.md §4.2 and the felhom.eu end-of-session checklist. Does NOT touch R-242's open half (nothing gates the VOUCH). |
WATCHING — rank P3-LOW; owner: CC (renew ≤ 14 days or bake); retire at the first external install | — | — | CC |
Monitoring & notifications — 6 rows (P2 1, P3 4, P4 1)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| R-528 | Monitoring & notifications | P2 | [P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: OOMKilled stays false and no oom event fires, so the v0.243.0 OOM line is not proven live. MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with OOMKilled=false and zero docker events --filter event=oom; a memory hog inside the running container was killed (rc 137) with the same silence (E2-oom-signal-measure-9202.txt). BIGNIGHT VM 333 did read oomkilled=true, so the shape differs by case. Fix shape: the agent reads the guest container cgroups' memory.events oom_kill counters (host-side, reliable), or the controller alarms on a restart-count trend RE-MEASURED 2026-09-16 on the DRILL box (fresh install, nested VM 334, Docker in an LXC guest, controller 0.243.0), so the finding is not a property of one machine: the Paperless webserver was capped at 128 M with docker update --memory; it restarted 9-10 times, and all three signals stayed silent - OOMKilled=false on every inspect, docker events --filter event=oom EMPTY for the whole window, the container's cgroup not visible from inside the guest, and dmesg unreadable there. Identical to scratch 9202. So the v0.243.0 OOM line cannot fire on ANY Felhom box as shipped, on either host. Evidence: audits/evidence-drill-0243-2026-09-16/phase2-m1-oom.txt. |
WATCHING — 2026-10-12: the built part is released (agent v0.150.0 carries the Docker-approval memory-kill check; the boxes run 0.154.0; hub 0.140.0+ requires a passing oom_check before a Docker set is approved). Per decision 157 nothing else is built; the row watches for a box whose Docker reports a false OOM flag. Dated look: 2026-11-12. READY — rank P2-MEDIUM; owner: CC 2026-09-17 (chaos night): an OOM WAS detected on a fresh box, and named precisely. On tester-1-022354 (controller 0.245.0, guest 9201, 6 GB RAM) immich’s Postgres was killed by the memory limit during its reverse-geocoding import, and the controller pushed app_oom (warning, operator-only): „Alkalmazás memóriája elfogyott: immich (immich-postgres) — egy folyamatát a memóriakorlát leállította” — naming the app AND the exact container. The visible consequence was write CONNECTION_CLOSED immich-postgres:5432 and twelve restarts of immich-server. So on THIS box the OOM scan works and was the fastest route to the diagnosis; recorded here rather than filed as a new row. Evidence: audits/evidence-chaos-night-2026-09-17/round-2.txt. 2026-10-05 (burn-down night): a one-page design proposal (no code) is in audits/night-burndown-2026-10-05/design-R-528.md — for the operator. 2026-10-06: STOPPED BEFORE ANY CODE — the measurement contradicted the design (decision 155 said A then C). On scratch 9202, Docker 29.8.2, the OOMKilled flag was TRUE in all four shapes tried: a child process killed while the container kept running (oom_kill 0 → 3, flag true) and three main-process kills (exit 137, flag true); an oom event each time. The false flags of 2026-09-15 were on Docker 29.8.0. Cost read for the record: one exec read of 21 containers on demo-hp = 1.6 s. audits/design-build-2026-10-06/C/. 2026-10-06 18:24: operator ruling, option A (decision 157): nothing is built; the row stays open as a WATCH for a box whose Docker reports a false flag. Added: a Docker engine set may not be approved until it reports a memory kill correctly on the boxes that ran it (built in this row's next session). 2026-10-06 (night): the Docker-approval memory-kill check is BUILT — hub v0.140.0 (LIVE: the approval waits for a passing oom_check on every ring-0 box; a failed, errored or missing one blocks) and the agent's wrapper (felhom-agent acccb66, unreleased — ships as v0.150.0 after the 2026-10-07 read-back). Proven by hand on demo-hp's guest: OOMKilled=true, exit 137, the oom event — seen only with an events window ending after the run (the wrapper waits 2 s). audits/readback-2026-10-07/F/. |
— | 2026-11-12: read the hub for any failed oom_check or an app_oom with no restart reason since 2026-10-12; none → close |
CC |
| R-340 | Monitoring & notifications | P3 | The new reachability check does not touch the surface that actually failed. R-339 reports when the hub cannot READ ep0 — but the read it performs is the usage op, which is proxmox-backup-manager plus df over SSH, and therefore rides the local API daemon. The 2026-08-18 incident explicitly CLEARED that daemon: proxmox-backup.service was healthy throughout, and it was the HTTPS proxy on 8007 that was wedged with a full accept queue. So R-339's check would have returned green for all 9 h 37 m of that outage. It closes the case where ep0 is unreachable as a host; it does not close the case that actually happened. This is not a defect in R-339 — it is the honest boundary of what it watches, recorded so a future reader does not mistake a green box gauge for a working off-site tier |
READY (M) — NEW 2026-08-18 | a tenantsync endpoint-script version bump (the op is added on ep0, so it needs the same version-gated rollout ErrUsageUnsupported already models) |
Add a health op to scripts/felhom-tenantsync.sh that probes https://127.0.0.1:8007/ on ep0 and reports the proxy's fd count and listen-queue depth, then surface it as a third signal. Overlaps the connections spike (R-336's remaining half): both want the same observations from ep0, so whichever runs SECOND must reuse the first's evidence rather than re-measuring a protected machine twice REUSE, per this row's own instruction — the connections spike ran FIRST (2026-08-20) and already produced most of what the health op wants; do not re-measure a protected machine a third time. Available in audits/evidence-ep0-established-connections-2026-08-20/: the proxy fd count and its type breakdown (lsof + /proc/<pid>/fd), the listen-queue depth (ss -lnt — Recv-Q 0, Send-Q 1024), the ESTAB/CLOSE-WAIT split, the per-peer connection histogram, a 31-minute persistence diff of full 4-tuples, and a 46.18 h slope with Poisson bounds. What the health op would still add beyond these: a loopback GET https://127.0.0.1:8007/ probe — the observation that distinguished "process problem" from "network problem" on 2026-08-18 and the one thing this spike did NOT take, because it is the surface R-339 cannot see. And this spike sharpens what the op should report: a rising ESTAB count is the live signal (CLOSE-WAIT was 0, not merely flat), and per R-344 the fd ceiling that matters may be the agent's, not only ep0's. |
CC |
| R-388 | Monitoring & notifications | P3 | PRODUCT DECISION (not a defect): the customer notification model is the wrong shape, and the settings page grows by one toggle per detector. The operator's framing, recorded verbatim 2026-08-23: "A customer should be notified only about things they can act on or are responsible for — the drive they unplugged, the storage they filled. A failed backup is our incident, not theirs. The intended shape is that we detect it, we tell them we noticed and are dealing with it, and they are not handed an error they cannot solve. The subscription should feel like being looked after, not like being on call." Today's page is the opposite shape — one switch per detector, and it grew from 12 to 15 in a single session (one new alarm plus two compound toggles split into four). That growth is the argument, not an aside: a page that grows per detector keeps asking a household to make engineering decisions. | READY — 2026-10-12 evening: slice 1 delivered (operator Q1/Q2 „as picked", 09 §3 decisions 234–235): the page shows three group switches; the per-detector switches and the cooldown sit under „Részletek"; an „egyedi" group is saved back unchanged; the default list fills groups 1 and 3 (households with no saved list), group 2 stays „egyedi" (app_start_failed off, R-329). Tests red-proved. controller 0.309.0, delivered 2026-10-12 18:08 to demo-hp, demo-felhom and Tester 1 (controller-0.309.0-delivery.txt). READY — 2026-10-12: the design is written (08 §8.1): three group switches („A gépem és a meghajtóim", „Az alkalmazásaim", „Amit mi intézünk") mapped onto today's 18 detectors; option A picked; slice 1 changes no wire. Two questions for the operator (group 3 default; the cooldown field). READY — operator ruling 2026-10-12 (S5, 09 §3 decision 200): design now (one page: options, a pick, the first slice), build in slices later. OPEN — DIRECTION, operator's call |
a decision on scope; nothing here is a bug | Slice 2 (hub): group 3's mails say „we noticed X and are handling it" instead of the raw error | CC (+ operator: two answers) |
| R-435 | Monitoring & notifications | P3 | The snapshot-drop detector is blind to a single-app deletion — the exact shape the box can produce. snapshotDropFraction = 0.5 and snapshotDropFloor = 5 (hub/internal/monitor/offsite.go) require a fall of MORE than half the previous count. demo-hp's baseline is 69 across 9 apps, so ~35 snapshots must go before it speaks; one app's tag is ~9 and is invisible. offbox.go:1388 runs forget --prune grouped by host,tags — a per-tag wipe is precisely the shape a faulty retention or a targeted deletion produces. This is deliberate, not accidental: the constant's own comment argues the insensitivity, and "a detector that cries wolf is switched off within a fortnight" is a lesson this project paid for. So this row is NOT a demand to lower the threshold. It is a demand that the blind spot be written where the operator reads it, because "an unexplained fall is noticed within a day" (R-431, and STATUS.md) is true only of falls above half. Discovered by arithmetic while planning the drill's Phase 2/3 pairing, which could not have worked: Phase 2 deletes one app and Phase 3 expects the alarm to fire. LIMITATION NOW WRITTEN INTO THE ALARM'S OWN DOCUMENTATION, hub v0.111.1 — the comment above snapshotDropFraction in hub/internal/monitor/offsite.go now states what the detector does NOT see, with the demo-hp arithmetic and the forget --prune grouping that makes the blind spot sit on the most likely single-app failure. It also says explicitly that the numbers must NOT be lowered to "fix" this and that per-app detection needs a SECOND signal keyed on the per-tag count. The row stays OPEN because documenting a blind spot is not covering it — and because STATUS.md and R-431 both still say "noticed within a day", which is true only of falls above half. -- 2026-10-08 design: audits/day-2026-10-08/design-R-435.md — the box's key can no longer delete (decisions 68–69); deletion through any other credential still needs >50 % to alarm. Question D7 on STATUS's decision sheet. -- 2026-10-08 14:16 operator ruling D7 (09 §3 decision 191): yes — option C, an error mail when even one snapshot disappears outside a hub-opened clean-up window (pinned tiers). |
VERIFY — 2026-10-12 read: not provable today. No mail for a snapshot drop reached the operator since 2026-10-11 (Gmail), but the window itself could not be read: no hub page shows offsite_windows, Tester 1's controller log restarted with 0.308.0 at 13:49Z, and the guest-agent path to its VM did not answer. Re-dated after the next weekly clean-up. VERIFY — 2026-10-09: DELIVERED in hub 0.144.0. Live proof NOT run: a window is granted only when the box asks for one, and Tester 1's weekly clean-up is next due ~2026-10-12 (last window 2026-10-05); prove it then (no alarm inside the window; the hub's window row as control). VERIFY — built on hub main 2026-10-08 (D7, decision 191), ships tomorrow; closes after the 9202 proof (a hand-granted window removes N, no alarm; the hub's window row as the control). OPEN — documented in code v0.111.1; the coverage gap itself is unclosed 2026-10-05 (burn-down night): NEEDS A DESIGN. Covering one-app deletions needs per-app snapshot counts from the controller and a per-tag threshold measured on real counts — two repos, a mechanism nobody has measured. |
— | 2026-10-20: read Tester 1's controller log (window opened/closed, count before/after) and the operator mailbox (no drop alarm) the morning after its weekly clean-up | CC |
| R-724 | Monitoring & notifications | P3 | [P3-LOW] The status pages disagree with each other in small ways a household notices. MEASURED 2026-09-29 on a fresh box: Beállítások reads „Mentés ütemezés 02:30 / 03:00" while Biztonsági mentés says 02:30 / 03:30 / 04:15 / 04:30–08:30; Beállítások shows a raw 2026-09-29T19:22:30Z; its „Helyi cím (LAN)" and „Átjáró" read „nem állapítható meg" on the page the household is asked to read out for remote help; the backups overview still says „Következő mentés — 0 órája" (the age of the LAST run, under the word "next" — noted 2026-09-14, never filed); the dashboard shows the backup at 19:33 where the backup pages say 21:33 (R-500, still reproducing). FIXED 2026-09-30 (controller v0.283.0): Beállítások shows the real schedule and a local time; the dashboard's last backup is local time (R-500); „az előző N órája készült" replaces „0 órája". NARROWED — remaining: „Helyi cím (LAN)" / „Átjáró" read through the file-sharing container, so a box without it says „nem állapítható meg" — not a text fix (the read needs another path). |
NARROWED — the LAN/gateway read only; owner: CC 2026-10-06 night: confirmed not a controller-only fix — the only host-network container is samba (infra/samba.go:123) and the agent's local API has no guest-network route; the fix is a new agent route (e.g. the PVE lxc interfaces read) + a controller reader — a two-repo design. |
— | — | CC |
| R-285 | Monitoring & notifications | P4 | A planned, supervised reinstall pages the operator as if the machine had died — there is no notion of expected downtime anywhere. During the 2026-08-09 rehearsal the hub sent, all status: sent to the operator channel: host_stale 08:58 UTC, node_stale 09:00, host_down 09:28 (error), node_down 09:30 (error), host_leaf_changed 09:31, host_recovered 09:31, node_recovered 09:34, offsite_delivery_stuck 09:34 — eight operator mails for work that was deliberate, attended and announced. This is the OPPOSITE gap from the one R-281 filed: the alarms are not missing, they are indiscriminate. host_stale at 30 min and host_down at 60 min (monitor/host_staleness.go:22-23, downAfter = 2 * threshold) cannot distinguish a wiped-on-purpose box from a dead one, and host_leaf_changed firing on a reinstall is correct-but-expected. Note the interaction with the mute used on 2026-08-09 evening: blocking a customer silences everything, so today the only two settings are page me for planned work and tell me nothing at all. What is owed is a middle: a maintenance window, or an operator-set expected-downtime flag, that suppresses staleness and leaf-change while leaving genuine faults audible |
READY (M) — NEW 2026-08-09 | — | The evidence is the operator's mailbox plus events/notification_log for 2026-08-09 |
CC |
Hub & operator — 10 rows (P3 6, P4 4)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| R-975 | Hub & operator | P3 | The hub database keeps every customer's tunnel token, Cloudflare API token and git token unsealed in one column, so anyone with a copy of the database (or its nightly snapshots) gets them. R-879 sealed the other secret columns but not this one. Fix: seal this column the same way. Security review 2026-10-12 (audits/security-review-2026-10-12/); detail held off-git for the operator (/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/). |
OPEN | — | Seal with a migration and a test | CC |
| R-952 | Hub & operator | P3 | Nothing fails when a new hub /api/v1 route is added without deciding who may call it — that is how three routes stayed open with no credential until v0.148.0 (R-951). Fix: a test that reads the API route table from source (not by text search) and fails on any route that is not classified (public with a reason, operator key, host key, customer key); every non-public route gets a request with no credential and must be refused. |
OPEN — filed 2026-10-12 by the security review; not built in v0.148.0 (the session could not write it) | — | Build the test in its own task, with a red-proof | CC |
| R-953 | Hub & operator | P3 | A full ArgoCD sync of the felhom app with prune would DELETE three live Secrets and so break the hub and Umami. Measured 2026-10-12: Secret/gitea-creds (read by Deployment/hub), Secret/umami-config (read by Deployment/umami and Deployment/umami-db) and Secret/healthchecks-config (read by nothing) are live but no longer in git since R-925 de-gitted felhom.secret.yaml; ArgoCD marks all three requiresPruning. Deployment/umami is also OutOfSync (not pruning). Today only the build-deploy skill's „sync only the hub" line prevents it. Fix: mark the three Secrets argocd.argoproj.io/compare-options: IgnoreExtraneous (or move them out of the app's scope), decide whether healthchecks-config is still needed, and bring Deployment/umami back in sync. |
OPEN — filed 2026-10-12 | — | Annotate or move the three Secrets; then a whole-app diff shows nothing to prune | CC |
| R-31 | Hub & operator | P3 | [P2-HIGH] Offsite provisioning is synchronous with no status affordance. Save runs the Hetzner sync in-request, so the request can hit the nginx 504 while succeeding server-side: the operator cannot tell failed from slow, and a retry races the first attempt. MIGRATED FROM ROADMAP.md 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state idea. Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. |
OPEN — migrated from ROADMAP 2026-08-22, rank unchanged Re-ranked 2026-10-03: P2→P3: operator-only; a known workaround (click once, wait, verify) exists. 2026-10-06 night: the race half fixed on felhom.eu main (hub, unreleased): a second Save while the first still provisions is refused with 409 („already running — wait about a minute, then reload; do not save again"); nothing saved, nothing created; per customer, in memory. TestProvision_R31_*, red-proof audits/night-burndown-2026-10-06/hub/R-31-red.txt. LEFT: the async save with a status card (the escrow-card idiom). |
— | Direction: make it async + a status card, reusing the proven awaiting-card/poll idiom (v0.138.0 escrow card). Interim mitigation belongs in R-3 as an operator note: click once, wait, verify — do not re-click. | CC |
| R-882 | Hub & operator | P3 | Longhorn on DooPlex could not grow a volume online: its instance-manager (116 days up) called a host process that no longer existed — nsenter: cannot open /host/proc/196610/ns/mnt on every expansion retry, and an offline growth was blocked by the expansion's own attachment ticket (found 2026-10-05 growing hub-data to 2 Gi). A restart of the instance-manager (operator-approved) fixed it: 77/77 volumes back attached/healthy in 110 s. Why the cached PID went stale was not established (likely a containerd/k3s or iscsid restart after the instance-manager started), so it will recur after the next such restart and stay invisible until a volume needs to grow. audits/hub-db-offsite-2026-10-05/partA/step1-*.txt |
OPEN | — | Find which host process the PID was and whether Longhorn 1.10.x re-resolves it; until then, before growing any volume, check the instance-manager's age against the last k3s/containerd restart | operator |
| R-883 | Hub & operator | P3 | 8 DooPlex workloads run an image by a moving tag (:latest or none), so any pod restart is a silent upgrade. Measured 2026-10-05: the Longhorn restart restarted zipline on ghcr.io/diced/zipline:latest (pull Always), which pulled 4.8.0; 4.8.0 refused its database (cannot safely migrate from prisma to drizzle: expected migration 20260508022000 … was not applied) and crash-looped. Fixed for zipline by pinning 4.7.0 (homelab-manifests 90f60e4, 4c8ec7a; 4.7.0 applied the four missing migrations; a dump from before is kept out-of-band). The other 7 were counted, not named or changed. Checked from source 2026-10-05 (burn-down round 2): homelab-manifests@87dfc29 still has moving tags (grep image lines without a numeric tag): admin-system/toolbox.yaml:12 nicolaka/netshoot:latest (a bare Pod); calibre-system/cwa.yaml:826 calibre-web-automated:dev; outline-system/outline.yaml:270 minio/minio:latest; tandoor-system/recipe-importer.yaml:26 gitea.dooplex.hu/admin/recipe-importer:latest (pull Always :27); adventurelog-system/adventurelog.yaml:100 and :256 adventurelog-backend/frontend:latest (pull Always :101,:257); jarrs-system/jarr- |
OPEN | — | List the 7 (kubectl get deploy,sts -A images without a fixed tag), pin each to the running version, and let Renovate move them |
operator |
| R-264 | Hub & operator | P4 | Twenty-one facts the boxes report that the hub can now decode nowhere, each allowlisted with a reason rather than silently skipped — and for these the reason is "no consumer today, and one is arguably owed". Split out of R-260 on 2026-08-08 so that closing the CLASS (gated) and fixing its sharpest instance (operator_key_configured) could not be mistaken for having decided what the hub should do with the rest. The list, grouped by what a consumer would be for. (a) Guest-network health — guest_net and its seven children (checked_at, has_route, dhclient_alive, heal_succeeded, heals_last_hour, last_heal_at, damped). The R-54 watchdog reports per-guest network state and self-heal counts every cycle and the hub — the component that emails the operator — models none of it. There is a live incident in this project's own record where a killed dhclient took a tunnel down for 1 h 15 m (audits/INCIDENT-guest-dhclient-killed-2026-07-20.md); a recurring-heal signal is exactly what would have surfaced it. This is the strongest candidate of the twenty-one. (b) selfupdate_pending + selfupdate_pending_version — an agent that has flipped its binary and never committed reports pending on every heartbeat so that "the operator sees WHY the version isn't advancing", and no operator can see it. (c) mgmt_plane.healed_recently — bounded: the hub DOES alarm on the privsep_healed_at timestamp beside it, so the recurring-clobber signal is not lost, only this flag. (d) restore_tests.mount_parity + mount_inventory — R-262's subject; the verdict is not lost (a mismatch fails before Pass is set) but the hub cannot tell a full-fidelity pass from a boot-only one. (e) pbs_dr.applied_at. (f) Controller-side: config_hash, reporting_disabled, stacks, storage.migrated_to, backup.last_db_dump, backup.last_integrity_check — the last two are backup-integrity timestamps, which is the "presence is not success" neighbourhood. For each the question is the same and is NOT answered here: is it wanted? If the hub should act on it, model it and name what consults it. If it should not, the honest end is that the emitter stops sending it — a fact emitted forever and consumed nowhere is a future false green waiting for someone to write a check against it. Deliberately not decided in the G-1 session, whose scope was the gate plus the operator-access instance; unilaterally removing emitters would also break the byte-identical cross-repo host-report golden and is a coordinated two-repo change. ⚠ DISPOSITIONS RECORDED 2026-08-13 — and the first thing to say is that they were NOT in this register before today. The rulings were made on 2026-08-12; this row still read READY — owner Viktor and the gate's twenty entries still all said "arguably owed", so a session told to "re-read the dispositions from the register" would have found none. They are written down now, which is the point of writing them down. THE COUNT WAS ALSO WRONG: this row says twenty-one; the gate's allowlist held twenty, measured. Twenty is the number the dispositions below account for, exactly. (1) BUILD A READER — four groups, fourteen facts. (a) guest-network health, (b) the staged-update-pending pair, (d) the restore-test depth pair, (f-part) the two backup-integrity timestamps. (2) NO READER WANTED — five facts, now recorded as not consumed, DELIBERATELY with the ruling and its date in scripts/wire_contract_gate.py, each with its own reason rather than a bare refusal: mgmt_plane.healed_recently (the hub already alarms on the timestamp beside it), pbs_dr.applied_at (pbs_dr.state is the verdict; the timestamp alone is the attempt-read-as-result trap), config_hash (the hub authors the config and knows its own generation), stacks (the app view is built from the purpose-built app_telemetry wire), storage.migrated_to (box-local bookkeeping with no hub-side intent to reconcile against). The emitters are deliberately left alone — removing one is a coordinated two-repo change and breaks the host-report golden; the honest end here is a recorded decision, not a deletion. The gate grew a THIRD entry kind to carry them, because an undecided fact and a decided one must not read alike. (3) reporting_disabled, decided on its own merits: RECLASSIFIED redundant — health.status = "disabled" travels in the same minimal report, is decoded into reports.health_status, and IS rendered. The decision surfaced a real defect the flag would not have fixed: the staleness checker is age-only, so a deliberately-silent box still alarms → R-321. PROGRESS, 2026-08-13: the first reader is BUILT — guest-network health, R-319. Its eight allowlist entries are removed (an allowlisted tag is skipped, so leaving them would have meant the new reader's fields were never checked); the gate's checked-tag count rose 182 → 190 and skipped fell 88 → 80, which is the positive control that the wiring is real. WHERE THE TWENTY NOW STAND: 8 read · 5 deliberately unread · 1 redundant · 6 still owed a reader (selfupdate_pending, selfupdate_pending_version, restore_tests.mount_parity, restore_tests.mount_inventory, backup.last_db_dump, backup.last_integrity_check) — counts measured from the allowlist, not estimated. Only ONE reader was built on purpose: four at once is a design session pretending to be an implementation, and this one now tells us what the other three cost |
READY — operator ruling 2026-10-12 (S7, 09 §3 decision 202): CC owns the six owed readers and builds them when the queue is idle, one group per session, attended hub deploys. OPEN — 6 of 20 still owed a reader; dispositions recorded 2026-08-13, first reader shipped (R-319) — owner Viktor |
— | Last in the 2026-10-12 arc | CC |
| R-451 | Hub & operator | P4 | [P3-LOW] UPDATE ARC SLICE 7 — a fleet sweep: the operator can SEE, and MOVE, how far behind every box is. Slices 1 and 2 make one box's state visible on that box's own pages. The operator has no fleet view, and it is not derivable from what is already reported: the hub's report payload carries container name, state, CPU and memory, and NO image field at all (spike §5, which is why Peti's box could only be recorded UNKNOWN). So this is a hub-side change as well as a controller one. Rank LOW today because the fleet is two enrolled boxes; it rises with the fleet. architecture/09-update-architecture.md §6, §8.4 BOTH SIDES VERIFIED 2026-09-21, and it is cheaper than this row implies. The controller's payload carries no image (internal/report/types.go L98–103) and the hub's Store.SaveReport (hub/internal/store/store.go:965) denormalises only container counts — but the hub stores the raw report JSON whole, so a new controller field lands there the day it is sent. What is missing is the denormalisation and the page, not the transport. Shape in 09 §6.2–6.3; the payload question is 09 §3b Q7. -- RULED 2026-09-23 (09 §3 decision 18): the report carries, per compose service (database included), the installed reference, the catalog reference and the badge state. Built later, when the fleet grows — Q7's recommendation, confirmed. |
DEFERRED (2026-10-03 triage: the row's verdict was finished, but it names open work no other row carries — the fleet sweep (report field, hub denormalisation, fleet page) is ruled and not built — deferred until the fleet grows (09 §6.3)) — RULED — build deferred until the fleet grows; owner: CC |
— | — | CC |
| R-814 | Hub & operator | P4 | PBS-storage-1 (u629193, box 611421) still status=active, 19.9 MB |
VERIFY (2026-10-03 triage: a July watch row with no id; given R-814. WAITING-ON-OPERATOR — no record found that the box was deleted.) — WAITING-ON-OPERATOR | operator console | Delete the box | operator |
| R-844 | Hub & operator | P4 | The household's OS-update line exists only on the hub's customer timeline. 2026-10-04: the box itself has no event surface for agent results (the controller UI shows no timeline), so os_update_applied is a hub customer event (info: recorded, never mailed). Its stored text is the hub's English sentence; the hu/en bundle text (mail.event.os_update_applied) is used only if it is ever mailed. Fix direction: a controller-side line (the controller already polls the agent's local API) when the box gets a household timeline. audits/os-guest-lane-2026-10-04/partG/hub-customer-timeline-demo-hp.txt |
READY — owner: CC 2026-10-05 (burn-down night): NEEDS A DESIGN — a household timeline on the box does not exist yet. | — | — | CC |
Business & legal — 12 rows (P2 4, P3 2, P4 6)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| R-784 | Business & legal | P2 | [P2-MEDIUM] SparkyFitness's licence forbids commercial use: "may not be used, directly or indirectly, in any product, service … intended for … commercial advantage … without prior written permission from the author" — and Felhom is a paid service that offers it in its catalog. READ 2026-10-01 (audits/visitors-2026-10-01/C/C0-license.txt): a custom licence (GitHub: NOASSERTION), the same at the pinned tag v0.17.3 and at main; termination clause 7 ("cease all use"). SparkyFitness was named as wger's replacement for fitness. Needs (operator): (A) hide it from new installs (lifecycle: hidden) until the author gives written permission, and ask; (B) ask first and keep it offered meanwhile; (C) keep it. Recommended A. If nothing is decided it stays offered. UPDATE 2026-10-02 — DECIDED (09 §3 decision 65, option B): SparkyFitness stays offered; the operator asks the author. The request is drafted (NOT sent): audits/licences-2026-10-02/EMAIL-DRAFT-sparkyfitness.md — the author publishes no e-mail; the routes are the project's Discord (private, recommended) or a GitHub Discussion. Trigger: no written permission before the first paying customer → lifecycle: hidden (a STATUS standing item). 2026-10-08: the public apps page lists SparkyFitness (Hungarian and English), badged Korlátozott licenc / Restricted licence, as it lists Tandoor (R-789) — the box offers both today; removing the cards is one small commit if the decision goes that way. |
NARROWED 2026-10-10 — the author AGREED (operator screenshot of the Discord thread, 2026-10-10 16:17): permission to run the official, unmodified images in customers' homes; €1 per paying household with SparkyFitness installed per month, paid quarterly with a household count, starting with the first paying customer who installs it; 90 days' notice from either side. The author needs no formal agreement, only the same three points sent by e-mail from Felhom's official address, to keep as the record. Closes when that e-mail is sent (operator: 2026-10-11). Once closed, SparkyFitness no longer blocks the first paying customer; the per-household fee is a cost to track from the first paying customer on. WAITING-ON-OPERATOR — rank P2-MEDIUM; owner: operator (send the request; the answer) | — | Operator: send the e-mail with the three points from the official address (2026-10-11); then CC closes this row and keeps the e-mail's date as the evidence | operator |
| R-789 | Business & legal | P2 | [P2-MEDIUM] Tandoor's licence is AGPL-3.0 WITH the Commons Clause: it forbids selling "a product or service whose value derives, entirely or substantially, from the functionality of the Software" — fees for hosting or support included. READ 2026-10-02 at 2.6.15 (audits/licences-2026-10-02/TABLE.md). Felhom charges for installing and caring for the household's apps; whether that value comes "substantially" from Tandoor is the question. Needs (operator): keep (the fee is for the box, not Tandoor), hide for new installs, or ask the authors. Nothing changed meanwhile. RULED 2026-10-02 afternoon (09 §3 decision 66): treated like SparkyFitness — stays offered; the operator asks the authors for written permission; without it before the first paying customer Tandoor is hidden (lifecycle: hidden). STATUS "Before the first paying customer". |
WAITING-ON-OPERATOR — rank P2-MEDIUM; owner: operator (the request; trigger: first paying customer) | — | — | operator |
| R-802 | Business & legal | P2 | [P2-MEDIUM] A lawyer reviews the non-OSI licence list before the first paying customer. Operator ruling 2026-10-02 (09 §3 decision 66): Tandoor (R-789), SparkyFitness (R-784), Emby, n8n, Plex (kept — R-790..R-792), the EE/BUSL parts (R-793), redis 7.4 (R-794); the table is audits/licences-2026-10-02/TABLE.md. STATUS "Before the first paying customer". |
WAITING-ON-OPERATOR — rank P2-MEDIUM; owner: operator (trigger: first paying customer) | — | — | operator |
| R-813 | Business & legal | P2 | [P2] The website collects personal data but publishes no privacy notice, no terms and no imprint. CHECKED 2026-10-03 (read-only): website/ holds nine Hungarian pages and one English page; none is an ÁSZF, an adatkezelési tájékoztató or an impresszum, and no page links to one (ASCII-fragment search aszf, adatkezel, impresszum, impressum, privacy over website/; positive control: the same search finds adatkezel in the contact form). The contact form makes the visitor tick a data-processing consent (website/kapcsolat.html:123-128) whose text names no controller, no retention and no rights, and links nowhere. The papers around it — contract, data-processing agreement, billing — are the intention R-809 in ROADMAP.md. 2026-10-08 (website refresh): the English twins are PUBLIC since today (felhom.eu/en/…, operator choice B) — the legal pages are needed in English too, or an English line saying the legal texts are Hungarian, linked from every English page. The English contact form translates the consent text as it is. The refresh CUT the FAQ's „önálló modell" (no row backs it, brief Part A) and LEFT the GDPR answer („nem harmadik félnél") for R-900; the English FAQ carries the same answer. |
NARROWED 2026-10-09 — the closed-test set is PUBLISHED. Live: https://felhom.eu/adatkezeles (website/adatkezeles.html) and https://felhom.eu/feltetelek (website/feltetelek.html), Hungarian only, version „Zárt teszt — 1.0 verzió, hatályos: 2026. október 9.” Text of record: documentation/legal/{adatkezelesi-tajekoztato,feltetelek}-1.0.md, derived from the published HTML so it cannot drift from the page. All 18 pages carry the operator, info@felhom.eu and both links in the footer (counted: 18 found = 18 with both links = 18 structurally valid); the English footers say the legal texts are Hungarian, and site_gates.py NO_TWIN holds the two pages deliberately — an unreviewed English legal text would be worse than an honest pointer. The contact form's consent text was replaced in both languages: the old one claimed „az adatokat harmadik félnek nem adjuk ki” while Resend, Cloudflare and Google carry the message. Operator rulings 2026-10-09: no company yet, so the operator is named as a PRIVATE PERSON (Nagyfenyvesi Viktor) with no postal address and no phone; publish before the lawyer, because the site was collecting data with no notice at all; and keep three retentions honestly open-ended (website statistics, web server logs, contact messages) rather than promise a date nothing enforces. Four load-bearing claims were MEASURED, not copied from a vendor or a README: zero cookies and no local storage (browser check with a positive control, after the tracker fired; no Set-Cookie on any response); the Umami beacon's exact payload (site id, screen, language, title, url, referrer — no visitor identifier); Cloudflare is DNS only (the public A record 37.191.56.193 is not a Cloudflare address); fsn1 = Falkenstein, Germany. Also measured: felhom-ep0-copy-gc.timer is installed and ran successfully (2026-10-09 08:00, exit 0), so the „deleted within 30 days” line is now true where on 2026-10-08 it was written but not switched on. WHAT REMAINS, which is why this is narrowed and not closed: the full ÁSZF and the impresszum are still unpublished (they need company data that does not exist); there is no English translation of either legal text; and no lawyer has reviewed any of it (R-802). Owner stays operator. |
— | (a) DONE 2026-10-09 — the two URLs are in the Meta app's Basic settings and the app is Live (R-915 closed). (b) when the company exists, commission the lawyer's review (R-802) and the full ÁSZF + impresszum. If nothing is done: the closed-test set stands as published and unreviewed | operator |
| R-89 | Business & legal | P4 | Retention as a per-customer commercial policy on the hub | READY (increment 2) | — | Policy object + reconciler → ep0 prune job; keep box tokens write-only | CC |
| R-794 | Business & legal | P4 | [P3-LOW] redis 7.4 (RSALv2 / SSPL, not OSI) runs as a private cache in seven apps: dawarich, docmost, immich, nextcloud, outline, paperless-ngx, romm. READ 2026-10-02 (audits/licences-2026-10-02/TABLE.md). Read as permitted (a private cache only its app uses is not Redis offered as a service — inferred). Valkey (BSD-3) or redis 8 (AGPL option) removes the question. Needs: a ladder step per app to valkey or redis 8, through the harness — no hurry. |
READY — rank P3-LOW; owner: CC Re-ranked 2026-10-03: P3→P4: the row itself says no hurry; usage read as permitted. | — | — | CC |
| R-914 | Business & legal | P4 | Write the Felhom Facebook Page skill from the spike's findings. Spike 2026-10-08 (audits/SPIKE-facebook-page-api-2026-10-08.md): key valid, never expires; the Page (1360018983863273) is reached with CREATE_CONTENT/MODERATE/ANALYZE; a scheduled text post and a scheduled photo were created, read back byte-equal and deleted (removal proven). Probe scripts/facebook/fb_probe.py. Gap to close in the skill: a scheduled photo's publish state was not read (no post_id returned). 2026-10-08 (Page pictures task) — two more for the skill: (1) the removal proof was not a proof: the text post's after-DELETE answer was (#10) „does not exist, cannot be loaded due to missing permission…" — that message also means a permission gap. The skill proves removal by listing the Page's scheduled posts before and after the delete (present, then absent); the operator's Planner view on 2026-10-08 showed nothing on 15 October (a different channel, by eye). (2) pictures by API (READ, developers.facebook.com/docs/graph-api/reference/page/picture and …/reference/page/): POST /{page}/picture needs the MANAGE task, which the robot does not have (measured task list); the cover is POST /{page} cover=<photo id>, „only by the Page Admin or Page Editor with EDIT_PROFILE" + business_management. Neither names pages_manage_metadata. Today the pictures are uploaded by hand (marketing/facebook/README.md). -- 2026-10-09: the first slice exists. scripts/facebook/fb_probe.py gained a schedule-post sub-command (text + link, scheduled only) reusing the probe's token loader, redaction, Bearer call and evidence writer. Guards, each with a test (30 tests; all 30 pass on DooPlex, 2 skip on Windows for want of a tz database): it cannot publish immediately — published is always false and the test enumerates every keyword schedule_form() takes to prove none flips it (RED-PROOFED: the guard was removed and the test seen failing, 'true' != 'false'); the body is read from COPY.md by section, never through a shell; check_when() refuses a time outside Meta's 10-minute..6-month window before the call; budapest_to_epoch() uses the real tz database and refuses rather than guessing a DST offset; list_scheduled() records which route answered and returns None when both are refused, so a caller cannot claim a removal it never saw. What the full skill still needs: the photo path (the spike could not prove a scheduled PHOTO stays hidden — deliberately not reused), the pin, comment moderation, and the insight read-back after a post. |
READY — owner: CC | — | Write the skill (drafts scheduled for operator review by default); read a scheduled photo back through the Page's scheduled-post listing | CC |
| R-917 | Business & legal | P4 | COPY.md §2, the Page's longer description (867 characters), has nowhere to go: Facebook's current Pages experience has no long-description field at all. FOUND 2026-10-08 (Page setup task, audits/facebook-page-setup-2026-10-08/). Looked in four places, all on the live Page as its admin: the Page's „Névjegy" tab (Rövid áttekintés / Személyes adatok / Részletek — only a 255-character „Bemutatkozás" and the pinned category), Business Suite's „Oldal módosítása" dialog (profile picture, cover, Bemutatkozás, category, phone, e-mail, address, website, social links — and nothing else), Facebook settings → „Oldal adatai" (redirects to the same Névjegy tab) and settings → „Oldal beállítása" (name, access, type, history, status, recommendation, messaging, data sharing). MEASURED by Graph with the Page token: description, general_info and bio all read null. Writing description by API would need pages_manage_metadata; the robot key's scopes are read_insights, pages_show_list, business_management, pages_read_engagement, pages_read_user_content, pages_manage_posts, pages_manage_engagement, public_profile, and the task's fences forbid adding a permission. So §2 is written, reviewed and unplaceable. Options for the operator: (a) leave §2 unused and let the 99-character intro plus the website carry it; (b) shorten §2 to ≤ 255 characters and make it the „Bemutatkozás" instead of §1 — but §1 was written for exactly that slot, so this is really „rewrite one of the two"; (c) publish §2 as the Page's first pinned post once the app is Live (R-915), which is where a long text actually gets read; (d) ask Meta support whether the field still exists for this Page type. Recommended (c) — the text reads like a post already. -- SCHEDULED 2026-10-09, option (c) carried out. The text was shortened from §2 into COPY.md §6 (two versions; the operator chose 6.2 Közepes, 638 Unicode characters, Hungarian only, 0 emoji, 0 hashtags). Scheduled through the robot, not published: post 1360018983863273_122096547315511222, due 2026-10-12 19:00 Europe/Budapest (epoch 1791824400 = 17:00 UTC, checked against the tz database). Read back: is_published False, scheduled time sent == read, message hex-equal to COPY.md §6.2 — and the sha256 recomputed OUTSIDE the probe agrees (887514383eff997c). Present in GET /{page}/scheduled_posts, and visible in Planner on H 12 at 19:00 with the link card attached (a different channel from the API). It can still be changed or deleted there. The number in the post was checked, not copied: the apps page carries 57 cards while saying 56, which is DELIBERATE — „6 alkalmazás + 1 beépített”, the 57th being FileBrowser, built into every box; index.html says „56 telepíthető alkalmazás” too. |
WAITING-ON-OPERATOR — 2026-10-12 19:02: the first post IS published (Graph: is_published true, created_time 17:00:07Z, the permalink answers 200 without a login; audits/answers-2026-10-12/r917-post-published.txt). Only the pin is left — the operator's one click. VERIFY -- scheduled on main 2026-10-09, due 2026-10-12 19:00. Not public yet; the operator reviews it in Planner and confirms after it goes out |
the scheduled time passing, then the operator's look | Operator: on the post, „…" → „Kiemelés" (pin); then close | operator |
| R-919 | Business & legal | P3 | On a phone the Facebook Page cuts the left edge of the cover: the „s” of „saját szabályaid” and the „f” of „felhom.eu” are gone. MEASURED 2026-10-08 on the live Page in Chrome DevTools device mode, Pixel 9 (412 × 924, mobile user agent, after a reload so Facebook serves the mobile bundle): audits/facebook-page-setup-2026-10-08/C2-phone-headline-cut-closeup.png. The mobile Page header is 412 × 274 = 1,504:1, so Facebook keeps our cover's full height and shows only the centre 938 px of its 1640 px width (57,2 %) — 351 px cut from each side. marketing/facebook/build.py builds to a safe area of the centre 1028 × 544 (306 px clear of each edge), which is 45 px wider per side than the phone actually shows; the light text in out/cover-c.png runs from x 333 to x 997, and the left crop edge is x 351, so the first 18 px of the text are cut. Covers A and B are built from the same safe area and will have the same edge. Two further facts this measurement establishes: Meta's own help page is wrong about its own rendering — it states the mobile cover is 2,4:1 where the Page header measures 1,504:1 — and the mobile profile circle is far bigger than assumed (172 px, centred, overlapping the bottom 112 px of the 274 px cover, i.e. source x 637–1030 × y 369–624 is hidden). Not re-cropped on Facebook, per the task's fence. -- 2026-10-09, FIXED in the build: marketing/facebook/build.py now takes the phone view from the measurement (PHONE_HDR = (412, 274) -> the centre 938 px), so SAFE is (391, 40)-(1249, 584) and the phone profile circle is the measured CENTRED box (637, 369, 393) rather than a left-anchored one. Every cover is drawn in a derived BAND (408, 48)-(1249, 340), and a cap check holds each headline's capital at >= 4 % of the cover height. The red-proof runs on EVERY build (control_old_window): it draws the headline where the old 640 x 360 assumption put it, x 328, and the check must reject it -- the phone's crop edge is x 351, so 23 px were cut. Against the three covers as committed at a76207945e the new check convicted 3 of 3 (A 23/22 px over the left/right edges, B 63/58 px plus 1017 content pixels under the phone circle, C 63/82 px plus 1778). The profile pictures are untouched -- sha256 identical before and after. -- 2026-10-09 (operator refinements, same day): the profile picture now carries the logo MARK only (the lettering was unreadable at 176 px; the mark grew 69 % -> 76 % of the circle, canvas 932 -> 648), and the covers set the headline the way site.css sets .page-index .hero-text h1 (Bold 700, letter-spacing -0.03em, not ExtraBold 800 untracked) with „felhom.eu” drawn from the logo's OWN lettering instead of typed. No geometry changed; every R-919 check and the red-proof stand. -- 2026-10-09 (second measurement): a phone has TWO views and they disagree. SIGNED IN the sides are cut (confirmed on the operator's REAL phone, Chrome/Android: the window solves to x 351..1298 against the emulator's 351..1289 - the left edge to the pixel). SIGNED OUT the FULL width is shown but the cover is top-anchored and only the top 525 px survives (the bottom 99 px is cut; the file's blue top rule is still visible, which is how the side was established), with a much bigger, higher circle at x 486..1150 from y 232. build.py now carries both views, models the circles as DISCS rather than rectangles running to the bottom, and splits the artwork into a READ layer (must survive every view) and a DECOR layer (may be cropped or covered; the build reports the cost - B 39 %, C 62 %). The two-view geometry convicted all three then-current covers before the redraw (A 908 px under a circle, B 1715, C 1962). Covers redrawn: C is the operator's laptop idea with the dashboard at 640 px (was 370), B's motif grown to match. -- 2026-10-09 (the APP measured): the operator checked the live Page in Facebook Lite and in Chrome on his phone. Solved against the laptop frame, both give a visible window of x 349..1290 / 349..1291 - the LITE APP CROPS EXACTLY LIKE SIGNED-IN MOBILE WEB, and both agree with the emulator's 351..1289. Their circle is at x 645..1021 from y 451, LOWER than the emulator's 369, so that figure was pessimistic rather than wrong. Five views measured; the signed-out one stays the binding constraint. Cover C redrawn to the operator's layout (wordmark 88 px on top, gap, catchphrase) - one line, not his two, because two measured 392 text pixels behind the signed-out circle („saját szabályaid” read „saját szab”) and sizing them to fit drops the capital to the 25 px floor. |
VERIFY -- rebuilt on main 2026-10-09. The app IS now measured and the cover renders correctly there; what is left is the operator's look at the NEW cover in the app after uploading it. |
— | Operator: upload the rebuilt cover-c (and the profile picture if not already), then confirm in the Facebook app that the wordmark and the catchphrase are whole. Close on that word | CC |
| R-920 | Business & legal | P4 | The Messenger „Gyakori kérdések" automation does not exist for this Page, so COPY.md §5 — four questions with answers condensed from gyik.html — has nowhere to go. FOUND 2026-10-09 (Page details task, audits/facebook-page-details-2026-10-09/). SEARCHED, not assumed absent: the create-automation catalogue („Az összes automatizálás") holds exactly THREE templates — Automatikus válasz, Távolléti üzenet, A megválaszolatlan üzenetek azonosítása (B5-no-faq-template-all-three.jpg); the template search for „kérdés" answers „Nincs a keresésnek megfelelő automatizálási sablon." while the POSITIVE CONTROL „üzenet" returns two, so the search works and the term genuinely misses; the existing instant-reply automation carries only channel, message and media — no FAQ and no quick replies; and the business-portfolio settings have no messaging/FAQ entry. The copy is written, sourced line by line to gyik.html and committed as marketing/facebook/COPY.md §5, ready to paste unchanged the day the feature appears. Same shape as R-917 (§2 has nowhere to go), and the same cause: Meta removed a Page field this project had planned copy for. Options for the operator: (a) leave §5 unused until Meta brings the feature back; (b) fold the four answers into the Messenger welcome message (§3) — it holds 500 characters and today uses 120, so one or two would fit, not four; (c) publish them as a pinned FAQ post once the app is Live (R-915); (d) ask Meta support whether the FAQ automation still exists for this Page type. Recommended (a) with (c) later — the welcome message stays short, and the website's own gyik.html already answers these. |
DEFERRED — operator chose (c) on 2026-10-09: park the text and publish it as a post once posting starts. Not a defect, and not waiting on CC | nothing — UNBLOCKED 2026-10-09: R-915 is closed, the app is Live, so posting can start whenever the operator wants | When posting starts: publish COPY.md §5 (the four Messenger FAQ answers) as one later post. If nothing is done: the text stays in COPY.md unused, which the operator has accepted |
operator |
| R-930 | Business & legal | P3 | The site's primary call-to-action fails WCAG AA: white on --blue #0083D8 measures 4.01:1 at 16px/600, against a 4.5:1 floor. MEASURED 2026-10-10 in the browser, on the LIVE home page and on the redesign preview: identical, so this is pre-existing and site-wide, not a regression of the redesign. It is the „Jelentkezem tesztelőnek" button and every .cta-button on every page. .cta-button-secondary is fine (7.17:1). Three fixes, all needing the operator because each touches brand: (a) darken --blue until the ratio clears 4.5 (changes a token used across hub, controller and website); (b) make the label 700 weight at 18.66px+, which moves it into WCAG's „large text" band where 3:1 applies and 4.01 passes, at the cost of a bigger button everywhere; (c) accept it and record the decision. Not changed unilaterally: felhom-web-design §6 puts a new accent colour behind the operator's word. |
WAITING-ON-OPERATOR — filed 2026-10-10 by the home-page redesign session; evidence audits/redesign-2026-10-10/AUDIT.md |
operator's choice between (a), (b), (c) | pick one of the three | operator |
| R-989 | Business & legal | P4 | Operator: read documentation/audits/selfhst-survey-2026-10-12/FELHOM.md §6 and rule on each recommendation. READ-ONLY research 2026-10-12: the selfh.st self-hosting survey 2023–2025 (1,869 / 3,655 / 4,081 respondents, hobby users) against our catalog, features, pitch and hardware. Ten ranked recommendations, none acted on; the top three: a supported-hardware page (none exists — finding, §4), off-site backup as a why reason in the pitch (only 25.2 % of the crowd backs up off-site), Syncthing as the largest catalog gap (25.9 %). Tables: TABLE.md; method and limits: METHOD.md. |
WAITING-ON-OPERATOR — filed 2026-10-12 | the operator's reading | Operator: rule on each of the ten (yes / no / later). If nothing is done: nothing changes; the research ages (the 2026 survey replaces it) | operator |
Process & tooling — 19 rows (P3 6, P4 13)
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|---|---|---|---|---|---|---|---|
| R-987 | Process & tooling | P4 | The 53 catalog app images have never been scanned for known vulnerabilities. Fix: a monthly trivy pass over the images the catalog pins, reported, not gating. | OPEN | — | Add to the monthly re-test | CC |
| R-986 | Process & tooling | P3 | No dependency or secret scanner runs on any push, so a known-vulnerable library or a committed key is found only by a review like this one. Proposed gate: govulncheck (called vulns only), gitleaks on the diff, trivy on our own images; start as WARN with an accepted-hits file, then FAIL. Tools and pinned versions: audits/security-review-2026-10-12/SCANNERS.md. Note: gitleaks' default rules did not find R-908's key. |
OPEN | — | Build the gate, warn-first | CC |
| R-733 | Process & tooling | P3 | [P3-LOW] The test bench has NO swap and the boxes have 512 MiB — so a box proof can pass on swap where the bench fails, and nobody records whether a customer guest has swap. MEASURED 2026-09-30 (R-732): immich's first start was OOM-killed 61–104 times on the bench (swap 0) and passed on 9202 by swapping ~108 MB; the bench given 512 MiB swap passed too. demo-hp 9201, 9202 and demo-felhom 9201 all read swap: 512; the golden's guest config is not recorded in its bake evidence, so a customer guest's swap is NOT measured. The harness's memory watch judges anon against the limit and never reads memory.swap.current. Needs: the golden's swap read and recorded; the box walk and the harness report memory.swap.peak beside anon; a decision whether proofs run with swap off (the stricter venue, as R-732's fix was proven). |
READY — rank P3-LOW; owner: CC (harness + golden evidence) 2026-10-06 night: NARROWED — the harness records each container's swap_peak and the venue's swap (catalog branch night-held-2026-10-06, 3f4611c; SwapRecorded red-proved; bench SwapTotal 0 kB, 9202 524288 kB). LEFT: the golden's swap in its bake evidence, the box walk's record, and whether proofs run with swap off. |
— | — | CC |
| R-887 | Process & tooling | P3 | Some CI jobs are never run, and Gitea fails them ~10–13 minutes later with no log. Seen 2026-10-05: felhom.eu job 1361 (commit 1122b5c) and felhom-controller job 1357 (114ff27): every step reads failure, including the first fetch, the log API answers file does not exist, and the runner pod's log has no task line for them (its task ids are job id + 1). A re-run through the API ran the controller job normally (success in 32 s) but the felhom.eu job was again never picked up and failed after ~12 min. The runner pod (gitea-system/act-runner, image felhom-act-runner:0.1.0) had restarted 5 times ~142 min earlier, around the Longhorn instance-manager restart (R-882). Suspected, NOT measured: a stale runner registration claims jobs it never runs — the session's Gitea token cannot list runners (read:admin scope). Consequence: a red CI verdict that is not about the code, and no failure mail (the alarm step never runs either), so only the pull check sees it. CORRECTED 2026-10-05 18:21 (operator's screenshot of Gitea → Site Administration → Runners): ONE runner only — ID 2, felhom-gates-runner, v0.6.1, label felhom-gates, Idle, last online „now". There is no old registration; the stale-registration guess (this row's first text and the reviewer's) was WRONG. RE-DIAGNOSED 2026-10-05 (round 2), from the logs that survive: (1) „lost in a runner restart" does NOT fit — the runner pod last restarted 13:24:42Z (restartCount 5, all around the 13:20Z Longhorn restart), the lost attempts started 1.5–2.5 h later. (2) FOUR attempts were lost, not two: controller job 1357 (start 15:05:41Z → failed 15:18:38Z), felhom.eu job 1359 (15:15:37 → 15:28:38 — the previous session blamed that one on the BusyBox fault; the runner never ran it), job 1361 (15:33:21 → 15:43:38) and its API re-run (15:46:48 → 15:58:38). None has a task line in the runner log; every one was failed at a :38-second mark on a 5-minute step, 10–13 min after it was handed out — the shape of Gitea's periodic „zombie task" stop (a task assigned to a runner that never reports is failed after ~10 min; no log exists because none was written). (3) The runner's task ids are NOT job id + 1 (the controller re-run was task 1363). (4) Gitea's own log for the window is gone — the pod log starts 16:16:32Z (rotated), so the assignment side cannot be read. Likely mechanism, NOT proven: the runner's fetch-task request timed out on its side after Gitea had already assigned the task, so the task was orphaned. In that same hour this session polled Gitea's jobs API hard (15 pages every 15 s per wait loop) and Gitea logged „slow" requests — a plausible load cause, and the session's own. Mitigation taken: the session's CI waiter now polls once a minute. Nothing changed on DooPlex. MECHANISM SEEN 2026-10-05 17:15–17:28Z, with Gitea's own log (round 2): catalog run 1368 (4828dc7) — 17:15:14 the job is marked started; 17:15:16 router: slow POST /api/actions/runner.v1.RunnerService/FetchTask for 10.42.0.42 (the runner), elapsed 3192ms, then UpdateRepoRunsNumbers … context canceled and GetActionWorkflow: EOF — the runner abandoned its fetch after Gitea had assigned the task; the runner log has no line for task 1371; 17:28:39 actions/clear_tasks.go:174 stopTasks() [W] Cannot transfer logs of task 1371 — Gitea's zombie-task stop. The load at that minute: an outside crawler (216.73.216.78) walking commit pages and archive/*.tar.gz, and THIS session's CI waiter, whose 15-page job listings took 13–31 s each. An API re-run passed in 7 s. Done in-session: the waiter now asks GET …/actions/runs?head_sha=<sha> once a minute (1 s). Not done (DooPlex, the operator's): the runner's fetch timeout and Gitea's exposure to the crawler. -- 2026-10-08 (seen): felhom-controller job 1549 (05e1292, a REPORT/CONTEXT-only commit) started 08:29:46Z and ended 08:43:17Z with all three steps failure at the same second and no log (logs endpoint HTTP 500); the same tree passed controller_gates.py --fast locally; a re-run (POST …/actions/runs/1549/rerun) completed success. A lost job inside the watch window. SEEN AGAIN 2026-10-10, and this time TWO JOBS IN TWO REPOS DIED AT THE SAME INSTANT — which is new information about the cause: app-catalog-felhom.eu job 1631 (b7f0f7c, started 10:31:54Z, three steps) and felhom.eu job 1632 (970616b7, started 10:34:17Z, seven steps) both have EVERY step failure at 10:45:58Z with 0 s each — the same wall-clock second — after ~14 and ~11.5 minutes in set-up with nothing run, and both logs answer file does not exist. A per-job problem cannot do that; one runner event took both. The SAME TREES passed on the next push (catalog job 1633, 8ea72aa, success in the normal time), and a shallow clone with no sibling — the CI shape — runs all nine applicable catalog gates green by hand. So this is the flake, not the code. That makes the DUE-CHECK's premise false: jobs HAVE been lost since 2026-10-05 16:00Z. THREE jobs were lost on 2026-10-10 — 1631, 1632 and 1634 — and ALL THREE ended at :45:58 past the hour (10:45:58Z, 10:45:58Z, 12:45:58Z) after sitting 10.5–14 minutes in set-up. A job that is simply never picked up would end at an arbitrary second; ending at the same second-of-the-hour twice two hours apart points at a periodic sweep or a fixed-deadline timer, not at the job. Between the lost ones the runner DID work: catalog job 1633 started 12:35:06Z and succeeded normally. Worth knowing for the diagnosis: the runner pod was restarted earlier the same day (audits/release-2026-10-10/ci-runner-restart.txt). Next step that would settle it: read the runner pod's own log around an HH:45:58 and see what it says about the task id (job id + 1 — 1645, 1646, 1648 here), and look for a 10-minute or hourly timer in its config. |
WATCHING — 2026-10-12 12:50Z, my own change broke pushes, fixed at once: the 2 r/s limit refused a hub image push with 503 (DooPlex and the home boxes reach gitea.dooplex.hu via the home public address 37.191.56.193). homelab-manifests d0767fe: that address added to the whitelist; a new Ingress gitea-registry serves /v2 without any limit (boxes anywhere pull in parallel). The rebuild then pushed. The security review noted that /v2 login attempts are not rate-limited — as before 2026-10-12, when nothing was. WATCHING — 2026-10-12 12:17Z: the S21 change is LIVE on DooPlex (operator yes in chat; homelab-manifests ad8618f): runner fetch_timeout 5s → 30s (/config/config.yaml, read in the running process); Gitea Ingress limit-rps 2 (burst 10), LAN not limited. Measured before: the crawler 20 546 requests in 24 h, up to 475/min; first minute after: crawler 7×503, our traffic 0×503 (audits/answers-2026-10-12/s21-runner-and-ratelimit.txt). Closes if no job is lost by 2026-10-19 (DUE-CHECKS). READY — operator ruling 2026-10-12 (S21, 09 §3 decision 216): yes — raise the act-runner fetch timeout and rate-limit the public Gitea pages the crawler walks (DooPlex, each change after the operator's yes in chat). 2026-10-12 dated check read: NOT met — 4 jobs lost 2026-10-10/11 (controller 1639, felhom.eu 1645, 1646, 1672; each re-run once, each passed); re-dated 2026-10-19, measured over the 7 days after the S21 change. OPEN DATED CHECK 2026-10-12 (DUE-CHECKS): if no job was lost since 2026-10-05 16:00Z (no completed job whose runner log has no task line / whose log API answers file does not exist), close. NIGHT WATCH 2026-10-05/06 (burn-down night): 2 jobs lost of ~30 runs — felhom.eu run 1384 (4aa4d837, 21:23→21:33Z, no log) and felhom-controller run 1401 (c67b26be, 00:45→00:58Z, no log); each re-run once through the API and each passed (2 m 05 s, 57 s). The night's waiter made one filtered call a minute. So the 2026-10-12 close condition („no job lost since 2026-10-05 16:00Z") is already NOT met. audits/night-burndown-2026-10-05/r887-lost-jobs.txt. NIGHT WATCH 2026-10-06/07 (second burn-down night): 0 jobs lost — every push checked by its commit (felhom.eu, controller, agent, catalog main and the held branch), each completed success, one filtered call per check. |
— | 2026-10-19: count lost jobs since 2026-10-12 12:17Z; none → close | CC |
| R-902 | Process & tooling | P3 | The contact form's mailer has no source in any repository: the running image cannot be rebuilt or changed. FOUND 2026-10-08 (website refresh, brief Part C5). manifests/contact-mailer.yaml names contact-mailer:latest with imagePullPolicy: Never (imported into k3s by hand); the binary in the pod is /app/contact-mailer dated 2026-02-05, Go module github.com/felhom/contact-mailer (devel). Searched: git log --all of felhom.eu (only the manifest), every repo under /mnt/5_hdd/felhom.eu/git, find /mnt/5_hdd/felhom.eu ~ -maxdepth 5 -iname '*contact-mailer*'. READ from the binary (strings, read-only kubectl exec … cat): ONE send per form (sendViaResend) to TO_EMAIL, reply_to = the visitor; nothing is sent back to the visitor; the operator's notification is Hungarian (buildEmailHTML, „Csatolmány … fájl"). The English form needs no mailer change (its subject labels carry „(EN)"). audits/website-refresh-2026-10-08/mailer.md |
OPEN — NARROWED 2026-10-08 (search done, NOT FOUND): owner: operator. A read-only search found no source on DooPlex (55,548 *.go/go.mod/Dockerfile* files, every disk, media/Longhorn/containerd/PBS excluded by name), in the web editor's storage and file history, in all 10 Gitea repositories' history, or in Docker's build cache (oldest record 2026-09-28). The binary's own clues: one file /build/main.go, go1.23.12, no vcs.* stamp; image built 2026-02-05 10:40 by BuildKit. The binary is now KEPT in git (audits/mailer-source-2026-10-08/contact-mailer.bin), so a DooPlex rebuild no longer loses the program. A replacement plan is written (audits/mailer-source-2026-10-08/PLAN-replacement.md); one question left: is the February folder on the operator's Windows workstation? |
— | Operator: look on the workstation; if absent, build the replacement from the plan (attended deploy) | operator |
| R-934 | Process & tooling | P3 | Pull the failed disk ST3000VX010 S/N Z6A07P2G out of DooPlex; follow the checklist. The data was moved off at the first alarm (2026-10-10); the disk is unmounted, its fstab line commented out since 2026-08-14, k3s no longer waits for it, and its empty Longhorn disk entry was removed 2026-10-12. Its two SMART alarms (352 unreadable / offline-uncorrectable sectors, flat since 2026-10-09) are silenced by serial until 2026-11-11 14:32 UTC — if the disk is still in the machine then, they come back on purpose. A TWIN of the same model (S/N Z6A070XZ, /mnt/3_hdd) is in use: read the serial before unplugging. Checklist: runbooks/dooplex-remove-disk-Z6A07P2G.md. |
OPEN — NEW 2026-10-12 | the operator (hands, a power-off window) | Follow the checklist; after the first boot remove the exporter line and the silence; tell CC | operator |
| R-935 | Process & tooling | P3 | Something sets core.bare = true on the shared felhom.eu checkout, and then every git command there fails („this operation must be run in a work tree"). SEEN 2026-10-12 evening twice: .git/config changed at 16:29:37 and 16:37:33 CEST, each time with core.bare=true; restored by hand both times (git config core.bare false, a flag only, no file touched). Not caused by the gates: repo_gates.py --fast run plainly and with GIT_DIR=.git (as a hook gets it) left the flag false; a scratch git init --bare <path> under GIT_DIR does not flip the parent either. The first flip came while this session did not touch the repo; another session pushed to main at 16:44 from elsewhere. |
OPEN — 2026-10-12 later: across this session's next push the flag stayed false (read before and after); not this session's pushes. OPEN — NEW 2026-10-12 Note 2026-10-12 (security review): at least some of these flips match the security review's pushes from a git worktree of this clone (about 16:29 and 16:37): inside the pre-push hook, scripts/test_gate_decoys.py's poster-facts case runs git init/git config/git commit with the hook's inherited GIT_DIR, so it wrote into the real repository — four test commits in the worktree and user.name d / user.email d@d in the shared .git/config (the review removed that [user] section at about 16:50; commits 6c2c3dd5 and 186f3aa3 carry that author). The review stopped pushing from the worktree. Fix: the test helper must drop GIT_DIR, GIT_WORK_TREE and GIT_INDEX_FILE from its environment. |
finding the writer | Watch .git/config (an inotify or audit rule on the file) during a day with the parallel sessions; name the command; fix it there |
CC |
| R-206 | Process & tooling | P4 | The build-cache cap and the weekly prune exist only as a hand-edited /etc/docker/daemon.json on DooPlex — not in Ansible, so a rebuild loses them. The node_housekeeping role must also carry the prune, which today it is forbidden to run |
NARROWED — 2026-10-12 evening: part (a) is in git (homelab-manifests eba52d7, role docker_daemon, tags never + docker_daemon: a byte-copy of the live daemon.json and a restart-and-verify handler). Not installed — installing restarts the Docker engine on DooPlex (the operator's window). Part (b), narrowing the housekeeping ban to docker builder prune -af, loosens a safety rule: the operator's word. Part (c) is in the role's handler. READY (M) — NEW 2026-08-05 |
— | Operator: install in a window (ansible-playbook site.yml --tags docker_daemon, check first) — and say yes/no to (b) |
operator |
| R-928 | Process & tooling | P4 | Two sessions running repo_gates.py at the same time in the shared felhom.eu clone see each other's DECOY plants. SEEN 2026-10-10 ~15:40: while another session's gate run was in its decoy suite, this session's working tree showed website/assets/crafty-controller-logo.png renamed to crafty-logo.png, website/en/apps.html with a leftover text language switch, then website/en/contact.html and scripts/felhom-host-install.sh modified — each a plant of test_gate_decoys.py, each gone when that run ended. This session's own site_gates.py failed three times in that window on plants, not on its change, and a git add of a broad path at that moment would have committed a plant. A killed run (timeout) leaves its plant in place (its cleanup never runs). Fix direction (a design): decoys plant into a temporary COPY of the tree (as the catalog's volume-persistence decoys already do), or a lock file that a second run waits on. |
OPEN — filed 2026-10-10 (register-shrink session) | — | Design the copy-or-lock; meanwhile stage explicit paths only (already the rule) and re-run a red site gate once another session's run has ended | CC |
| R-394 | Process & tooling | P4 | felhom-build-deploy/SKILL.md is 179 lines, over the 150-line limit its own repo now enforces. Found 2026-08-25 by scripts/check_skills.py on its first run — the over-length was discovered BY the new checker, on the day the limit was written down, which is the checker working as intended. It is not edited and not trimmed here: the task that introduced the limit explicitly scoped the four pre-existing skills out, and trimming a build-and-deploy skill without exercising its commands is how a wrong command ships to a live host. It is a named single-entry exception in GRANDFATHERED in scripts/check_skills.py, printed as a WARN on every run, so it cannot fade; a NEW skill over the limit is convicted normally, and growing the set requires editing that file in a commit with a row to name. The rationale for the limit — attention thins across the excess, so the lines that matter are not the ones that survive — is in skills/felhom-doc-authoring/SKILL.md §5. |
OPEN — LOW | — | Trim felhom-build-deploy/SKILL.md under 150 lines in a session that can VERIFY the commands it keeps, then delete its GRANDFATHERED entry in the same commit. The likely trim is the per-artifact command blocks moving behind a pointer to the runbooks, keeping the gotchas inline — but that is a judgement for a session with a build to run, not a line-count exercise. |
CC |
| R-421 | Process & tooling | P4 | THE CLASS: an instrument that matches a LABEL rather than the fact it names — five instances, every one found by accident. R-410 (a mkdir turned the release gate green), R-400 (seven debug controls answering nothing), R-378 (a status word inside a sentence), R-419 (a phrase inside prose, including prose saying the marker was ABSENT), R-94 (a test comparing a constant to itself). The gates are the machinery that enforces everything else in this project, and they were the one part nothing had ever checked. The 2026-09-01 decoy sweep read all 29 scripts and fooled 16. Ten were fixed the same day; four remain with rows (R-422..R-425); six could not be given a plausible decoy and are named. The shapes, so the next one is cheap to recognise: (1) name-for-fact — it matches a path or directory NAME while the fact lives inside the file; (2) substring-for-field — it matches a token anywhere in a body instead of in the field that carries it; (3) declaration-for-reachability — it checks a thing is declared, not that it RESOLVES; (4) constant-for-measurement — it compares a value against itself. The single largest cause was mundane: eight gates set their SCOPE with os.listdir (one level), so every one was green and correct today and would have gone blind the moment anyone added a subdirectory. decoy_coverage_gate.py now refuses a new gate that ships without a decoy. |
OPEN — the class row; it stays open as the place the next instance is recorded | — | — | CC |
| R-507 | Process & tooling | P4 | [P3-LOW] The proof-install harness cannot drive the graphical installer, so a release's graphical entry is proven only up to its password screen. MEASURED 2026-09-14 on VM 332 (ISO 1.27.0): qm sendkey 332 tab did not move focus (both password copies landed in one field), mouse_move 1237 772 + mouse_button 1 did not move the cursor or press Next, while alt-n did advance a page. The TUI entry is fully drivable. The gate's "proof install on BOTH menu entries" was met for 1.26.1 (by a person) and not for 1.27.x. Fix shape: measure QEMU input-send-event with absolute coordinates, or a VNC client on DooPlex; until then a release's graphical proof is an operator click-through. |
READY — rank P3-LOW; owner: CC Re-ranked 2026-10-03: P3→P4: release-test tooling; an operator click-through covers it. | — | — | CC |
| R-579 | Process & tooling | P4 | [P3-LOW] Five page shells loaded style.css with NO cache-buster, so a browser holding an older copy kept being served CSS that did not know about the newest UI. FOUND 2026-09-18 by the operator's screenshot of the v0.254.0 language globe: it rendered as a bare, unstyled <details> — a stray triangle and two plain words outside the card — because login.html, claim.html, recovery.html, launcher_shared.html and launcher_share_password.html requested /static/style.css with no ?v=, while layout.html has used ?v={{.Version}} since v0.166.0. .Version was also absent from three of those five data maps. FIXED AND CLOSED in the same session (controller v0.255.0): the parameter on all five, and Version set once in executeTemplateLang so a new shell cannot miss it; TestGlobeOnAnonymousShells now refuses an absent or EMPTY ?v=. The general form, which is the part worth keeping: a template that loads a versioned asset WITHOUT its version is invisible to every test that reads markup — the markup is correct and the browser fetches the wrong file. A gate over "every stylesheet/script link in a template carries ?v=" would catch the class; not built, because 8 first-boot-wizard templates would fail it and R-554 deletes them. |
DEFERRED (2026-10-03 triage: the row's verdict was finished, but it names open work no other row carries — the gate "every stylesheet/script link carries ?v=" was deliberately not built until R-554 deletes the old wizard; nothing else owns it) — CLOSED 2026-09-18 - controller v0.255.0 |
— | — | CC |
| R-652 | Process & tooling | P4 | [P3-LOW] The memory watch counted the kernel's file cache as the app's memory. MEASURED 2026-09-23 night on the bench: nextcloud 34.0.4 read 100 % of its 1 GiB and immich's PostgreSQL 100 %, each with 0 kernel oom_kills — memory.peak includes page cache, which the kernel drops before it kills anything. Under the watch as built (R-635 follow-up) both would be marked memory_tight, and the gate would demand a raised mem_limit — a customer-box capacity figure — for cache. Done the same night (09 §3 decision 22, CC — operator may reverse): the watch samples the app's own memory (anon of memory.stat) every 15 s; the mark and the ladder's memory_peak_pct read it where measured; the cgroup peak stays beside it (memory_cgroup_peak_pct). Open: romm's backfilled entry carries M1's 80.9 % cgroup peak (measured before the anon sample existed) — re-measure it on its next step; and decide whether an app whose anon is low but whose cgroup stays pinned at its limit (cache thrash) should be marked at all. Evidence: audits/night-2026-09-23/apps/nextcloud/bench-1024M/, apps/immich/bench-noanon/. |
READY — P3; owner: CC (catalog harness) Re-ranked 2026-10-03: P3→P4: the main fix is done; what is left is harness tuning, no household meets it. | — | — | CC |
| R-693 | Process & tooling | P4 | [P3-LOW] The memory watch marks a Node app memory_tight at any limit — its heap sizes itself from the limit. Measured 2026-09-25 on the bench (docmost 0.96.0, harness v4): the app's own memory (anon) peaked at 349 MB of 384 MB (90.9 %), then, with the limit raised to 512 MB, at 431 MB of 512 MB (80.4 %) — 0 OOM kills and 0 restarts in both 10-minute watches (~12 000 requests each). So the mark (decision 22's "does not fit the memory") fires for an app that fits, and the gate's remedy (raise the limit) cannot clear it. docmost moved with the limit raised to 512 MB (decision 39). Needs: a basis that tells growth-to-fill from pressure (e.g. kills/restarts plus a GC-pressure signal, or a second watch at a higher limit showing the peak scales), or a per-app memory_scales_with_limit fact. audits/night-2026-09-26/C/bench-run1/, …/bench-run2/ |
OPEN — P3; owner: CC Re-ranked 2026-10-03: P3→P4: a harness judgement problem; no household meets it. 2026-10-06 night: skipped — a design: changing when an update is marked „memory tight" changes what the update ladder records and accepts; the row offers options. | — | — | CC |
| R-739 | Process & tooling | P4 | [P3-LOW] The test bench cannot run wanderer at all: its web server calls the database at the PUBLIC name https://<SUBDOMAIN_DB>.<domain>, which the bench has no name or TLS for. MEASURED 2026-09-30 (more-night-apps brief, Part B): on bench 9401 the catalog template (v0.20.0) came up wanderer-db healthy, wanderer-search healthy, wanderer unhealthy for 12 min, every page 500 („Error 0: Something went wrong"); PUBLIC_POCKETBASE_URL=https://hike-db.gate.invalid. So the harness can only ever answer inconclusive at FROM for wanderer, never about an update. Its step v0.20.0 → v0.21.0 (web + db) and meilisearch v1.36 → v1.54 stay untested; the brief's question — does the search index survive meilisearch's move or is it rebuilt — is NOT measured. Needs: a bench venue that gives the stack the DB name (an extra_hosts + plain-http override in the bench's render only, stated in the verdict), or wanderer proven on the box venue alone with the operator's word. audits/more-night-apps-2026-09-30/B/wanderer-probe.txt -- 2026-09-30 late: the bench CAN run wanderer now — upgrade-test.py BENCH_ENV_OVERRIDES points PUBLIC_POCKETBASE_URL at the database container on the bench only (the only address the web image reads, measured in v0.20.0 and v0.21.0), recorded in every verdict: web 200, sign-up (PUT /api/v1/user) 200, login 200. The meilisearch question, answered: v1.54.2 REFUSES v1.36's database („Your database version (1.36.0) is incompatible”); with MEILI_UPGRADE_DB=true it upgrades in place and all three indexes (actors, lists, trails) are there — so wanderer's step needs that switch in the template first. Not done: a fixture (creating a list answered PocketBase's „Failed to create record” — likely a rule for unverified users), whether a household's record survives in the index, the step itself. audits/night-rulings-2026-09-30/ |
NARROWED — the bench runs it; the step needs a fixture and MEILI_UPGRADE_DB; owner: CC Re-ranked 2026-10-03: P3→P4: test bench coverage; the template switch and fixture are tooling work. |
— | — | CC |
| R-759 | Process & tooling | P4 | [P3-LOW] wger's onboarding record (the checklist pilot) keeps rows open that no other row owns. 2026-10-01, app-catalog-felhom.eu/onboarding/wger.md: 2.5 no backup → remove → restore → read back of wger exists — the box has no per-app backup press outside an Update (R-648) and wger has no newer step to carry one; 3.7 changing the password and adding a family member not measured, and the template has no add_people text; 6.3 no forced-fail undo for wger; 8.2 the app page not read on 9202 this session; 9.1 the runtime volume-persistence gate not re-run (last CLEAN 2026-08-02). The other open rows have their own: 1.5 (R-755), 1.7/2.8 (R-762), 3.4 (R-763), 7.1 (R-764). wger is exempt from the onboarding gate (published before the checklist), so nothing blocks; this row is what keeps the record honest. Needs: the five measured on 9202 — 2.5 and 6.3 ride wger's next ladder step (the update's backing-up phase is the per-app backup). audits/new-app-checklist-2026-10-01/ |
READY — rank P3-LOW; owner: CC (catalog) Re-ranked 2026-10-03: P3→P4: record-keeping for a hidden app; nothing blocks. | — | — | CC |
| R-786 | Process & tooling | P4 | [P3-LOW] SparkyFitness's onboarding record has six open rows (app-catalog-felhom.eu/onboarding/sparkyfitness.md): 0.5 runtime internet (food search providers), 0.7 the phone app's sign-in route through traefik, 1.6 the env names the server reads, 1.7 the entrypoint read, 5.4 a second memory watch at another limit, 8.3 no logo/screenshots on felhom.eu (404). Everything else measured this session (bench + 9202). Needs: each row measured, or n/a with a reason. |
READY — rank P3-LOW; owner: CC (catalog) Re-ranked 2026-10-03: P3→P4: onboarding record completeness; no household meets it directly. | — | — | CC |
| R-807 | Process & tooling | P4 | [P3-LOW] 13 apps stay UNDETERMINED because their seeds write only to the database, so their upload / media / cache / redis volumes stay empty. MEASURED 2026-10-02 (audits/persistence-sweep-2026-10-02/A/sweep/TABLE.md): claper, crafty-controller, dawarich (redis), docmost, gramps-web, immich (ML cache), outline (+redis), sparkyfitness, tandoor, vikunja, wger, wishlist, zipline — plus plex (no seed route: a plex.tv claim token) and wanderer (no fixture; unhealthy under the gate). None is BROKEN: nothing was written outside a preserved folder. Needs: per app, a seed that uploads one file (or the volume named n/a with a reason: a redis cache, an ML model cache). |
READY — rank P3-LOW; owner: CC Re-ranked 2026-10-03: P3→P4: test coverage; nothing was found broken. | — | — | CC |
| item | due (UTC) | what to measure |
|---|---|---|
| R-887 | 2026-10-19 | no CI job lost in the 7 days after the S21 DooPlex change (act-runner fetch timeout + crawler rate limit); detail in the R-887 row |
| R-231 | 2026-10-13 | dooplex-backup.service ran green after the 2026-10-12 install from git (Result=success, ExecMainStatus=0, exit after 2026-10-13 03:00 CEST); detail in the R-231 row |
| R-435 | 2026-10-20 | Tester 1's weekly off-site clean-up ran inside a hub window with no drop alarm (box log + mailbox); detail in the R-435 row |
| R-528 | 2026-11-12 | no failed oom_check / unexplained app_oom since 2026-10-12; detail in the R-528 row |