2026-10-26 afternoon: R-1295 and R-1292 closed (hub 0.172.0 + tenantsync v1.4.0 on ep0; controller 0.345.0, floor 13:27:04Z); R-1297 filed (CI speed, operator: both); STATUS (D53, D54), CONTEXT, the run's report.
gates / gates (push) Successful in 13m30s
gates / gates (push) Successful in 13m30s
This commit is contained in:
1 parent
69443d3c4e
commit
2354b9084c
6 files changed
+99
-8
No files matched your search
@@ -16,6 +16,15 @@
|
||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||
|
||||
|
||||
> **2026-10-26 (afternoon run) — R-1295 fixed at the cause and released; R-1292 released; R-1294 read; R-1285 committed;
|
||||
> R-1291 designed.** R-1295's cause, measured on a throwaway PBS 4.2.5: the PBS proxy caches permissions up to 5 s and an
|
||||
> ACL-only change does not clear it, so under concurrent requests a new token is refused for ~5 s (v1.3.0 11/30 failed,
|
||||
> v1.4.0 0/30). Fixed: felhom-tenantsync v1.4.0 INSTALLED on ep0 (operator yes), hub v0.172.0 RELEASED (retry + operator
|
||||
> mail `pbsdr_provision_failed`); live: a fresh Tester4 install made its first whole-guest copy. Controller v0.345.0
|
||||
> RELEASED (global floor 13:27:04Z): the 50 % rule owns qBittorrent's NORMAL slot (live on Tester 1). R-1294: demo-hp's
|
||||
> plan is honest but can never catch up with the cap (D54). R-1285: agent `b1bc63d`, the pass after the commit, not
|
||||
> released (live needs a box from 0.164.0). R-1291: `08` §10 design (D53). New R-1296, R-1297 (CI speed: operator „both").
|
||||
|
||||
> **2026-10-26 (attended release run) — released: hub 0.171.0, agent 0.166.0 (vouched 10:04:55 CET), controller 0.344.0
|
||||
> (global floor 09:07:47 UTC, MinAgent 0.158.1), catalog `73c165d`; golden 0.344.0 baked and vouched (10:26:23 CET).**
|
||||
> Live on the release builds (`audits/release-monday-2026-10-26/`): R-1287 (the set-aside mails „⚠️ … nothing failed"),
|
||||
|
||||
@@ -0,0 +1,72 @@
|
||||
# REPORT — 2026-10-26 afternoon: R-1295 fixed at the cause, R-1294 read, R-1285 / R-1292, R-1291 designed, releases
|
||||
|
||||
**Plain words first.**
|
||||
1. **R-1295 cause:** PBS's proxy keeps permissions in a 5-second cache, and a new access key's grant does not clear it.
|
||||
When other boxes are talking to ep0 at that moment, the new key is refused for a few seconds, and the old script
|
||||
gave up after one quick retry. **Fixed at the cause on ep0** (the script now waits up to 15 s) **and in the hub**
|
||||
(retries, and **a failure now mails you** — yes). Live: a fresh Tester4 install set up its whole-box off-site copy.
|
||||
2. **Tester 2's reinstall is safe** from this failure: the fix is on ep0 and in the hub, and if anything still fails you
|
||||
get a mail that says what to do.
|
||||
3. **R-1294:** demo-hp's clean-up plan is **honest** (old copies only, every app keeps its copies), but it can never get
|
||||
back under the safety cap by itself. **Your pick D54:** a one-time window with a raised cap (my pick).
|
||||
4. **R-1292 fixed and released; R-1285 fixed, not released** (its live proof needs a box coming from 0.164.0).
|
||||
5. **Released:** hub 0.172.0, controller 0.345.0 (every live box), felhom-tenantsync v1.4.0 on ep0. Not released: agent.
|
||||
|
||||
Safest readings taken alone (said first): R-1292's live test ran on Tester 1, not 9202 (9202 has no tested line, so the
|
||||
rule never acts there); R-1285 has no live test today (a box coming from 0.164.0 needs a signed downgrade chain); the
|
||||
fresh-install proof ran after the hub rc deploy, both on your yes.
|
||||
|
||||
## Each fix, before → after
|
||||
|
||||
- **R-1295 (hub v0.172.0 + tenantsync v1.4.0):** a new access key refused by the PBS cache → rolled back, no off-site copy,
|
||||
only a log line → the script waits up to 15 s (the admin key too); the hub retries 3×, 7 s apart; a final failure mails
|
||||
you („… could not be set up — save the customer config to retry"). Measured 11/30 → 0/30 under load; red-proved.
|
||||
- **R-1292 (controller v0.345.0):** with alternative speed on, the 50 % rule changed the alternative limit or gave up →
|
||||
it sets only the normal limit. Live on Tester 1; red-proved.
|
||||
- **R-1285 (agent `b1bc63d`, not released):** the old agent installed the bundle, so the new policy pass waited a release
|
||||
→ the new agent runs it once after its update commits. Tested in the real order; red-proved.
|
||||
- **R-1294 (read only):** no change; D54.
|
||||
- **R-1291 (design only):** `08` §10; D53.
|
||||
|
||||
## Versions, CI by commit
|
||||
|
||||
| What | Version / digest | Commit → CI |
|
||||
|---|---|---|
|
||||
| hub | 0.172.0, `sha256:7c8d129a…` (rc.1 `f6cb8f57…` for the proof) | `722013da` 2362 ✓, `232f42d2` 2366 ✓; manifests `ee38dee2` 2364 ✓ |
|
||||
| ep0 | felhom-tenantsync v1.4.0, sha256 `751b3c26…` (v1.3.0 kept beside it) | `722013da` |
|
||||
| controller | 0.345.0, `sha256:2dc2c0a5…` (rc.1 `1378e4e2…`) | `2619191` 2365 ✓, `6bfb784` 2368 ✓ |
|
||||
| agent | — (not released) | `b1bc63d` 2363 ✓ |
|
||||
|
||||
Global floor 0.345.0 at 13:27:04Z (MinAgent 0.158.1), read back from the hub DB with its WAL and the hub log; the
|
||||
per-box floors cleared. Every live box reports 0.345.0.
|
||||
|
||||
## Rows
|
||||
|
||||
Before **212**, after **212**; opened **2** (R-1296 the hub drops ep0's refusal reason; R-1297 CI speed — your „both"),
|
||||
closed **2** (R-1295, R-1292). R-1285 VERIFY; R-1294 and R-1291 wait for D54 / D53.
|
||||
|
||||
## Observations
|
||||
|
||||
- FILED: R-1296 — PBS's real refusal reason never reaches the hub log (found only by reading ep0's journal).
|
||||
- FILED: R-1297 — felhom.eu CI ~13 min, one runner; ruled „both".
|
||||
- NOT-A-FINDING: the `backup_tier_skipped` 6 min after the fresh install — before the box applied its descriptor; the hub
|
||||
held the mail by design (box enrolled < 1 h); 15 min later the box took its key and made its first copy.
|
||||
- NOT-A-FINDING: 9202's FileBrowser WARN on the move from 0.343.0 (twice now) — healed in 3 s, log only, every live box is
|
||||
past that move.
|
||||
- NOT-A-FINDING: R-1292 run 1 „FAIL" — my probe's quoting broke its reads (kept as evidence); run 2 with a fail-fast
|
||||
instrument check passed.
|
||||
- My mistake, fixed: removing Tester 1's test record the first time raced a path unit that restarted the controller; the
|
||||
second time I stopped the path unit first.
|
||||
|
||||
## Teardown
|
||||
|
||||
- **Machines:** VM 9373 (the R-1295 fresh install) and LXC 9374 (the throwaway PBS) destroyed with their disks; Tester 1's
|
||||
two throwaway qBittorrents removed through the product, its rule record removed (it had none before), helper files
|
||||
gone; 9202 back on 0.343.0 exactly as found (FileBrowser on `traefik-public`, no `felhom-files` network, the rc image
|
||||
deleted — 9202 is not bound to the hub, so no floor moves it). Nothing of mine runs on demo-hp into Tuesday 10:00.
|
||||
- **Product uninstall of VM 9373: not run** — tried root SSH with the original password (refused: day-0 replaced it with
|
||||
the vaulted one) and DooPlex's key (never installed); the vault read is yours. The VM was destroyed instead.
|
||||
- **Hub:** delete host `Tester4-b5ec39` (STATUS). Its first whole-guest copy stays in Tester4's ep0 namespace (a host
|
||||
delete keeps the tenancy, by design).
|
||||
- **ep0:** only the script swap you approved; v1.3.0 kept as `/usr/local/bin/felhom-tenantsync.v1.3.0`.
|
||||
- **Secrets:** evidence scanned (the sec values + the bind token and pairing code): 0 hits.
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
|
||||
|
||||
**Updated 2026-10-26 (release day).** Every box runs hub 0.171.0, agent 0.166.0 and controller 0.344.0, and new installs start on golden 0.344.0 — all vouched: the torrent line hold, FileBrowser off the app network, a set-aside mails „nothing failed", a signed job lasts at most 24 h. One leftover: on a BYO host the SSH clean-up still comes one release late (R-1285).
|
||||
**Updated 2026-10-26 (afternoon).** A fresh install's whole-box off-site copy no longer fails silently: the cause (PBS's 5-second permission cache) is fixed on ep0, and the hub retries and mails you if it still fails (hub 0.172.0). Every box runs controller 0.345.0: the torrent rule leaves the household's alternative speed alone. Two picks wait for you (D53, D54).
|
||||
|
||||
## What works
|
||||
- **The Plex attack (10-17 → 10-23) did not reach Felhom's secrets** (R-1241).
|
||||
@@ -23,7 +23,7 @@
|
||||
- DooPlex: something sets the shared checkout to „bare" (R-935).
|
||||
|
||||
## What we work on
|
||||
- Built today, not released: a fresh install's off-site set-up waits out PBS's permission cache and mails you if it still fails (R-1295); the BYO SSH clean-up in the same update (R-1285); the 50 % torrent rule leaves the alternative speed alone (R-1292).
|
||||
- Built, not released: a BYO box removes our SSH file in the same update (R-1285; the agent waits for a live BYO proof). CC trims felhom.eu's 13-minute CI run.
|
||||
- Dated checks, run by CC. 2026-10-27: the qBittorrent line test on a seeding box, with DooPlex's qBittorrent as the peer (R-1250, your yes); the weekly clean-up window (R-435; the logs of the last one were lost to restarts). 2026-10-30: D35, website and installer to Cloudflare, starts (R-1228, ruled yes).
|
||||
|
||||
## Waiting on you — decide
|
||||
@@ -50,7 +50,7 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
||||
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14.
|
||||
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
|
||||
|
||||
**2026-10-26.** On the hub's Hosts page, delete `Tester4-b5ec39` (CC's R-1295 test VM, destroyed). If not: the hub alarms that the box went silent.
|
||||
**2026-10-26.** On the hub's Hosts page, delete `Tester4-b5ec39` (CC's test VM, destroyed). If not: the hub alarms that the box went silent. And a second CI runner on DooPlex (R-1297): if not, jobs queue behind felhom.eu's run.
|
||||
**2026-10-23 do list.**
|
||||
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
|
||||
2. On the hub's Hosts page: Discard appliance 49. If not: a dead entry alarms.
|
||||
|
||||
@@ -26,6 +26,13 @@
|
||||
|
||||
---
|
||||
|
||||
## 2026-10-26 (afternoon) — R-1295 fixed at the cause (ep0) and released (hub 0.172.0); R-1292 released (controller 0.345.0) (full text: `git show 69443d3c:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|---|---|---|---|
|
||||
| **R-1295** | **A re-enrolled customer's whole-guest off-site tier (PBS DR) can fail to provision at the fresh install and only the hub log says so.** (P3) | CLOSED 2026-10-26 — cause measured (PBS's proxy caches permissions up to 5 s; an ACL-only change does not clear it; under concurrent requests the new token is refused up to ~5 s and the script's quick retry fell in the window — throwaway PBS 4.2.5 under load: v1.3.0 11/30 failed, v1.4.0 0/30). Fixed in both places: **felhom-tenantsync v1.4.0 installed on ep0 13:09Z** (operator yes; sha `751b3c26…`, read-only ops ok) and **hub v0.172.0 released 13:08Z** (retry 3×, 7 s apart; a final failure mails the operator, `pbsdr_provision_failed`, operator-only). Live on hub rc.1: a fresh Tester4 install provisioned, the box consumed the credential and made its first whole-guest copy. Not seen live: a retry firing or the failure mail (both first tries passed) — pinned by `TestR1295_*`, red-proved. Tester 2's reinstall: safe. | `audits/r1295-pbsdr-selfcheck-2026-10-26/` |
|
||||
| **R-1292** | **qBittorrent's 50 % rule reads and writes the limit of the CURRENT speed mode: with the household's alternative speed on, it sets the alternative limit, and a later switch reads as „the household changed it" — the rule stops for good.** (P4) | CLOSED 2026-10-26 — controller v0.345.0 released (global floor 13:27:04Z; every live box on 0.345.0): the 50 % rule owns the NORMAL slot only. Live on Tester 1 (rc.1): alternative speed ON with the household's 500 KiB/s — the rule set the normal limit (5892096 B/s) and left the alternative; after a switch back and one more tick it was still the box's own. Test red-proved. | `audits/r1292-qb-normal-slot-2026-10-26/` |
|
||||
|
||||
## 2026-10-26 — the attended release run: hub 0.171.0, agent 0.166.0, controller 0.344.0, catalog `73c165d` (full text of the rows: `git show 6efb4760:documentation/backlog/OPEN-ITEMS.md`)
|
||||
|
||||
| Row | What | Closed | Evidence |
|
||||
|
||||
@@ -124,7 +124,7 @@ stopping line that lies.
|
||||
| **R-494** | Install & onboarding | P4 | **NARROWED 2026-09-14 by operator ruling → [P3-LOW] the hub COULD create the tunnel at customer creation, for a domain already on Cloudflare. Not blocking: every customer has their own domain and the operator creates the tunnel per day-0 A.1 (`architecture/01-topology-and-trust.md`).** *Original finding, kept:* **[P1-HIGH] A new customer's dashboard has NO reachable address unless the operator hand-makes a Cloudflare tunnel — the link in the setup-code mail is dead.** MEASURED 2026-09-14 on a fresh install from the public ISO (drill intervention **I1**): the claim mail points at `https://felhom.drill0242.felhom.eu`; that name has **no A and no AAAA** record (`dig @1.1.1.1`, control `felhom.enkisfelhom.hu` resolves); the hub has **no tunnel- or DNS-creation code** (`hub/internal/cloudflare/` holds only geo-rule removal; `cf_tunnel_token` is a pasted, optional form field, `configs.go:1478`) — day-0 runbook A.1 makes it a manual Cloudflare-dashboard step that nothing on the customer-create page asks for; the box's own split-horizon resolver on the appliance LAN IP answered `google.com` but not the dashboard name at 13:27:39Z; the agent applied the record at **13:27:44Z** (`lanresolver: applied split-horizon record … ip=192.168.0.158`, 3 m 46 s after the controller started), so the box CAN answer the name — **but only to a device that uses the box as its DNS server, and no document, screen or mail tells a household to do that**; the router and the installer-offered DNS answer nothing. The page was reachable only at the guest's LAN address with the name forced (`curl --resolve …:443:192.168.0.158`). **A volunteer could not have done that.** **What it needs:** an operator ruling — the hub creates the tunnel and DNS at customer creation, or the product gives a household a LAN address that works with no DNS change. | **READY — rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3→P4: operator-side automation; the operator creates the tunnel by hand per the day-0 runbook.** | — | — | CC |
|
||||
| **R-504** | Install & onboarding | P4 | **[P3-LOW] `iso.felhom.eu` cannot show an index page on its own — its root returns 404, and the download page lives on the website instead.** MEASURED 2026-09-14: `https://iso.felhom.eu/` and `/index.html` → 404; only named objects answer. The host is an R2 bucket behind a custom domain; whether R2 would serve an uploaded `index.html` at `/` was **not measured** (uploading anything to the public bucket is a publication). The ISO v1.27.0 task puts the Hungarian download page at `felhom.eu/letoltes` (published with the ISO, after the operator's yes). **Remaining:** a redirect from `iso.felhom.eu/` to that page needs a Cloudflare rule the session has no credential for. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (Cloudflare rule)** **Re-ranked 2026-10-03: P3→P4: households are sent to the website's download page; the bare address is cosmetic.** | — | — | operator |
|
||||
|
||||
## Apps & catalog — 18 rows (P3 7, P4 11)
|
||||
## Apps & catalog — 17 rows (P3 7, P4 10)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -145,7 +145,6 @@ stopping line that lies.
|
||||
| **R-1215** | Apps & catalog | P4 | **One Immich photo upload hung 10 minutes with no answer and was not stored, while the box was healthy.** Night soak 2026-10-21 (`audits/night-2026-10-21/P1-soak/08-load/proof.log`): cycle 36, `IMG_20120021.jpg` sent 19:38:30 UTC, the uploader gave up at its 600 s read limit; Immich's database holds 302 photos for that user, the other cycles 303 (`P1-soak/09-immich-cycle36.txt`). At that moment memory „full" pressure was 3.9 % and `/api/health` answered in 0.26 s. 1 of about 26 000 uploads that night. Not diagnosed: Immich restarted at 02:16 for the off-site dump, so its log of 19:38 is gone. The Immich phone app retries a failed upload. | WATCHING — **2026-10-22 night:** not reproducible tonight; the Immich server log of 19:38 is gone (Immich restarted at 02:16 for the off-site dump). What the household sees: the upload fails after 10 minutes and the Immich phone app sends it again. | — | Watch the next long upload run with the Immich server log saved; file upstream only with a log | CC |
|
||||
| **R-1267** | Apps & catalog | P4 | **Every recreate of Immich leaves one empty anonymous Docker volume behind: the image declares `VOLUME /data`, the template does not map it.** Found 2026-10-24 (review round 2, while putting 9202 back): 9202 holds 296 empty anonymous volumes; each night chain's recreate of `immich-server` made a new one (12:58, 13:00, 13:01 — the three test runs; the two dangling ones this session made were removed by name, nothing pruned). The photos are NOT affected: Immich picks `/usr/src/app/upload` whenever both folders exist (`detectMediaLocation` in the image's `server/dist/services/storage.service.js:28-39`), and the template binds that folder (`templates/immich/docker-compose.yml:49`) — NOT-A-FINDING for data placement. **Seen again 2026-10-24 night on Tester 1:** privatebin's image declares `/run`, `/tmp` and `/var/lib/nginx/tmp` as volumes — each recreate (the night dump's stop/start) leaves three empty anonymous volumes; the same class as Immich's `/data` (`audits/restore-holes-2026-10-24/P1-tester1/42-anon-volumes.txt`). | OPEN **2026-10-24 (catalog sweep):** the same class for OnlyOffice and RomM (RomM's image volume `/romm` holds only the empty mount points of its binds); after every catalog app had been removed WITH data, the sweep box still held 52 such volumes (PrivateBin 21, OnlyOffice 8, RomM 8, Immich 3, and 12 from 12:50–12:52 before the batch, holding the same nginx folders as PrivateBin's) — remove-with-data does not see them (no compose project label). All small (4–72 KB), no household data (`audits/catalog-sweep-2026-10-24/_box/anon-volumes.txt`). | — | declare the image's VOLUME paths as tmpfs or named volumes in the catalog (Immich, privatebin), or remove the dangling ones a recreate leaves, by name | CC |
|
||||
| **R-1268** | Backup & restore | P3 | **Remove with „remove backups” leaves a restore's undo copy on the drive (`backups/restore-undo/<app>/`).** Seen 2026-10-24 night on Tester 1 (restore-holes Part 1, run 1): MeTube was held for support after a failed crash put-back, then removed with `remove_backups=true` — the 1.5 GB copy stayed (the removal deletes `backups/primary` and the second-drive mirror, not `restore-undo`), and nothing lists it (`audits/restore-holes-2026-10-24/P1-tester1/14-run1-teardown.txt`; removed by hand, CC's own test data). The copy is kept on purpose while a hold exists; after the app is gone it is orphaned. | OPEN | — | Remove with `remove_backups` also deletes the app's `restore-undo` folder (under the same path guard), or lists it as a kept copy | CC |
|
||||
| **R-1292** | Apps & catalog | P4 | **qBittorrent's 50 % rule reads and writes the limit of the CURRENT speed mode: with the household's alternative speed on, it sets the alternative limit, and a later switch reads as „the household changed it" — the rule stops for good.** Measured 2026-10-25 night on qBittorrent 5.2.4 (`transfer/uploadLimit`/`setUploadLimit` follow the mode; `audits/night-2026-10-25/r1261-qbittorrent-api-measured.txt`); found while fixing R-1261 (the hold now names its slot). | **VERIFY** — 2026-10-26: controller `2619191` (unreleased) — the 50 % rule owns the NORMAL slot only (`up_limit` via app/preferences); test with the alternative mode on, red-proved; live on Tester 1 (rc `0.345.0-rc.1`; 9202 has no tested line, so the rule never acts there): `audits/r1292-qb-normal-slot-2026-10-26/`. **Earlier:** OPEN | — | make the rule own the NORMAL slot only (`up_limit` through `app/preferences`, as the hold does) + a test with the alternative mode on (red first); one controller release | CC |
|
||||
|
||||
|
||||
## App updates — 5 rows (P2 1, P3 4)
|
||||
@@ -158,7 +157,7 @@ stopping line that lies.
|
||||
| **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
|
||||
| **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. | **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC |
|
||||
|
||||
## Backup & restore — 52 rows (P2 9, P3 25, P4 18)
|
||||
## Backup & restore — 51 rows (P2 9, P3 24, P4 18)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -213,7 +212,6 @@ stopping line that lies.
|
||||
| **R-1235** | Backup & restore | P3 | **The release copy on ep0 (D33) is proven to load, not yet to restore a registry — and it follows the newest versions, not the ones the fleet needs.** Built 2026-10-23 (`scripts/release-copy/`, `audits/day-2026-10-23/08-release-copy-D33.txt`). (a) `total-loss-of-dooplex.md` „Restore the release copy” steps 2 (skopeo push back, same digest) and 3 (agent packages back) were never walked into a real new registry — only a `docker load` on 9202. (b) The copy keeps the newest 3 image tags and 5 agent versions; it does not ask the hub for the floor and the vouched agent and always include them — a vouch more than 5 releases behind would fall out. (c) The golden (`felhom-golden`, 637 MB for 0.331.0) is not copied: a new install after a DooPlex loss needs it (the brief named images + agent packages only; one golden fits under the 3 GB cap). | OPEN | — | (a) a throwaway registry on demo-hp, push one tag back, compare digests; (b) read `/configuration`'s floor + vouched agent; (c) the operator says whether the golden goes in | CC |
|
||||
| **R-1263** | Backup & restore | P4 | **Small items from review round 2:** a LOCAL restore's put-back mails the operator „its off-site restore FAILED” (`felhom-controller/controller/cmd/controller/main.go:1514-1518`, shared via `restore_unit.go:517-518`); Remove — even „keep my data” — deletes undeclared volumes labelled with the app's project, which no tier holds (`internal/stacks/delete.go:763-764`, R-1117); the capped copy's `--delete` reads the source in the daemon's mount view, the emptiness check in the controller's (`tier2.go:834` vs `:464`); stale comments — `felhom-agent/internal/localapi/backup_cap.go:15` („keep-last”) and `restore_crash_putback.go:32` (names a test file that does not exist); the qBittorrent password goes to whatever answers the DNS name `qbittorrent` (`upwatch/explain.go:94,108-113`); kept-copy Remove has a check-to-remove gap (`tier2_kept.go:92-111`); the release-copy restore check trusts hashes stored on ep0 (`runbooks/total-loss-of-dooplex.md`); rrsync without `-munge`/`-wo` (`audits/day-2026-10-23/08-release-copy-D33.txt:6`). | OPEN | — | fix in each repo's next attended pass | CC |
|
||||
| **R-1294** | Backup & restore | P3 | **demo-hp's weekly off-site clean-up was refused by its own guard on 2026-10-26 (window 16): the plan would remove 209 of 359 snapshots, above the week's cap (179) — nothing was deleted, and each refused week adds to the next plan.** Seen in the operator mailbox during the 2026-10-26 release run (03:17Z, `offsite_prune_guard_refused`, error; details `count_before 359, count_after 359, outcome guard-refused`). The cap refusal is kept on purpose (R-824, R-867 closed the false ones); the open question is whether a store that once exceeds the cap can ever prune again by itself, or refuses every week while it grows. demo-hp is the Tier-0 box with many hand-made backups (sweeps, probes). Earlier cap refusals: demo-hp 2026-10-03 (window 1), demo-felhom 2026-10-05 (window 3). | **WAITING-ON-OPERATOR** — 2026-10-26 (read only, `audits/r1294-offsite-cap-2026-10-26/`): the plan is HONEST (restic's own dry-run: keep 150, remove 209; every removal 7–30 days old; every app keeps ≥ 9 or its last copies; the bulk is one test day, 2026-10-16, 152 snapshots). It can NOT recover by itself: the cap is half the store and grows ~49/week while the plan grows ~84/week. Only demo-hp has this shape (demo-felhom and Tester 1 pruned normally 10-26). The pick is STATUS D54. **Earlier:** OPEN | — | read the guard's cap rule (controller off-site prune) against demo-hp's snapshot list: is the plan honest (old, superseded) or not; if honest, say how such a store gets back under the cap (an attended one-time prune, or a cap that follows the store) — a design question for the operator if it changes what a box may delete | CC |
|
||||
| **R-1295** | Backup & restore | P3 | **A re-enrolled customer's whole-guest off-site tier (PBS DR) can fail to provision at the fresh install and only the hub log says so.** Seen 2026-10-26 on Tester4's fresh install (golden 0.344.0 proof, host `Tester4-eb7371`): 10:31:15 CET `[ERROR] pbsdr auto-provision for Tester4 (WG-registration hook): pbsdr: F-14 auto re-issue for Tester4: tenantsync: endpoint refused: own-namespace list as felhom@pbs!Tester4 failed twice — rolled back (code self_check_failed) — save the customer config to retry`. No operator mail came (mailbox read 09:33Z: only the household's setup-code mail). On a household reinstall this would leave the box without its whole-guest off-site copy until someone reads the log. Not retried here: the retry lists ep0, fenced in this run. Evidence `audits/release-monday-2026-10-26/fresh-install/proof.txt`. | **VERIFY** — 2026-10-26 afternoon: **cause measured** — PBS's proxy caches permissions up to 5 s and an ACL-only change does not clear it; under concurrent requests the new token is refused for up to ~5 s and the script's one quick retry falls in the window (throwaway PBS 4.2.5 under load: v1.3.0 11/30 failed, v1.4.0 0/30; candidates ruled out with evidence, `audits/r1295-pbsdr-selfcheck-2026-10-26/`). Fixed: tenantsync v1.4.0 (`722013da`, waits up to 15 s, the admin token too) — **not installed on ep0** (an ep0 change, the operator's yes); hub v0.172.0 (`722013da`) retries a rollback 3×, 7 s apart, and a final failure mails the operator (`pbsdr_provision_failed`, operator-only). Red-proved. **Live on hub rc.1:** a fresh Tester4 install provisioned on the first try; the box consumed the credential 12:07:25Z and made its first whole-guest copy (1 PBS snapshot). Left: the hub release and the ep0 script install. **Earlier:** OPEN | — | read why the own-namespace list fails right after the re-issue (a token not yet valid on ep0? a namespace left by the deleted host?); make the rollback mail the operator (an event through the allow-list); retry once with the operator present | CC |
|
||||
|
||||
## Storage & devices — 13 rows (P3 4, P4 9)
|
||||
|
||||
@@ -346,7 +344,7 @@ stopping line that lies.
|
||||
| **R-990** | Business & legal | P4 | **Operator: read `documentation/audits/forum-research-2026-10-12/FELHOM.md` §5 and rule on each recommendation.** READ-ONLY research 2026-10-12: three years (#7943–#11092, 2,908 posts, 195 people, 82 beginner threads) of the public PROHARDVER! topic „Home server / házi szerver építése”, tagged and counted by person (`FINDINGS.md`, method and spot-check in `METHOD.md`; no username or post text in git). Ten ranked recommendations, none acted on; the top three: a „which machine works” page (8 of 33 beginners who name their hardware own only a Synology, a router disk or a Raspberry Pi — the installer cannot run there; the same gap is R-989's first recommendation from a different source, and the shipped-page half is R-991), „you do not choose an operating system — we do” (the biggest beginner question, 20 of 82 threads), and „bring your old office PC” as the cheap entry. Catalog gap (≥ 3 people): a DLNA server (7), an ad/DNS blocker (5), a camera recorder (5). | **WAITING-ON-OPERATOR** — filed 2026-10-12 | the operator's reading | Operator: rule on each of the ten (yes / no / later). If nothing is done: nothing changes; the research ages | operator |
|
||||
| **R-1035** | Business & legal | P4 | **Did the „felhő tárhely” page earn anything? Two dated checks.** **Baseline (reported by the operator from Search Console, 2026-10-14, last 3 months):** 1 click, 48 impressions, CTR 2.1 %, average position **73.6**; „felhő tárhely” 19 impressions, „felhőtárhely” 5, „felhő alapú tárhely” 3. CC measured separately that the phrase „felhő tárhely” appeared **0 times** on the home, FAQ and backup pages — Google was showing the home page for those queries by accident. `/sajat-felho-tarhely` + `/en/own-cloud-storage` published **2026-10-14**; full baseline, price sources and the power provenance in `audits/seo-2026-10-14/BASELINE.md`. **What to measure, both times:** impressions and average position for „felhő tárhely” and „saját felhő tárhely”, plus the new page's own impressions, clicks and position. **A null result at +4 weeks means nothing on its own** — a new page on a site at position 73.6 takes weeks to months to move, and „felhő tárhely” alone is held by Microsoft, Google and the big list sites; the page aims at the two longer phrases. That is why there are two checks and not one, and why neither should be read as a verdict on the copy. **There is no Search Console connector in this project, so CC cannot read these numbers** — the operator reads them, CC records and compares. **Blocked until the operator asks for indexing** (Search Console → URL-ellenőrzés → the URL → „Indexelés kérése”): until that happens, or Google finds the page itself, both checks read zero for a reason that is not the page. **This is a slow channel and does not replace talking to people.** | **WATCHING** | operator: request indexing, then supply the Search Console read | 2026-11-11: first read (+4 weeks); 2026-12-09: second (+8 weeks), compared with the first | operator + CC |
|
||||
|
||||
## Process & tooling — 32 rows (P3 7, P4 25)
|
||||
## Process & tooling — 33 rows (P3 8, P4 25)
|
||||
|
||||
| ID | Category | Sev | What | State | Blocked on | Next action | Owner |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
@@ -401,3 +399,4 @@ stopping line that lies.
|
||||
| R-528 | 2026-11-12 | no failed oom_check / unexplained app_oom since 2026-10-12; detail in the R-528 row |
|
||||
| R-1035 | 2026-12-09 | the same three numbers 8 weeks after publishing, compared with the 4-week read; a null at 4 weeks is not a verdict, both reads are needed; detail in the R-1035 row |
|
||||
<!-- DUE-CHECKS-END -->
|
||||
| **R-1297** | Process & tooling | P3 | **CI waits dominate a working day: felhom.eu's CI run takes ~13 min and the ONE runner makes every repo's job wait behind it (up to 10–17 min queued).** Measured 2026-10-26 (operator: „these CI waiting times are getting out of hand"): since 10-25 felhom.eu runs median 793 s, the controller/agent/catalog 57–102 s; queue median 5 s, max 548–1023 s. In run 2366, ~10 of 13 minutes are four suites — `test_gate_decoys.py` 232 s, `test_due_checks_gate.py` 133 s, `test_repo_gates.py` 132 s (it runs the whole gate set a SECOND time), `dooplex-offsite/test_dooplex_offsite.py` 69 s — plus `hub-gofmt` ~110 s (a cold Go cache per run). **Ruled 2026-10-26 by the operator: BOTH** — (A) trim felhom.eu CI: the heavy suites only when their own files change, no duplicate full run, a Go cache for hub-gofmt; (B) a second CI runner on DooPlex. | OPEN | — | (A) CC, a run of its own (a CI scope change, ruled); (B) a DooPlex change — a brief or the homelab session; and CC waits on CI only before a release or at the end of a run | CC + operator |
|
||||
@@ -5,6 +5,10 @@ Moved here byte for byte on 2026-10-14 (night session), newest first, so that `S
|
||||
|
||||
**Updated 2026-10-13 (night): demo-hp, demo-felhom and Tester 1 run agent 0.155.0; demo-felhom runs controller 0.310.0, the other two 0.309.0 until the floor is raised. The open-items list is at 127. Reports: `REPORT-poweroff.md`, `REPORT-plugmeter.md`, `REPORT-usenet-2026-10-12.md`, `REPORT-newapps-2026-10-12.md`, `REPORT-disk-evening-2026-10-12.md`, `REPORT-answers-2026-10-12.md`, `REPORT-m3-2026-10-11.md`, `REPORT-redesign-2026-10-10.md`, `REPORT-testers-2026-10-10.md`, `REPORT-register-shrink-2026-10-10.md`, `REPORT-new-apps-2026-10-10.md`, `REPORT-release-2026-10-10.md`, `REPORT-break-the-circle-2026-10-09.md`, `REPORT-dooplex-survival-2026-10-09.md`, `REPORT-day4-2026-10-09.md`.**
|
||||
|
||||
## Moved from STATUS on 2026-10-26 (afternoon run)
|
||||
|
||||
**Updated 2026-10-26 (release day).** Every box runs hub 0.171.0, agent 0.166.0 and controller 0.344.0, and new installs start on golden 0.344.0 — all vouched: the torrent line hold, FileBrowser off the app network, a set-aside mails „nothing failed", a signed job lasts at most 24 h. One leftover: on a BYO host the SSH clean-up still comes one release late (R-1285).
|
||||
|
||||
## Moved from STATUS on 2026-10-26 (attended release run)
|
||||
|
||||
**Updated 2026-10-26 (morning).** Installer 1.36.1 is live: a wrong passphrase now says so. Overnight, commit only: the torrent line test holds every seeder; FileBrowser is off the app network; a set-aside mails as a set-aside; the four D44 picks are built and proven on a test VM.
|
||||
|
||||
Reference in new issue
Block a user