diff --git a/CONTEXT.md b/CONTEXT.md index 70ce060..47a411c 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,16 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-11 — NAS/CWA RCA (diagnostic, no fixes):** `documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md`. + The nas-media re-add verified legitimately (new host automounts DO propagate into running + guests); a second `pct reboot 9201` at 15:39 severed the trigger → guest-side stub; CWA + deployed correctly to the NAS (app.yaml + binds + data all nas-media) but is DOWN since 15:39 + (docker can't recreate the bind source; perms luckily blocked any system-drive writes; zero data + loss). UI "felhom-usb" = display bug (deploy.html selects by IsDefault, never stored HDD_PATH); + hub device-ID WARN was CORRECT and the only guard that fired; NAS badge masks because it asks + the agent (host ns). 6 prioritized fix tasks in the audit §5; remediation recipe §6 DEFERRED + pending operator GO. Demo left as-found. + - **2026-07-11 — OBSERVABILITY PASS (hub v0.46.0 LIVE Synced/Healthy; agent v0.83.0 + controller v0.116.1 on demo).** Remote pull-only log access to both box components' always-DEBUG capture rings. Hub: `log_bundle_requests`/`log_bundles` store (gzip, newest-3, **72 h TTL** on the 60 s diff --git a/REPORT.md b/REPORT.md index aae454f..e2abf1e 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,55 +2,32 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## Hub v0.46.0 — observability pass: per-box log pulls, bundle custody, TTL + secret gate — 2026-07-11 +## NAS-backed CWA deploy RCA — diagnostic audit (no fixes) — 2026-07-11 -felhom.eu's third of the cross-repo observability task (agent v0.83.0 + controller v0.116.1 shipped -in their repos). Commits: hub `e35b1ae`, manifest bump + conventions runbook `7c038df`. -**Deployed:** `felhom-hub:0.46.0`, ArgoCD `Synced/Healthy`, rollout complete (17:03 CEST). +Deliverable: `documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md` + CONTEXT.md note. No code +changes anywhere; demo left as-found (remediation §6 operator-gated, deferred). -### What shipped +**Root cause chain (all five hypotheses resolved):** +- The 14:16 nas-media re-add verified **legitimately** — a newly created host automount propagates + into *running* guests (proven: the in-guest controller triggered the host autofs every 5 min, + 14:18–15:36). H1's real trigger was a **second** `pct reboot 9201` at 15:39:11: a fresh guest + namespace inherits ext4 submounts but not the autofs trigger → empty root-owned stub since. +- CWA was deployed **correctly** to the NAS (app.yaml `HDD_PATH`, docker binds, and the physical + library — 5.4 MB incl. metadata.db — all on the DooPlex export). H3 FALSE. +- The UI showing "felhom-usb" is a pure display bug (H4): `deploy.html:567` marks `selected` by + `IsDefault` only, never the stored HDD_PATH; with no default flagged, the browser shows the + first option. +- The hub "nem külön meghajtón" WARN was **correct** (H5) — the device-ID check in the controller + namespace was the only guard that fired. The dashboard NAS badge masked the state because it + asks the **agent** (host namespace, genuinely healthy). +- CWA has been DOWN since 15:39 (docker restart fails: `mkdir .../userdata: permission denied` on + the stub) — which also blocked any writes to the system drive. **Zero data loss, zero misplaced.** +- Probe hole (H2) confirmed in code (no fstype/device check, `netprobe.go`) but did not fire here. +- S-D confirmed real: settings.json stores the felhom-usb label CP1250-mangled since 2026-07-08. +- Observability shakedown: capture ring wraps in ~10 min (status-refresh spam) and dies with + container recreation — rings could not answer H3; host journal + durable artifacts did. -- **Store** (`hub/internal/store/logbundle.go` + schema): `log_bundle_requests` (pending intent per - scope+component — controller scope = customer_id / report ACK; agent scope = host_id / heartbeat - envelope) + `log_bundles` (gzip, newest-3, **72 h TTL** purged on the existing 60 s sweep). - `SaveLogBundle` runs the **token-pattern secret gate BEFORE storing**: a hit stores a - `blocked: possible secret` flag row and NO payload (fail-closed, WARN hub-side); `[REDACTED]` - shapes + public checksums pass by design. -- **Channels** (additive both directions): report ACK `controller_log_requested` + ingest of - `controller_log_tail`; heartbeat envelope `log_tail_requested` + ingest of `log_tail`. - Consume-once on arrival. A pre-0.83 agent never fulfills → the request stays visibly `pending` - (S6-tested, harmless). -- **UI** (host detail, English per the hub operator surface): Diagnostics section — "Request - controller logs" / "Request agent logs" (CSRF forms), state rows (`pending` with the honest - latency hint: controller ≤ ~15 min report interval, agent ≈ heartbeat cadence; `available` with - View/Download; `blocked`), 72 h custody note. The hosts "read-only" invariant is amended by test: - these two forms are the ONLY actions. -- **Conventions codified:** `documentation/runbooks/logging-conventions.md` (levels, logs-are- - English, keys-never-values, capture layers, pull posture) + a one-liner in each repo CLAUDE.md. - -### Red-proofs (demonstrated, restored) - -| Scenario | Reverted shape | Failure seen | -|---|---|---| -| S4 secret gate | gate disabled | `secret-shaped token not blocked` | -| S2/consume-once | clear-on-arrival removed | `request survived fulfillment` (store + API tests) | - -S5 TTL: injectable clock — kept at +71 h, purged at +73 h. Keep-newest-3 + byte-cap belts tested. - -### Live evidence / honest latencies - -- Hub 0.46.0 live (ArgoCD Synced/Healthy, image verified, clean startup log). -- Box halves live-proven from 9201: controller ring + agent ring both readable at - `logging.level=info` through the new Debug tabs (see felhom-controller/REPORT.md timestamps). -- **Operator follow-up (HUMAN — hub UI is password-gated, CC cannot log in):** on - `hub.felhom.eu/hosts/` click both request buttons; expect the agent bundle within one - heartbeat (~15 min) and the controller bundle on the next report (≤ ~15 min); download both and - confirm the controller bundle carries the NAS-add phase lines + the `operator log pull served` - INFO line. The full round-trip is already proven by `internal/api/logbundle_ack_test.go`. -- Blocked-secret branch: test-level only by design (no real secret planted live). - -### Open / observations - -- Hub bearer key rotation still DUE (pre-existing; manifests/hub.yaml + screenshot-exposed). -- `log_bundles` UI states `requested → pending` are merged (a request IS pending from the box's - view); `expired` renders as absence after the purge — states documented in the template hint. +**Fix tasks (audit §5, priority order):** 1) agent ReassertNetworkMounts on guest start +(automount-unit restart mechanism now proven); 2) verify in the consuming namespace (probe fstype +check + deploy-time refusal + NAS badge combines controller-ns view); 3) boot-time app-start +failure alerting; 4) H4 select fix; 5) label repair + writer RCA; 6) ring spam exclusion. diff --git a/documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md b/documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md new file mode 100644 index 0000000..d351c50 --- /dev/null +++ b/documentation/audits/AUDIT-nas-cwa-rca-2026-07-11.md @@ -0,0 +1,232 @@ +# AUDIT — NAS-backed CWA deploy: storage mismatch + "not a separate drive" RCA (2026-07-11) + +**Scope:** diagnostic only — no fixes applied, demo left as-found. Demo box felhom-pve +(192.168.0.162), guest 9201, controller v0.116.1, agent v0.83.0, hub v0.46.0. NFS export +`192.168.0.180:/mnt/5_hdd/felhom-data` (DooPlex). All times CEST unless marked UTC. + +--- + +## TL;DR + +The re-add of `nas-media` at 14:16 **worked end-to-end and verified against the real NFS** — the +new host automount propagated live into the running guest (shared→slave carries *new* mounts). CWA +was deployed correctly against the NAS (`app.yaml` `HDD_PATH: /mnt/felhom-drives/nas-media`) and +its library physically lives on the DooPlex export. What broke everything was a **second guest +reboot at 15:39:11** (`pct reboot 9201`, root@pam): the fresh guest namespace received the +`felhom-usb` ext4 submount but **not** the autofs trigger — since then the guest path is an empty, +root-owned local stub. CWA has been **down since 15:39** (Docker could not recreate the bind +source: `mkdir /mnt/felhom-drives/nas-media/userdata: permission denied` — which, silver lining, +also prevented a single byte from landing on the system drive). The UI showing "felhom-usb" is a +pure display bug: the deployed-app storage select marks `selected` by `IsDefault` only, never by +the stored `HDD_PATH`. The hub WARN was correct and was the **only** signal that fired. + +**Blast radius: zero data loss, zero data misplaced.** Library (434 KB metadata.db + 3 authors, +5.4 MB total) safe on the export. CWA down ~4 h, silently. + +--- + +## 1. Timeline (reconstructed from host journal, docker inspect, file mtimes) + +| Time (CEST) | Event | Evidence | +|---|---|---| +| Jun 15 / Jul 6 | Old CWA cycle leftovers on felhom-usb (`userdata/import/calibre`, `backups/primary/calibre-web`) — unrelated to today | mtimes on felhom-usb | +| Jul 11 morning | Q1c supervised test: guest reboot #1, NAS trigger lost, old share later removed | Q1c findings (prior session) | +| 14:16:35 | `nas-media` re-added: agent `mkdir` + automount unit installed, agent verify OK ("netverify: mount verified") | host journal, agent log | +| 14:16:37 | Registered in controller settings (`kind: network`, mapped uid/gid 1000) | settings.json | +| 14:16:50 | **CWA deployed with `HDD_PATH: /mnt/felhom-drives/nas-media`** | app.yaml `deployed_at` | +| 14:18–15:36 | In-guest controller (comm `felhom-controll`, cgroup `/lxc/9201/...docker...`) triggers the **host** automount every ~5 min — proof the trigger was live *inside the guest* and the periodic checks saw the real NFS | host journal ("Got automount request … triggered by … (felhom-controll)") | +| 14:20–14:24 | Calibre library ingested (Pogátsa Zoltán, Robin Cook, Stephen King) onto the NFS | export file mtimes | +| 15:05:47 | CWA container (re)start — the last running instance | docker inspect StartedAt | +| **15:39:11** | **`pct reboot 9201` (root@pam) — reboot #2.** Automount requests from the guest stop dead at 15:36:54 and never resume | host journal UPID `vzreboot` | +| 15:39:15 | CWA exits (128); Docker restart (`unless-stopped`) fails permanently: `error while creating mount source path '/mnt/felhom-drives/nas-media/userdata/media/books': mkdir /mnt/felhom-drives/nas-media/userdata: permission denied` | docker inspect State | +| 17:13:25 | Controller container recreated (cause unrecorded; RestartCount=0 → not a crash loop) | docker inspect | +| ~19:40 | Audit evidence captured (below); marker test + host-retrigger probe run and cleaned up | this session | + +## 2. Evidence + +### 2.1 Mount reality (H1) — guest first, then host + +Guest 9201 (`pct exec`, before any host-side access this session): + +``` +/dev/mapper/pve-root /mnt/felhom-drives ext4 rw,relatime,errors=remount-ro 0 0 +/dev/sdd1 /mnt/felhom-drives/felhom-usb ext4 rw,relatime 0 0 +(no nfs4, no autofs anywhere) + +/mnt/felhom-drives/nas-media: empty dir, drwxr-xr-x nobody:nogroup, +Device: 252,1 (pve-root), Birth == Modify == 2026-07-11 12:16:35 UTC ← never written since creation +df: /dev/mapper/pve-root 98497780 blocks → the SYSTEM drive +``` + +Controller container (`docker exec felhom-controller`): identical — plain dir on pve-root. +This is the namespace the add-time probe and the 5-min health checks run in. + +Host: + +``` +systemd-1 /mnt/felhom-drives/nas-media autofs rw,relatime,...,timeout=60,direct ... ← armed trigger +mnt-felhom\x2ddrives-nas\x2dmedia.automount loaded active waiting +mnt-felhom\x2ddrives-nas\x2dmedia.mount loaded inactive dead ← normal 60 s idle unmount +``` + +Propagation (mountinfo): host `/mnt/felhom-drives` = `shared:179`, autofs child = `shared:221`; +guest bind = `shared:262 master:179` (a proper slave!), and the guest **did** receive the +felhom-usb ext4 submount (`master:51`) — but not the autofs mount. Observed rule (matches Q1c): +the guest-start bind replicates ext4 submounts but **not autofs direct mounts**; new host mounts +created **while the guest runs** do propagate (proven at 14:16). + +Marker test: `RCA-host-marker` created in the export on DooPlex → **NOT visible in the guest**. +Guest-side `touch` into the stub → `permission denied` (root-owned dir, unmapped in the userns) — +the guest path is not just misdirected, it is completely unwritable. Markers cleaned up. + +### 2.2 Where CWA's data actually is (H3 vs H4) + +- `app.yaml` (`/opt/docker/stacks/calibre-web/app.yaml`): **`HDD_PATH: /mnt/felhom-drives/nas-media`**, + locked field, `deployed_at 2026-07-11T12:16:50Z`. +- `docker inspect calibre-web` binds: `/mnt/felhom-drives/nas-media/userdata/media/books + → /calibre-library` and `.../userdata/import/calibre → /cwa-book-ingest`. Config in a named volume. +- DooPlex export: full CWA skeleton + **library** (`userdata/media/books/metadata.db` 434 KB, + 3 author dirs, `.calnotes`), plus `backups/primary/calibre-web` manifest+compose. 5.4 MB total. + `metadata.db-shm/-wal` left behind by the reboot kill (routine SQLite recovery on next open). +- Guest stub: empty (mtime == birth). felhom-usb: only Jun 15 / Jul 6 leftovers from an earlier + CWA cycle; `userdata/media/books` on felhom-usb is empty. + +### 2.3 Observability shakedown (§3.3) + +The controller/agent capture rings could **not** answer H3 after the fact: +- The controller container was recreated at 17:13 (in-memory ring + docker logs of the old + container gone — the bootstrap deploy mechanism `docker run`s a fresh container). +- Even live, the ring wraps in **~10 minutes** on an idle demo box: earliest entry 17:39:30 UTC at + a 17:47 UTC pull; the buffer is dominated by 10-second `[stacks] Status refresh` + periodic + `[agentapi] GET /disks` DEBUG spam. +- Hub-side bundles (72 h custody) requested by the operator during the observability validation + predate this incident's window; only a bundle pulled between 14:16 and 15:39 would hold the add + and deploy flows. Worth a look on hub.felhom.eu, not counted on. + +H3 was answered from durable artifacts instead (app.yaml + docker inspect + export contents). +The add-flow agent-side lines *were* recoverable — from the host **journal**, not the ring. + +### 2.4 Code paths (all cited against v0.116.1 tree) + +- **Probe (H2):** `controller/internal/web/netprobe.go:32-47` (`NetProbeChild`) — write + readback + + delete only; `netprobe_linux.go:21-43` re-execs as uid/gid 1000 **in the controller + container's namespace**. There is **no fstype / statfs-magic / device-identity check** anywhere + in the probe. Orchestrated from `netstorage_job.go:129-138`. +- **Deploy persist (H3):** the dropdown submits `HDD_PATH` as the option `value` (the real path), + saved to app.yaml as a locked field — verified correct on the box; no substitution point found or + needed (live evidence already refutes H3). +- **Deploy view render (H4):** `internal/web/templates/deploy.html:562-571` — the select renders + `{{if .IsDefault}}selected{{end}}` per option and, for a deployed app, just adds `disabled` + (line 563). **The stored `HDD_PATH` is never compared.** Handler side + (`internal/web/handlers.go:330-342`) passes only the schedulable-path list; the app's current + value isn't passed into the select at all. Live page confirms: *neither* option carries + `selected` (no path currently has `is_default` in settings.json), so the browser displays the + first option — felhom-usb. +- **Hub warning (H5):** `internal/monitor/healthcheck.go:299-302` → `system.IsMountPoint` + (`internal/system/mounts_linux.go:19-29`): `Stat_t.Dev` of path vs parent, evaluated in the + controller container. With the stub both are 252:1 → WARN. **Correct positive.** +- **The masking (why nothing else fired):** `internal/web/handlers.go:1383-1436` + (`networkStorageWarnings`) decides the dashboard "NAS unreachable" badge by asking the **agent** + (`ListNetStorage`) — and the agent's *host-namespace* view is genuinely healthy (`idle`). + Nothing evaluates the share from the namespace the apps actually consume it in. + +### 2.5 Propagation-repair probe (§3.5) + +Host-side access re-triggered the autofs fine (nfs4 mounted on the host) → guest still saw the +stub. The NFS mounts as a **child of the host's autofs mount** (peer group `shared:221`), which has +no counterpart in the guest — so re-triggering can never heal a guest that lost the trigger. +What must be re-established is the **autofs trigger mount itself** in the guest namespace. Two +facts from today constrain the fix: +1. A **newly created** host mount under `/mnt/felhom-drives` (`shared:179`) propagates into running + guests' slave binds — proven by the 14:16 re-add healing the then-running guest. +2. Therefore **restarting the automount unit while guests run** (cheap: the NFS mount is + idle-unmounted anyway) re-plants the trigger in every running guest in one shot. This is the + natural `ReassertNetworkMounts` mechanism; it must fire after every guest start (and prudently + verify guest-visibility afterwards rather than assume). + +## 3. Hypothesis verdicts + +| # | Hypothesis | Verdict | +|---|---|---| +| H1 | Guest-side stub | **TRUE, refined.** The re-add itself propagated into the live guest and verified against the real NFS. The stub was created by the *second* reboot (15:39) — a fresh guest namespace does not inherit a pre-existing autofs trigger (Q1c mechanism re-confirmed), while it *does* inherit ext4 submounts. | +| H2 | Probe hole (no fstype check) | **TRUE in code, did not fire here.** The probe verified the real NFS at add time. Today's stub would fail the probe anyway (`not_writable`, root-owned dir) — the hole only bites when a stub happens to be uid-1000-writable. The *operative* gap is broader: no guest/controller-namespace re-verification ever runs after add-time, and the periodic NAS health signal is measured host-side (see H5/masking). | +| H3 | Deploy persisted wrong path | **FALSE.** app.yaml, docker binds and physical data all say nas-media. | +| H4 | Display-resolution bug | **TRUE.** `deploy.html:567` selects by `IsDefault` only; stored `HDD_PATH` never consulted; with no default flagged, the browser shows the first option (felhom-usb). Display-only. | +| H5 | Hub warning correct | **TRUE.** Device-ID check (healthcheck.go:299) correctly detected path-on-system-device from the controller namespace — the only guard that caught the state. Promote its mechanism to add/deploy-time refusal and to the NAS-unreachable badge. | +| S-D | Garbled "TÁ*rhely" label | **REAL, data-level.** settings.json stores the felhom-usb label as `Tárhely (felhom-usb)` ("Tárhely" UTF-8 bytes re-decoded as CP1250) since its registration on **2026-07-08**; nas-media's label (2026-07-11) is clean. Current label-generation code (`settings.go:891-893`) is clean UTF-8 — one-time corruption in whatever wrote the 07-08 registration. Needs data repair + writer RCA. | + +## 4. The four RCA questions + +1. **Guest-side reality / did the re-add restore propagation?** The guest path is an empty, + root-owned local directory on the system device (252:1) with no NFS or autofs anywhere in the + guest. The re-add **did** restore propagation into the then-running guest (14:16→15:36 proven + healthy); the 15:39 `pct reboot` severed it again. Fresh adds heal running guests; nothing + heals a rebooted guest today. +2. **Where is CWA's data / H3 xor H4?** Physically on the DooPlex export (5.4 MB, intact incl. + metadata.db); `app.yaml` says nas-media. **Display bug (H4)**, not a persistence bug. CWA + itself has been down since 15:39 — the operator's "runs and looked like working" was the + pre-reboot state. +3. **Why did the verify pipeline pass?** Legitimately — at add time the mount was real in the + probing namespace. The design gaps: (a) the probe asserts writability, never + network-filesystem identity (statfs magic / device ≠ rootfs-device); (b) verification runs + once, at add time, in a state that a guest reboot silently invalidates; (c) ongoing NAS health + is asked of the agent (host namespace), so the guest-side stub is invisible. The minimal + add-time check that would refuse today's state: statfs the path in the controller namespace and + require the expected network fstype — cheap, and it also converts the post-reboot stub into a + hard, attributable failure whenever any re-verification runs. +4. **Hub warning correct or false positive?** Correct behavior, right namespace, only working + guard. Promote it: a path that fails the device-ID / fstype test must not be deployable-to, and + should flip the NAS badge, not just a generic system WARN. + +**Blast radius:** no data on the system drive (docker's own `mkdir` was blocked by the root-owned +stub — accidental but effective backstop); no data lost; CWA down ~4 h with no alert beyond the hub +storage WARN. **Customer-plausible: fully.** Any NAS-backed app + any guest reboot (agent update +rollout, host reboot, power blip) reproduces this on a real customer box: app down or — worse, if +the stub is ever writable (e.g. a future uid-1000-owned mountpoint dir) — silent writes to the +system drive. This ordering drives the fix list. + +## 5. Prioritized fix tasks (each a separate follow-up; none implemented here) + +1. **HIGH — agent `ReassertNetworkMounts` on guest start** (existing backlog + `FOLLOWUP-nas-automount-guest-reboot-reassert.md`, now enriched): mechanism = restart/re-create + the automount unit after a guest starts (new-mount propagation into running guests is proven); + must cover all net shares, fire on every guest start (not only reboot-the-verb), and + post-verify guest visibility instead of assuming. Host-side re-triggering alone provably + cannot heal a guest. +2. **HIGH — verify in the consuming namespace, continuously:** (a) add-time probe gains a statfs + fstype/device check (H2 fix — a stub must never verify); (b) deploy-time refusal: a network + `HDD_PATH` whose in-namespace fstype is not the expected network fs is not selectable/deployable + (promotes the H5 check); (c) `networkStorageWarnings` must combine the agent's host view with a + controller-namespace check so a guest-side stub raises the NAS badge (today's masking). +3. **MEDIUM — boot-time app-start failure must alert:** CWA sat in `Exited(128)`/restart-failed + for hours with no issue event. The restart error string ("error while creating mount source + path") is exactly the signal to surface (app-issue event → hub), likely reusing the shipped + app-log issue pipeline. +4. **MEDIUM — H4 display fix:** deployed-app storage select must mark `selected` by the stored + `HDD_PATH` (fallback label when the path is no longer in the schedulable list), not by + `IsDefault`. Also check why no storage path currently carries `is_default`. +5. **LOW — S-D label repair + writer RCA:** fix the stored felhom-usb label (one relabel call), + then find/red-proof the 2026-07-08 write path that CP1250-mangled it (drive attach/registration + flow). +6. **LOW — ring usefulness:** exclude/rate-limit the 10-second status-refresh DEBUG spam from the + capture ring (10-minute effective window defeats after-the-fact RCA), and note that bootstrap + container recreation discards both ring and docker logs — the host journal and hub bundles are + the only durable channels. + +## 6. Remediation — DEFERRED (operator-gated, operator not in session) + +Demo left as-found. Proposed recipe when GO is given (no rsync needed — nothing is misplaced): + +1. `systemctl restart 'mnt-felhom\x2ddrives-nas\x2dmedia.automount'` on the host (guest running). +2. Verify in-guest: `pct exec 9201 -- grep nas-media /proc/mounts` after an in-guest `ls` of the + path (expect autofs, then nfs4), and confirm `RCA`-free export content is visible. +3. `pct exec 9201 -- docker start calibre-web`; confirm the library loads (SQLite recovers the + -wal/-shm on open). +4. Optionally re-run the uid-1000 write path from inside the CWA container. + +This doubles as a live pre-test of the fix-1 mechanism. + +--- +*Evidence gathered non-destructively 2026-07-11 ~19:35–19:50 CEST; only artifacts created were the +two `RCA-` markers (removed). Sources: host journal, pct exec, docker inspect, settings.json, +app.yaml, DooPlex export listing, live page fetch, v0.116.1 source.*