Compare commits
151
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
599ea69551 | ||
|
|
085dc93cea | ||
|
|
e57d0c6171 | ||
|
|
a7e86fd273 | ||
|
|
01a44d5626 | ||
|
|
57d46201d6 | ||
|
|
47f5011797 | ||
|
|
e95d083f47 | ||
|
|
329d769d89 | ||
|
|
17820abbe3 | ||
|
|
03a71b68e6 | ||
|
|
d02c0c2cd8 | ||
|
|
2f7e836f68 | ||
|
|
a560e69954 | ||
|
|
4afa8c3cde | ||
|
|
c25b790ce9 | ||
|
|
f196e74e60 | ||
|
|
38baf4b543 | ||
|
|
3cda7e8d6c | ||
|
|
e0e55bf852 | ||
|
|
53107f1b48 | ||
|
|
bd74a613bf | ||
|
|
cea8a90763 | ||
|
|
f1ab634c4b | ||
|
|
a13c69b01a | ||
|
|
be538975fc | ||
|
|
28f3e15738 | ||
|
|
e7d9c46370 | ||
|
|
8cc555be6c | ||
|
|
636022dc31 | ||
|
|
dd3e4137ee | ||
|
|
e4d91043e4 | ||
|
|
c1f3a163ea | ||
|
|
2ccf180b3c | ||
|
|
d0dbc6f1cd | ||
|
|
682015db86 | ||
|
|
3d0c907b56 | ||
|
|
70862fa7b3 | ||
|
|
7c69c726a0 | ||
|
|
650aa0cf99 | ||
|
|
d6352cd8bf | ||
|
|
41f8f67015 | ||
|
|
bf69bb91ac | ||
|
|
19afefab55 | ||
|
|
7799ef5f54 | ||
|
|
7a6b9ff19d | ||
|
|
ecae56559c | ||
|
|
6a6dd94dac | ||
|
|
0abed37f3e | ||
|
|
d8752de9e3 | ||
|
|
8a3df79fda | ||
|
|
1ee6d5a507 | ||
|
|
05595ecff6 | ||
|
|
1be7aa87dc | ||
|
|
3c0a33c1a0 | ||
|
|
8644450db2 | ||
|
|
fe677bae94 | ||
|
|
1c8f9ef924 | ||
|
|
ac95ca4fc3 | ||
|
|
e381172846 | ||
|
|
47e1ef062b | ||
|
|
29e9cd36a6 | ||
|
|
73c91eacce | ||
|
|
a3247d9f44 | ||
|
|
9b4b5fe5e7 | ||
|
|
3d3cd3ad7c | ||
|
|
f5a4995da5 | ||
|
|
503692a1da | ||
|
|
a69ddbde56 | ||
|
|
f633d48bab | ||
|
|
714ee236b2 | ||
|
|
de57c090bf | ||
|
|
1796112508 | ||
|
|
33f3cd7af0 | ||
|
|
d85198b835 | ||
|
|
7610f69e7c | ||
|
|
7c75162a18 | ||
|
|
acd79a78cc | ||
|
|
708a34512e | ||
|
|
bbc3acb817 | ||
|
|
29134c52c2 | ||
|
|
ba06b873b9 | ||
|
|
c6248dbfc5 | ||
|
|
8744eab80d | ||
|
|
3c8c1ded1f | ||
|
|
626275b187 | ||
|
|
ed843da788 | ||
|
|
6ee23a3b8f | ||
|
|
1f62d62915 | ||
|
|
97ae7086c0 | ||
|
|
ae3db89ab3 | ||
|
|
cc0fb962f3 | ||
|
|
f680720ecd | ||
|
|
ffb91d89e7 | ||
|
|
3720bd1f03 | ||
|
|
d625a1437d | ||
|
|
09b5f02fc3 | ||
|
|
99c0f5e310 | ||
|
|
3ca9b71c73 | ||
|
|
4955744462 | ||
|
|
e33ec8210a | ||
|
|
7e57e824a2 | ||
|
|
ecd82abab8 | ||
|
|
8143a2c872 | ||
|
|
2354b9084c | ||
|
|
69443d3c4e | ||
|
|
a90a9745a8 | ||
|
|
232f42d21e | ||
|
|
ee38dee2c9 | ||
|
|
722013da90 | ||
|
|
c69f53055d | ||
|
|
6efb47607f | ||
|
|
7ad7c99684 | ||
|
|
1550b1da5a | ||
|
|
b00d5c6107 | ||
|
|
9258cdd11c | ||
|
|
f63284861e | ||
|
|
68ee667a83 | ||
|
|
b841192670 | ||
|
|
79ee3e3392 | ||
|
|
36ef8dcdec | ||
|
|
ade357a14e | ||
|
|
a86ab64f6c | ||
|
|
1d4abca54a | ||
|
|
9647f1d158 | ||
|
|
67053335d1 | ||
|
|
3add386aa9 | ||
|
|
dcf4f84cb7 | ||
|
|
80f1ab3167 | ||
|
|
876d92367e | ||
|
|
77015997f8 | ||
|
|
44d1904286 | ||
|
|
6018486fcc | ||
|
|
01bf4e83a9 | ||
|
|
1a788f332d | ||
|
|
3363ffa89a | ||
|
|
0003115092 | ||
|
|
20fa831eaf | ||
|
|
a91d0b2421 | ||
|
|
af5c005a4e | ||
|
|
5277d859d5 | ||
|
|
8257ff1cbe | ||
|
|
87dc50743c | ||
|
|
4b9b0af3cb | ||
|
|
2c9152d912 | ||
|
|
52af38df6f | ||
|
|
34979d5d25 | ||
|
|
0bb57a4fee | ||
|
|
2aebf1bde1 | ||
|
|
d306b8c2d7 | ||
|
|
be7e97dc1a |
No files matched your search
@@ -38,7 +38,7 @@ to probe a refusal at all.**
|
|||||||
## If a probe does create something, remove it through the product
|
## If a probe does create something, remove it through the product
|
||||||
|
|
||||||
Not by hand, and not by `docker rm`: stop it, then `POST /api/stacks/<name>/remove` with
|
Not by hand, and not by `docker rm`: stop it, then `POST /api/stacks/<name>/remove` with
|
||||||
`remove_hdd_data` and `remove_backups`, and then **verify** — no container, no `/opt/felhom/stacks/<name>`,
|
`remove_hdd_data` and `remove_backups`, and then **verify** — no container, no `/opt/docker/stacks/<name>/app.yaml` (the folder itself stays — it holds the catalog's template files; path corrected 2026-10-24),
|
||||||
no volume. Say in the report that it happened. A tidy-up nobody is told about is how the next session
|
no volume. Say in the report that it happened. A tidy-up nobody is told about is how the next session
|
||||||
learns nothing.
|
learns nothing.
|
||||||
|
|
||||||
|
|||||||
@@ -6,8 +6,10 @@ paths: ["website/**"]
|
|||||||
|
|
||||||
## Deploy
|
## Deploy
|
||||||
|
|
||||||
The website **auto-deploys via git-sync from `main`** — just push, live in 1–2 min. No ArgoCD sync,
|
**Since 2026-10-30 (D35 / R-1228) the public site is served by Cloudflare Pages** (project `felhom-site`): a push to
|
||||||
no image build. Emergency edits: https://files.felhom.eu.
|
`main` reaches the public site only after `sec run CF_PAGES_TOKEN -- scripts/publish-site-cloudflare.sh` (`--dry-run`
|
||||||
|
first; it uploads `website/` from `main` and `/scripts/` from the installer tag). DooPlex's git-sync copy keeps tracking
|
||||||
|
`main` as the way back (`documentation/audits/release-2026-10-30/p3-cloudflare/PLAN.md` §5). No image build.
|
||||||
|
|
||||||
**Do NOT pin the website to the installer tag.** The sparse-checkout used to cover `/website/` and
|
**Do NOT pin the website to the installer tag.** The sparse-checkout used to cover `/website/` and
|
||||||
`/scripts/` in one sync; pinning that would turn every copy edit into a release. The installer's
|
`/scripts/` in one sync; pinning that would turn every copy edit into a release. The installer's
|
||||||
|
|||||||
@@ -18,8 +18,13 @@
|
|||||||
name: gates
|
name: gates
|
||||||
# R-1129: `workflow_dispatch` is the manual PROOF that the failure mail can be sent. A push run is
|
# R-1129: `workflow_dispatch` is the manual PROOF that the failure mail can be sent. A push run is
|
||||||
# unchanged by it: the proof step below is skipped on a push, and `force_alarm` is then empty.
|
# unchanged by it: the proof step below is skipped on a push, and `force_alarm` is then empty.
|
||||||
|
# R-1297 (A, 2026-10-26): the nightly schedule is the run that still executes the three path-filtered suites when no
|
||||||
|
# push touched their files (scripts/script_tests_gate.py PATH_FILTERED). It sets no file list, so every suite runs —
|
||||||
|
# like a release tag (a tag push) and a manual dispatch. 03:20 UTC: after the 01:40 release copy and the night chain.
|
||||||
on:
|
on:
|
||||||
push:
|
push:
|
||||||
|
schedule:
|
||||||
|
- cron: "20 3 * * *"
|
||||||
workflow_dispatch:
|
workflow_dispatch:
|
||||||
inputs:
|
inputs:
|
||||||
force_alarm:
|
force_alarm:
|
||||||
@@ -149,6 +154,12 @@ jobs:
|
|||||||
if not commits:
|
if not commits:
|
||||||
sys.stderr.write("the payload carries no commits array - unknown\n")
|
sys.stderr.write("the payload carries no commits array - unknown\n")
|
||||||
raise SystemExit(0)
|
raise SystemExit(0)
|
||||||
|
# R-1297: Gitea trims the payload's commits array (the newest few only); a push of more commits than it
|
||||||
|
# lists has files nobody named here. That list is UNKNOWN, never short - so nothing is printed.
|
||||||
|
total = ev.get("total_commits")
|
||||||
|
if isinstance(total, int) and total > len(commits):
|
||||||
|
sys.stderr.write("the payload lists %d of %d commits - the file list is unknown\n" % (len(commits), total))
|
||||||
|
raise SystemExit(0)
|
||||||
seen = []
|
seen = []
|
||||||
for c in commits:
|
for c in commits:
|
||||||
for key in ("added", "modified", "removed"):
|
for key in ("added", "modified", "removed"):
|
||||||
@@ -172,6 +183,18 @@ jobs:
|
|||||||
[ "$SCOPE" = "docs" ] || SCOPE=code
|
[ "$SCOPE" = "docs" ] || SCOPE=code
|
||||||
echo "PUSH_SCOPE=$SCOPE" >> "$GITHUB_ENV"
|
echo "PUSH_SCOPE=$SCOPE" >> "$GITHUB_ENV"
|
||||||
echo "scope: $SCOPE"
|
echo "scope: $SCOPE"
|
||||||
|
# R-1297 (A): the heavy suites read this list and skip themselves when none of their paths changed. ONLY a
|
||||||
|
# branch push with a known list sets it; a tag, the nightly schedule and a dispatch run every suite.
|
||||||
|
case "${GITHUB_EVENT_NAME:-}:${GITHUB_REF:-}" in
|
||||||
|
push:refs/heads/*)
|
||||||
|
if [ -s /tmp/pushed-files.txt ]; then
|
||||||
|
echo "FELHOM_CHANGED_FILES=/tmp/pushed-files.txt" >> "$GITHUB_ENV"
|
||||||
|
echo "suite filter: ON (a branch push with a known file list)"
|
||||||
|
else
|
||||||
|
echo "suite filter: OFF (no usable file list) - every suite runs"
|
||||||
|
fi ;;
|
||||||
|
*) echo "suite filter: OFF (${GITHUB_EVENT_NAME:-?} ${GITHUB_REF:-?}) - every suite runs" ;;
|
||||||
|
esac
|
||||||
|
|
||||||
- name: Run the gate entry point
|
- name: Run the gate entry point
|
||||||
# The ONLY thing CI runs. No go build, no go test, no linting, no deploy — those are either
|
# The ONLY thing CI runs. No go build, no go test, no linting, no deploy — those are either
|
||||||
@@ -188,6 +211,9 @@ jobs:
|
|||||||
# R-986: the security-scan gate fetches the PINNED gitleaks here (sha256-checked); no Go, so govulncheck says
|
# R-986: the security-scan gate fetches the PINNED gitleaks here (sha256-checked); no Go, so govulncheck says
|
||||||
# NOT CHECKED in CI, out loud. Warn-first either way: the scan lists, it never fails this job.
|
# NOT CHECKED in CI, out loud. Warn-first either way: the scan lists, it never fails this job.
|
||||||
FELHOM_SCAN_FETCH: "1"
|
FELHOM_SCAN_FETCH: "1"
|
||||||
|
# R-1327: test_image_runtime.py reads a zstd layer with the PINNED zstandard wheel (sha256-checked); with this
|
||||||
|
# set, a missing decoder FAILS the suite instead of saying NOT CHECKED.
|
||||||
|
FELHOM_WHEEL_FETCH: "1"
|
||||||
run: python3 scripts/repo_gates.py --fast --scope="${PUSH_SCOPE:-code}"
|
run: python3 scripts/repo_gates.py --fast --scope="${PUSH_SCOPE:-code}"
|
||||||
|
|
||||||
- name: Manual proof - alarm key present, optional forced failure (dispatch only, R-1129)
|
- name: Manual proof - alarm key present, optional forced failure (dispatch only, R-1129)
|
||||||
|
|||||||
@@ -0,0 +1,128 @@
|
|||||||
|
# security-watch — every night, read the security notices of everything Felhom ships, and mail the operator
|
||||||
|
# (admin@felhom.eu, never a household) what affects a pinned version (R-1303). The logic, its classes and its mails:
|
||||||
|
# scripts/security_watch.py; the list of pins: scripts/security_watch_inventory.py; why these sources:
|
||||||
|
# documentation/audits/security-watch-2026-10-26/SPIKE.md.
|
||||||
|
#
|
||||||
|
# THE MEMORY is documentation/security-watch/state.json on `main`, committed back by this job with its own Actions
|
||||||
|
# token. Why there and not elsewhere (2026-10-26): the two runners each have their OWN disk (act-runner-data,
|
||||||
|
# act-runner-2-data — a job lands on either), so a runner file would be two memories; Actions cache needs a node
|
||||||
|
# `uses:` step, which this python-and-git runner cannot run; anything on the DooPlex host is a host change. A file in
|
||||||
|
# the repo is one memory, readable by a person, and its history says when each advisory was first mailed.
|
||||||
|
# If the push of the memory fails, the job fails and the alarm below mails — the next night may then mail an advisory
|
||||||
|
# twice, which is the safe direction.
|
||||||
|
#
|
||||||
|
# Manual run (`workflow_dispatch`, mode):
|
||||||
|
# run a normal night, now
|
||||||
|
# seed record every advisory open now as known and mail nothing new (the first run; prints what it WOULD mail)
|
||||||
|
# dry no mail, no memory change — read the result table in the log
|
||||||
|
# test-mail send ONE clearly marked test advisory to the operator; touches nothing else
|
||||||
|
name: security-watch
|
||||||
|
on:
|
||||||
|
schedule:
|
||||||
|
- cron: "40 3 * * *" # 03:40 UTC: after the 03:20 gates run
|
||||||
|
workflow_dispatch:
|
||||||
|
inputs:
|
||||||
|
mode:
|
||||||
|
description: "run | seed | dry | test-mail"
|
||||||
|
default: "dry"
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
watch:
|
||||||
|
runs-on: felhom-gates
|
||||||
|
timeout-minutes: 20
|
||||||
|
permissions:
|
||||||
|
contents: write
|
||||||
|
steps:
|
||||||
|
- name: Fetch main (latest, for the memory) and the two sibling repos
|
||||||
|
run: |
|
||||||
|
echo "WATCH_STARTED_AT=$(date +%s)" >> "$GITHUB_ENV"
|
||||||
|
base=http://gitea.gitea-system.svc.cluster.local:3000/admin
|
||||||
|
git init -q . && git remote add origin $base/felhom.eu.git
|
||||||
|
git fetch -q --depth 1 origin main && git checkout -q -B main FETCH_HEAD
|
||||||
|
echo "felhom.eu main at $(git rev-parse --short=12 HEAD)"
|
||||||
|
for r in felhom-controller app-catalog-felhom.eu; do
|
||||||
|
git init -q ../$r && git -C ../$r remote add origin $base/$r.git
|
||||||
|
git -C ../$r fetch -q --depth 1 origin main && git -C ../$r checkout -q FETCH_HEAD
|
||||||
|
echo "$r at $(git -C ../$r rev-parse --short=12 HEAD)"
|
||||||
|
done
|
||||||
|
|
||||||
|
- name: Run the watch
|
||||||
|
env:
|
||||||
|
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
|
||||||
|
GH_ADVISORY_TOKEN: ${{ secrets.GH_ADVISORY_TOKEN }}
|
||||||
|
MODE: ${{ github.event.inputs.mode }}
|
||||||
|
# R-1327: zstd layers (Home Assistant and others) are read with the PINNED zstandard wheel, sha256-checked
|
||||||
|
# (scripts/image_runtime.py); without it they are NOT CHECKED, out loud.
|
||||||
|
FELHOM_WHEEL_FETCH: "1"
|
||||||
|
run: |
|
||||||
|
set -u
|
||||||
|
mkdir -p documentation/security-watch
|
||||||
|
mode="${MODE:-run}"
|
||||||
|
[ "${GITHUB_EVENT_NAME:-}" = "schedule" ] && mode=run
|
||||||
|
echo "mode: $mode"
|
||||||
|
case "$mode" in
|
||||||
|
test-mail) python3 scripts/security_watch.py --test-mail; exit $? ;;
|
||||||
|
dry) extra="--no-mail" ; state=/tmp/state-dry.json
|
||||||
|
cp documentation/security-watch/state.json "$state" 2>/dev/null || true ;;
|
||||||
|
seed) extra="--seed" ; state=documentation/security-watch/state.json ;;
|
||||||
|
run) extra="" ; state=documentation/security-watch/state.json ;;
|
||||||
|
*) echo "unknown mode $mode"; exit 2 ;;
|
||||||
|
esac
|
||||||
|
python3 scripts/security_watch.py --state="$state" --catalog=../app-catalog-felhom.eu \
|
||||||
|
--controller=../felhom-controller --json=/tmp/watch-results.json $extra
|
||||||
|
|
||||||
|
- name: Commit the memory (only when it changed)
|
||||||
|
if: github.event.inputs.mode != 'dry' && github.event.inputs.mode != 'test-mail'
|
||||||
|
env:
|
||||||
|
PUSH_TOKEN: ${{ secrets.GITEA_TOKEN }}
|
||||||
|
run: |
|
||||||
|
set -u
|
||||||
|
f=documentation/security-watch/state.json
|
||||||
|
if [ ! -f "$f" ]; then
|
||||||
|
echo "no memory file was written - nothing to commit"; exit 0
|
||||||
|
fi
|
||||||
|
if git ls-files --error-unmatch "$f" >/dev/null 2>&1 && git diff --quiet -- "$f"; then
|
||||||
|
echo "memory unchanged - nothing to commit"; exit 0
|
||||||
|
fi
|
||||||
|
cp "$f" /tmp/state.json
|
||||||
|
auth=$(python3 -c 'import base64,os;print(base64.b64encode(("x:"+os.environ["PUSH_TOKEN"]).encode()).decode())')
|
||||||
|
for attempt in 1 2 3; do
|
||||||
|
git fetch -q --depth 1 origin main && git checkout -q -B main FETCH_HEAD
|
||||||
|
mkdir -p "$(dirname "$f")" && cp /tmp/state.json "$f"
|
||||||
|
git add "$f"
|
||||||
|
git -c user.name="felhom security watch" -c user.email=monitoring@felhom.eu \
|
||||||
|
commit -q -m "security-watch: memory of $(date -u +%F) (R-1303). Docs only." || { echo "nothing to commit"; exit 0; }
|
||||||
|
if git -c http.extraHeader="Authorization: Basic $auth" push -q origin HEAD:main; then
|
||||||
|
echo "memory pushed: $(git rev-parse --short=12 HEAD)"; exit 0
|
||||||
|
fi
|
||||||
|
echo "push refused (attempt $attempt) - re-fetching main"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
echo "MEMORY NOT SAVED after 3 attempts"; exit 1
|
||||||
|
|
||||||
|
- name: Alarm on failure
|
||||||
|
if: failure()
|
||||||
|
env:
|
||||||
|
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
|
||||||
|
run: |
|
||||||
|
python3 - <<'PY'
|
||||||
|
import json, os, sys, time, urllib.request
|
||||||
|
key = os.environ.get("RESEND_API_KEY", "")
|
||||||
|
if not key:
|
||||||
|
sys.exit("ALARM FAILED: RESEND_API_KEY is empty")
|
||||||
|
run = os.environ.get("GITHUB_RUN_ID", "?")
|
||||||
|
link = "https://gitea.dooplex.hu/%s/actions/runs/%s" % (os.environ.get("GITHUB_REPOSITORY", "?"), run)
|
||||||
|
started = os.environ.get("WATCH_STARTED_AT", "")
|
||||||
|
elapsed = "%d s" % (int(time.time()) - int(started)) if started.isdigit() else "unknown"
|
||||||
|
print("ALARM-LINK " + link)
|
||||||
|
body = json.dumps({"from": "Felhom CI <monitoring@felhom.eu>", "to": ["admin@felhom.eu"],
|
||||||
|
"subject": "[felhom CI] security watch FAILED",
|
||||||
|
"text": ("The nightly security watch did not finish (R-1303).\n\nRun : %s\nElapsed : %s\n\n"
|
||||||
|
"Until it runs again, silence from the watch does NOT mean that nothing affects us.\n"
|
||||||
|
"The run log names the step: the watch itself, a mail it could not send, or its memory "
|
||||||
|
"it could not save.\n") % (link, elapsed)}).encode()
|
||||||
|
req = urllib.request.Request("https://api.resend.com/emails", data=body, method="POST", headers={
|
||||||
|
"Authorization": "Bearer %s" % key, "Content-Type": "application/json", "User-Agent": "felhom-ci/1.0"})
|
||||||
|
with urllib.request.urlopen(req, timeout=30) as r:
|
||||||
|
print("RESEND-ACCEPTED id=%s" % json.load(r)["id"])
|
||||||
|
PY
|
||||||
@@ -62,8 +62,9 @@ box**.
|
|||||||
runs two git-syncs: the website from `main`, and `/scripts/` from the tag
|
runs two git-syncs: the website from `main`, and `/scripts/` from the tag
|
||||||
`installer-v<SCRIPT_VERSION>`.
|
`installer-v<SCRIPT_VERSION>`.
|
||||||
- **To publish:** cut `installer-v<new SCRIPT_VERSION>`, bump the `--ref` in `webpage.yaml` (**both**
|
- **To publish:** cut `installer-v<new SCRIPT_VERSION>`, bump the `--ref` in `webpage.yaml` (**both**
|
||||||
the sidecar and the init container), commit, sync.
|
the sidecar and the init container), commit, sync — **and** (since 2026-10-30, D35: felhom.eu is served by Cloudflare
|
||||||
- **To roll back:** move the tag back and wait ~30 s. No ArgoCD sync, no deploy — that is the
|
Pages) run `sec run CF_PAGES_TOKEN -- scripts/publish-site-cloudflare.sh`, which uploads the tag the manifest names.
|
||||||
|
- **To roll back:** move the tag back, then run the same script (~1 min; it re-reads the tag). That is the
|
||||||
emergency lever; fix forward afterwards.
|
emergency lever; fix forward afterwards.
|
||||||
- **Do NOT pin the website to the tag**, and the URL never carries a ref — `felhom-bootstrap.sh` and
|
- **Do NOT pin the website to the tag**, and the URL never carries a ref — `felhom-bootstrap.sh` and
|
||||||
the hub's day-0 command follow the tag with no edit.
|
the hub's day-0 command follow the tag with no edit.
|
||||||
|
|||||||
+358
@@ -16,6 +16,364 @@
|
|||||||
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
> and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
|
||||||
|
|
||||||
|
|
||||||
|
> **2026-11-03 (day, attended) — RELEASED: controller v0.350.0 (the global floor since 08:18 CET, MinAgent 0.158.1, `sha256:9ee53908…` from `5c336d82`), agent v0.172.0, golden 0.350.0 (both VOUCHED 10:18:10 CET), hub v0.179.0 (09:05 CET), catalog `f4c5c79`.** Traefik v3.7.17 on every box (R-1354): on 9202 v3.7.16 grew 18.7 → 79.0 MiB after 920 NTLM connections, v3.7.17 stayed near 25 MiB for 25,116. Golden 0.349.0 was never vouched. The fresh install ran the public installer with one added line naming the unvouched pair, because an install refuses any golden but the vouched one (R-297). A reinstall of a customer whose controller has reported is refused until the operator deletes the host, however long one waits (R-1159). R-1302's ERROR is a BYO box's: the appliance path pre-positions the grant. Report: `REPORT-release-2026-11-03.md`; evidence `documentation/audits/release-2026-11-03/`.
|
||||||
|
>
|
||||||
|
> **2026-11-02/03 (night, unattended) — the security watch and the new notices; no release.** (1) **R-1334's fix held:** 0 Docker Hub 429 on five nights. A streak is per SOURCE, so seven pins that can never be read (`gitea.dooplex.hu/admin/*` ×6, `minio/minio`) held `registry:docker.io` and `runtime` open: `security_watch.py` `STATED_UNREADABLE` + private registry = GAP with a reason; `Transient` retried on every source (budget 30); `FAILED-SOURCE` lines; R-1310's rules (`PACKAGE_NOT_OURS`, `WORD_ONLY`, one entry per GHSA, date tags). (2) **Traefik v3.7.16 has GHSA-xv5x (high):** pin `c4b6844`, unreleased (R-1354). (3) **Catalog live `c760fca`:** gitea 28.3.0, uptime-kuma 2.5.9 (crosses 2.5.8's TLS removal on TCP monitors — decided by CC unattended, operator may reverse), grafana 13.2.5, actualbudget 26.11.0, n8n 2.46.4, vaultwarden 1.37.5-alpine. (4) **Commit only:** controller `c4b6844` + `3764445` (R-1347, R-1348, R-1339, R-1351, R-1323), agent `50eeccc` (R-1302), hub `2f7e836f` (R-1352). (5) **Walks:** 8 claims moved to walked (31 → 23 not walked). R-1361: an event sent while the hub is away is lost after 9 s. R-1362, R-1364 filed. Report: `REPORT-security-2026-11-02.md`; evidence `documentation/audits/security-2026-11-02/`.
|
||||||
|
|
||||||
|
> **2026-10-30 (day, attended) — RELEASED: hub v0.178.0 (1 GiB), agent v0.171.0 (VOUCHED 10:47:36 CET); the site on
|
||||||
|
> Cloudflare (D35, switched 10:02:35 UTC).** (1) **R-1344 decided from the measurement** (4 733 probes through the night
|
||||||
|
> chain, slowest 303 ms, no failure): the agent restarts a running-but-unanswering controller at 10 unhealthy sweeps
|
||||||
|
> (alarm at 6) unless the busy file or the brake refuses (`controllersupervisor.go restartUnresponsive`,
|
||||||
|
> `supervisorBrakeAllows`, `recordSupervisorRestart`); the hub mails `controller_unresponsive_restart`
|
||||||
|
> (`monitor/controller_supervisor.go`). Proven on Tester 1 (SIGSTOP → restart in 6.5 min). (2) **R-1346 fixed**: five
|
||||||
|
> whole-table reads on Longhorn with a 256 MiB pod (`store/r1346_speed.go`: four indexes, one INDEXED BY hint, plans
|
||||||
|
> pinned); manifest save in one transaction; `store/timeddriver.go` logs any statement over 1 s (SQL only); memory 1
|
||||||
|
> GiB. The vouch save 38 255 ms → 132 ms. First start built the indexes in 103 s (startup probe allows 180 s; once).
|
||||||
|
> (3) **D35**: `scripts/publish-site-cloudflare.sh` uploads `website/` (main) + `scripts/` (the manifest's installer
|
||||||
|
> tag) to Pages project `felhom-site` (account `9e432dea…`, token `CF_PAGES_TOKEN` in `sec`); `felhom.eu`/`www` are
|
||||||
|
> proxied CNAMEs to `felhom-site.pages.dev`; the contact form posts to `api.felhom.eu` (ingress `contact-mailer-api`,
|
||||||
|
> exact `/api/contact`, limit-rpm 20). DooPlex's webpage copy is the way back until 2026-11-06 (DUE-CHECKS R-1228).
|
||||||
|
> Rows: 228 → 228 (closed R-1344, R-1346; opened R-1352, R-1353). Report: `REPORT-release-2026-10-30.md`.
|
||||||
|
|
||||||
|
> **2026-10-29 (evening, attended) — RELEASED: hub v0.177.0, agent v0.170.0 (VOUCHED 18:57:48 CET), controller v0.349.0
|
||||||
|
> (global floor 19:24 CET, MinAgent 0.158.1). Operator ruling D65 = A (15:50; `09` §3 decision 284, `04` §6.1):**
|
||||||
|
> `reboot_guest <vmid>` joins the AGENT's closed list, no signature (a reboot destroys nothing, `03`). **Built:** (1) the
|
||||||
|
> agent's `opreboot.go` — `pct reboot` of that guest only; refused for a stopped/locked/parked/unprovisioned guest, a held
|
||||||
|
> heavy-operation gate (held until the guest is up), `jobBlocker`, a fresh controller busy file (one new read-only sudo
|
||||||
|
> vector, fail closed but „no file"), a second reboot within the hour (persisted), the restart brake (a reboot counts);
|
||||||
|
> the watchdogs leave the guest alone meanwhile; the result names both times. (2) the controller's `internal/busymark`
|
||||||
|
> (`<data>/ctl/busy`, 15 s) and the dashboard line „A Felhom ügyfélszolgálata HH:MM-kor újraindította a dobozt." for a
|
||||||
|
> day (`GET /guest/operator-reboot`). (3) the hub's button, the 30-minute expiry of an unpicked reboot, the slow-request
|
||||||
|
> line (> 5 s, path only, `/api/v1/wait` exempt) and the vouch save's per-leg timing. (4) R-1350: the hold clear records
|
||||||
|
> the stop as wanted. **Proven live on Tester 1:** refused during an off-site run; 67 s / 79 s; refused inside the hour;
|
||||||
|
> R-1350 no alarm in 6.5 min; on the release builds 66 s / 78 s with every app back and the household line. **R-1346
|
||||||
|
> measured:** the vouch save's time is the hub's own „fleet check + save" (38.1 of 38.3 s; registry 135 ms; page 6 ms in
|
||||||
|
> the cluster); ingress-nginx's default 60-s read timeout makes the 504. **R-1344 is still alarm only** — read the health
|
||||||
|
> log after 2026-10-30 07:00 UTC (planned events noted in the row). New row R-1351 (FileBrowser boot race, P4).
|
||||||
|
|
||||||
|
> **2026-10-29 (afternoon, attended) — RELEASED: hub v0.176.1, agent v0.169.0 (VOUCHED), controller v0.348.0 (global
|
||||||
|
> floor). Operator ruling D64 = B (`09` §3 decision 283, `04` §6.1):** „the hub's „Clear hold" works for every hold reason
|
||||||
|
> with no operator signature; the operator accepted that a compromised hub could clear a data-protecting hold. R-379
|
||||||
|
> („only a person clears a hold") now reads „a person — on the box, or the operator on the hub"." **Built:** (1) „Restart
|
||||||
|
> controller" per guest on the hub's host page, done by the AGENT (its own closed list: `restart_controller <vmid>` on
|
||||||
|
> the host-report reply, the result on the next host report; the hub pokes the agent over the tunnel). The supervisor's
|
||||||
|
> guards; the brake 3 restarts / 15 min counts operator restarts, then 30 min refused (`opBrakeUntil`). (2) `clear_hold`
|
||||||
|
> as the controller's fifth verb; the running controller clears and saves at once through `<data>/ctl/hold.sock`, and the
|
||||||
|
> box command goes through it too (R-1343). The hub's button per held-app row has a confirm step and, for a
|
||||||
|
> data-protecting hold, the restore warning. (3) R-1344: a controller that runs but does not answer → `controller_unresponsive`
|
||||||
|
> (operator-only) — ALARM ONLY until the night's health log (to 2026-10-30 07:00 UTC) is read; so far 631 probes, max
|
||||||
|
> 163 ms, no failed probe. **Proven live on Tester 1 on the release builds:** restart 14:49 CET, „done" on the hub 16 s
|
||||||
|
> after the press; crash-loop hold cleared 14:57:48 CET, gone on the box 3 s later and still gone after the next settings
|
||||||
|
> save (15:04:23). Earlier on rc builds: a restart with the controller stopped by hand, the fourth press refused; a
|
||||||
|
> restore_mixed hold cleared with the warning; R-1340/R-1341/R-1342 live. **Versions:** hub 0.176.0 (`sha256:53372198…`,
|
||||||
|
> `3d0c907b`) then 0.176.1 (`sha256:c105cb9a…`, from `2ccf180b`, manifest `c1f3a163`): 0.176.0 gated the restart button
|
||||||
|
> on a controller version the agent never reports, and named a token press „basic auth" — both found live, fixed red
|
||||||
|
> first. Agent 0.169.0 (`fc066de6…`, bundle `3ca33f9c…`, from `637c92c`), vouched 14:06:09 CET, `pinned.day0_agent`
|
||||||
|
> 0.169.0 (`db25eb0`). Controller 0.348.0 (`sha256:672fc1c1…`, from `2a0a17a`, CI 2515; rc.2 on 9202): per-box floors
|
||||||
|
> 14:41 CET, walk green on all three + https-only, global floor 14:48 CET (MinAgent 0.158.1). Rows: closed R-1343,
|
||||||
|
> R-1342, R-1340, R-1341; opened R-1344…R-1350 (R-1350: an `app_start_failed` 30 s after a clear — the hub's cooldown
|
||||||
|
> hid it). Register 226 → 229. Tester4 (a demo-hp test VM, off for 2 days) moves to the floor when it starts. Report:
|
||||||
|
> `REPORT-release-2026-10-29b.md`.
|
||||||
|
|
||||||
|
> **2026-10-29 (attended release run) — RELEASED: hub v0.175.0, agent v0.168.0 (VOUCHED), controller v0.347.0 (global
|
||||||
|
> floor), catalog `313eaf6`. Operator rulings D62 + D63 (`09` §3 decisions 281/282, `07` §6.3).** D62: an unused restore
|
||||||
|
> copy stays 30 days (was the night's 14, decided by CC — superseded); the kept-copies list shows each restore copy (date,
|
||||||
|
> size, „automatikusan törlődik") with „Törlés most" (typed name; a held/waiting copy never; 409 while a backup/restore
|
||||||
|
> runs or the app updates; one log line + operator-only event `restore_copy_deleted`, hub `19afefab`); controller
|
||||||
|
> `e3fce09`. D63: R-1256 H4 + H5's failed stop proven live on Tester 1 with a `-tags faultinject` test build
|
||||||
|
> (`cmd/controller/faultpoint_{on,off}.go`; the release image has no switch, `grep -c "FAULT INJECTED"` = 0) — nothing
|
||||||
|
> reverted; D62 proven end to end on the release build (hub stored the event). Hub 0.175.0 (`sha256:642e94b6…`, from
|
||||||
|
> `bf69bb91`, CI 2487; manifest `d6352cd8`, Deployment/hub only): one ceremony on Tester 1 end to end, 30 min clean. Agent
|
||||||
|
> 0.168.0 (`1e70196c…`, bundle `3415c034…`, tag `v0.168.0` at `91bd96c`): signed `agent_update` to the three test boxes,
|
||||||
|
> each committed, self-check 76/76; vouched 10:21:56 CET; `pinned.day0_agent` = 0.168.0 (`6b0b093`). Controller 0.347.0
|
||||||
|
> (`sha256:c79b06e6…`, from `fd75c7c`, CI 2491; rc.1 = `e3fce09` on 9202): per-box floors 09:57 CET, walk green on all
|
||||||
|
> three + https-only, a local restore on Tester 1, the OOM line (one `app_oom`, one mail), global floor 10:31 CET
|
||||||
|
> (MinAgent 0.158.1). Catalog: the night's two patches + CHANGELOG (`313eaf6`), Tester 1 read it. Tester 1's recovery code
|
||||||
|
> re-minted and stored as `TESTER1_RECOVERY_CODE_2026_10_29` (the 10-25 one opens only older packages; removal is the
|
||||||
|
> operator's, D60). Filed R-1340 (the mixed-hold mail tells the option-C story), R-1341 (the OOM WARN every 30 s); R-1342 (a copy deletable during the stable-start watch; a removed app alarmed — fixed on main, not released); closed R-1238, R-1338, R-1256, R-1255; R-528 WATCHING to its 11-12 check. Open for
|
||||||
|
> the operator: Radicale's hold record on Tester 1. Report: `REPORT-release-2026-10-29.md`.
|
||||||
|
|
||||||
|
> **2026-10-28 night (unattended) — restore edges, escrow edges, the host OOM count on main; NOTHING released.**
|
||||||
|
> **Decided by CC unattended — operator may reverse:** (1) an unused restore copy goes after 14 days, swept every 6 h
|
||||||
|
> (`07` §6.3 [DESIGN] 2026-10-28; options 14 d / for ever / 3 d); (2) R-1256's H4 and H5's failed-stop branch stay on main
|
||||||
|
> with test proofs only — the brief wants a live proof or a revert; H4 runs only on an off-site restore (9202 has no
|
||||||
|
> target) and both branches fail safe (hold or keep, never pour); the attended live proof or a revert of those two branches
|
||||||
|
> of controller `342a394` is owed; (3) the catalog changes (R-1338 measurement, Plex fixtures, the Plex ladder entry) are a
|
||||||
|
> prepared patch, `audits/night-2026-10-28/catalog-morning/`, not a push. Controller `342a394` (R-1256), `e33cffa`
|
||||||
|
> (R-1254 controller half), `eed23e7` (R-528), `20c5877` (unknown app → 404); agent `5ddd124` (R-1254), `6f933d6` (R-528,
|
||||||
|
> `GET /guest/oom-kills`); hub `1ee6d5a5` (R-1255). Live: H5 busy volume + H8 sweep on 9202; R-528 on Tester 1 (one
|
||||||
|
> `app_oom` „seen by docker,cgroup"); R-1238 Plex: the unclaimed route works from Plex's own container (its front door answers 401 to the
|
||||||
|
> proxy), bench + box 9202 proven, the ladder entry in the prepared patch (`catalog_since` kept at 2026-10-23: the pin did
|
||||||
|
> not move, the writer had set today). Claims walked: fail.customer-self-restore (route proven, stays partial:
|
||||||
|
> no non-operator yet), fault.selfheal (built → walked). R-1339 filed (a false „not mounted" log line).
|
||||||
|
|
||||||
|
> **2026-10-28 (evening, attended) — Gitea 28.2.0; Wanderer's search memory measured, not changed.** Gitea 1.27.3 → 28.2.0
|
||||||
|
> live `0baf8f9` (bench + box + undo; 1.27.3 refuses a 28-migrated database, the undo copy restores it). D57 → decision
|
||||||
|
> 280: imports only from github.com/codeberg.org/gitlab.com (`[migrations] EGRESS_MODE=strict` + `ALLOWED_HOST_LIST`, the
|
||||||
|
> 28.x names — 1.x's ALLOWED_DOMAINS is gone; the security review's „wrong key" notice is pre-28). The Gitea fixture posts
|
||||||
|
> every installer field (28 renamed the form). R-1338: `MEILI_MAX_INDEXING_MEMORY`=128 MiB helped on the bench (17.7 %)
|
||||||
|
> but NOT on 9202 (95.7 %, steady) → not pushed; the ladder has no entry type for a same-digest re-measure.
|
||||||
|
|
||||||
|
> **2026-10-28 (afternoon, attended at the STOPs) — Docker Hub stops refusing the watch; the four big steps.** R-1334:
|
||||||
|
> measured first — the 429s were the BURST (211 tag-list requests in ~2 min from six workers), not the count; `fe677bae`
|
||||||
|
> spaces Docker Hub requests (1/s), retries a 429 (Retry-After; NOT CHECKED after the 4th try) and caches tag lists 24 h
|
||||||
|
> per registry path on the runner (two runners = two caches); `8644450d` retries a registry 429 in the image reader. Dry
|
||||||
|
> runs 2458/2459: 0 refused — **no Docker Hub login needed** (the operator's key was not asked for). Big steps, each a
|
||||||
|
> NORMAL ladder step (`09` §3 [DESIGN] of today: no pin move without a step, also when no box runs the app), each with
|
||||||
|
> bench + box 9202 + the UNDO proven on 9202 (`audits/dockerhub-bigsteps-2026-10-28/undo_proof2.py`, probe on port 1):
|
||||||
|
> Wanderer v0.21.2 `1825247` (PocketBase internal, no `SUBDOMAIN_DB`, inbound federation off — no
|
||||||
|
> `POCKETBASE_PROXY_SECRET`; Meilisearch 384M, still `memory_tight` → R-1338; offered again, D59 ended, decision 279);
|
||||||
|
> Termix 2.8.0 → 2.9.2 `e0b9594` → 26.10.5 `589c957` (two pushes: `check-test-record-move` judges the whole push range
|
||||||
|
> as ONE move); Homepage v2.7.0 `1a254d4` (fixture reads its first-start config). New tools: `box_walk.py
|
||||||
|
> --to-definition`, fixtures Wanderer/Homepage/Seerr (Seerr brings a throwaway Jellyfin, `felhom-drill-jellyfin`).
|
||||||
|
> Termix X-Forwarded-For on 9202: the 6th wrong login is 429 on 2.8.0, 2.9.2 and 26.10.5 from the LAN (traefik
|
||||||
|
> replaces the header) and through the tunnel's address with Cloudflare's appended real address.
|
||||||
|
|
||||||
|
> **2026-10-28 (day, attended at the STOPs) — HTTP2Shell lane batches; the runtime cache fills itself in CI.** **Operator:**
|
||||||
|
> D61 = keep RomM offered (`09` §3 decision 277; the first RomM with Node ≥ 24.22.1 is a lane step). **Decided by CC (the
|
||||||
|
> brief's choice):** zstd in CI by a hash-pinned `zstandard` 0.25.0 wheel, not `zstd` in the runner image (decision 278) —
|
||||||
|
> this REPLACES the 10-27 night's „cache filled on DooPlex" (2) above. **Live on the operator's yes, CI green by hash:**
|
||||||
|
> catalog `b5b78f6` (HA 2026.10.4, Wanderer hidden), `0158b50` (Immich v3.3.5, repair re-proven), `abf3db3` (ghost,
|
||||||
|
> metube, zipline), `71e3fdf` (karakeep, tandoor), `651a991` (papra, dawarich), `0829994` (adventurelog v0.13.1 + health
|
||||||
|
> `start_period` 300 s — its backend needs ~2 min 45 s after an update). Box-walk tool fixes (catalog `box_walk.py`):
|
||||||
|
> `move_images` moves two services sharing one image; the walk loads the drill into the box mirror before the install.
|
||||||
|
> felhom.eu: `a3247d9f` (zstd), `73c91eac` (decoder named in words), `29e9cd36` (the watch's memory keyed per group —
|
||||||
|
> `dooplex:<image>|<id>`, old keys copied once; W13). Corrected: Calibre-Web DOES load Brotli (`/proc/<pid>/maps` from the
|
||||||
|
> guest; in-container maps were unreadable). Rows: R-1313, R-1322, R-1327, R-1332 closed; R-1334–R-1337 opened (229 → 229).
|
||||||
|
> 9202 ends on the LIVE catalog; the drill is back at FROM for all lane apps (`5c51ddc`). Trap: a `git pull --autostash`
|
||||||
|
> in felhom.eu emptied a walk's `walk.txt` that was being written in the clone — walk output goes to the scratchpad now.
|
||||||
|
|
||||||
|
> **2026-10-27 night (attended to 22:40, then alone) — D60, the runtime watch, HTTP2Shell.** **Decided by CC unattended
|
||||||
|
> — operator may reverse:** (1) Home Assistant's monthly release is the same main version (one comparator) → a LANE step
|
||||||
|
> with CVE-2026-111497 (`09` §3 decision 276); (2) the watch's runtime cache is filled on DooPlex (zstd, the stored Docker
|
||||||
|
> Hub login through `sec run`) and committed in `state.json`, instead of a zstd dependency in CI (`08` §11); (3) no
|
||||||
|
> Paperless step (it decodes only replies to its own calls, R-1331); (4) RomM kept (STATUS D61, R-1330); (5) the live
|
||||||
|
> catalog changes stay PATCHES/drill commits at night — the push waits for the operator. **Done:** D60 enforced
|
||||||
|
> (cc-secrets 0.4.0 `092e282` + two review hardenings `27fd806`/`e99f937`; proven in two new CC sessions); D56–D59
|
||||||
|
> recorded (decisions 271–275); hub `startupProbe` 180 s live on the operator's yes (`a69ddbde`, one start, 0 restarts;
|
||||||
|
> R-1326 closed); `scripts/image_runtime.py` + the watch's „runtime" source and „dooplex" group (`f5a4995d`; 154 images
|
||||||
|
> cached; positive controls HA + RomM, negative audiobookshelf); ghcr.io tag paging fixed; lane walks PROVEN on 9202 and
|
||||||
|
> written to the DRILL: Home Assistant 2026.9.4 → 2026.10.4 (`3c8ebb5`), Immich v3.2.4 → v3.3.5 (`34b14d5`); Wanderer
|
||||||
|
> hidden in the drill (`ee18bff`). Rows R-1327–R-1332 opened, R-1326 closed. Morning: the catalog push (writer on the
|
||||||
|
> live catalog), the homelab list to the homelab session.
|
||||||
|
|
||||||
|
> **2026-10-27 evening — R-1299 the clean-up window button (hub v0.174.0 + v0.174.1); cc-secrets 0.3.0 `sec rm`.** The
|
||||||
|
> customer page's „Backup & DR" tab has the „Off-site clean-up window" card (`internal/web/offsite_window.go`): the
|
||||||
|
> one-shot grant through a confirm step, same route and rule (`/offsite/window-grant/<id>`, `wr(grpOffsite)`), cancel,
|
||||||
|
> a limit bounded by the store's snapshot count (ceiling 500). Every grant/cancel is a STORED operator-only info event
|
||||||
|
> `offsite_window_grant` — 0.174.0 only called the dispatcher (which writes no row) and its test asserted the call; the
|
||||||
|
> operator's live click exposed it, 0.174.1 writes the row (`store.SaveEvent`) and the test reads rows. Hub builds came
|
||||||
|
> from a clean clone (`FELHOM_HUB_REPO_DIR`) — the shared clone held another session's uncommitted installer edit.
|
||||||
|
> Start-up 31–36 s vs a liveness kill at ~65 s: R-1326. `sec rm NAME` (cc-secrets `dc82cf9`); 46 dead names removed on
|
||||||
|
> the operator's yes (92 → 46). Agent 0.167.0 vouched by the operator; `pinned.day0_agent` = 0.167.0 (agent `3055bc5`).
|
||||||
|
|
||||||
|
> **2026-10-27 (attended release block) — RELEASED: hub v0.173.0, agent v0.167.0, controller v0.346.0.** Hub 0.173.0
|
||||||
|
> deployed 09:35 CET (image `sha256:458bfb03…`, from `29134c52`, ArgoCD sync of Deployment/hub only, 30-min watch clean).
|
||||||
|
> Agent 0.167.0 by `release-agent.sh` from a clean clone (agent `28bdd282…`, bundle `9d034416…`; tag `v0.167.0`), signed
|
||||||
|
> `agent_update` to the three test boxes (configs/ unchanged → no bundle job); **NOT vouched yet** — after the vouch set
|
||||||
|
> `pinned.day0_agent` in `felhom-agent/scripts/retention-policy.json` (D50). Controller 0.346.0 from `b1b9be3` (image
|
||||||
|
> `sha256:3d910b03…`, CI 2423): per-box floors 15:09 CET, box walk green, held-apps lists match (0 = 0), global floor
|
||||||
|
> 15:37 CET (MinAgent 0.158.1). **R-1250 fixed the same day (`09` §3 decision 269):** the hold failed four times
|
||||||
|
> („not quiet" — qBittorrent sends ~14 s after its limit falls; the agent reads 5–15 s after the call); `710ceac`
|
||||||
|
> `upwatch.waitDrained` asks only after the held app reports its own upload near zero; live on `0.346.0-rc.1` on 9201:
|
||||||
|
> 94.5 Mbit/s, the limit back exactly (`audits/linetest-2026-10-27/`). **R-1218's live proof waived (decision 270):** the
|
||||||
|
> hub page's off-site switch did not reach Tester 1's saved target (R-1312). demo-hp's slot is minute 95 (11:35 CET),
|
||||||
|
> Tester 1's 213, demo-felhom's 20 — each host's own. D55's window ran 03:18Z: 223 removed, 373 → 150. R-1174 closed
|
||||||
|
> (the status gate refuses since today). Closed 11 rows; opened R-1312.
|
||||||
|
|
||||||
|
> **2026-10-26 night (alone) — the data-safety fixes, on `main`, NOT released.** Controller: R-1220 `7e15a37`
|
||||||
|
> (`liveDriveBinds(drives)`: an unreadable app.yaml = installed), R-1218/R-1219 `1170b48` (stage when the password FILE
|
||||||
|
> exists; no ceremony during a recovery offer), R-1187 `dfd0f8e`…`90a2530` (.fab: links as links, FIFO refused, `os.Root`
|
||||||
|
> writes, downward-only links, owners/modes, busy and bad-path refusals; the import's compose gets `USERDATA_PATH`/
|
||||||
|
> `IMPORT_PATH`; the DB loads through `appbackup.ImportDump` — the import's `mysqladmin`/`mysql` broke every MariaDB 11+
|
||||||
|
> import), R-1298 `b156e45`, Go 1.27.3 `2560cf1`. Agent: R-1232 `3911cc6` (own-key archives only in the due-check),
|
||||||
|
> Go 1.27.3 `fbbb168`. Hub: R-1218 `09b5f02f` (409 on a key-less package over a keyed one), Go 1.27.3 `d625a143`.
|
||||||
|
> Live: `audits/night-fixes-2026-10-26/` (R-1220, R-1187, R-1298 on 9202 with local test images; R-1232 on a BYO Tester4
|
||||||
|
> VM, uninstalled and destroyed). R-1218's live shape needs the operator (off-site off on Tester 1) — else reverted
|
||||||
|
> before the release. R-528: design in the row, not built. New: R-1299, R-1302.
|
||||||
|
|
||||||
|
> **2026-10-26 (night) — D55 = 260 (`09` §3 decision 268); the grant is SET.** CC used the operator's one-purpose token
|
||||||
|
> `HUB_TOKEN_D55_WINDOW` („cc-d55-window", Off-site write only) once: `POST /offsite/window-grant/demo-hp` `max_remove=260`
|
||||||
|
> → 200, 19:10:31Z; read back `offsite_window_grant:demo-hp = max:260` (hub DB with WAL) and the hub log line „granted ONE
|
||||||
|
> clean-up window … 260". The grant is consumed at demo-hp's next window request (after its next nightly off-site run).
|
||||||
|
> The token is the operator's to revoke after that. The hub has no page button for the grant (R-1299).
|
||||||
|
|
||||||
|
> **2026-10-26 (evening run) — R-1297 A built (`ecd82aba`, `e33ec821`); D53 built, commit only (controller `5bd03ad`, hub
|
||||||
|
> `7e57e824`); D54 blocked on a scope.** CI: `script_tests_gate.py` `PATH_FILTERED` (decoys / due-checks / dooplex-offsite
|
||||||
|
> run on a branch push only when their paths change; `FELHOM_CHANGED_FILES` set only for `push:refs/heads/*` in
|
||||||
|
> `gates.yml`; nightly 03:20 UTC + tags run all); `test_repo_gates.py` / due-checks group G narrow `repo_gates.GATES` by
|
||||||
|
> label (the wiring, not a second full run). The decoy suite's register-shape cases now assert the script's exit code
|
||||||
|
> (an inert `main()` passed them). D53: `internal/holdreport` (controller) → `app_held` / `app_hold_cleared` (info) →
|
||||||
|
> hub `held_apps` + `HeldAppsChecker` (`app_held_reminder`, operator-only, per-app cooldown; grace 6 h / 1 d / 3 d, then
|
||||||
|
> weekly; `alerting.held_app_grace_override` test-only). Live on 9202 against a TEST hub process (no hub deploy):
|
||||||
|
> `audits/d53-held-apps-2026-10-26/`. D54: CC's hub token lacks `offsite:write` (403) — STATUS D55 (the cap: the plan
|
||||||
|
> grows ~14 a night, 220 would be refused). New R-1298 (9202's orphan holds would be reminded weekly after D53 ships).
|
||||||
|
|
||||||
|
> **2026-10-26 (evening) — operator rulings by starting the evening brief: D53 = operator-only held-app reminders first
|
||||||
|
> (`09` §3 decision 265; build `08` §10); D54 = one clean-up window for demo-hp with cap 220 through the hub's one-shot
|
||||||
|
> grant `POST /offsite/window-grant/<customer>` (decision 266; the cap rule unchanged); R-1297 (B) DONE by the operator — a
|
||||||
|
> second CI runner, both runners 2 CPUs (decision 267). CI trim (R-1297 A) is CC's: every removed run keeps a path filter,
|
||||||
|
> a tag run or a nightly run.
|
||||||
|
|
||||||
|
> **2026-10-26 (afternoon run) — R-1295 fixed at the cause and released; R-1292 released; R-1294 read; R-1285 committed;
|
||||||
|
> R-1291 designed.** R-1295's cause, measured on a throwaway PBS 4.2.5: the PBS proxy caches permissions up to 5 s and an
|
||||||
|
> ACL-only change does not clear it, so under concurrent requests a new token is refused for ~5 s (v1.3.0 11/30 failed,
|
||||||
|
> v1.4.0 0/30). Fixed: felhom-tenantsync v1.4.0 INSTALLED on ep0 (operator yes), hub v0.172.0 RELEASED (retry + operator
|
||||||
|
> mail `pbsdr_provision_failed`); live: a fresh Tester4 install made its first whole-guest copy. Controller v0.345.0
|
||||||
|
> RELEASED (global floor 13:27:04Z): the 50 % rule owns qBittorrent's NORMAL slot (live on Tester 1). R-1294: demo-hp's
|
||||||
|
> plan is honest but can never catch up with the cap (D54). R-1285: agent `b1bc63d`, the pass after the commit, not
|
||||||
|
> released (live needs a box from 0.164.0). R-1291: `08` §10 design (D53). New R-1296, R-1297 (CI speed: operator „both").
|
||||||
|
|
||||||
|
> **2026-10-26 (attended release run) — released: hub 0.171.0, agent 0.166.0 (vouched 10:04:55 CET), controller 0.344.0
|
||||||
|
> (global floor 09:07:47 UTC, MinAgent 0.158.1), catalog `73c165d`; golden 0.344.0 baked and vouched (10:26:23 CET).**
|
||||||
|
> Live on the release builds (`audits/release-monday-2026-10-26/`): R-1287 (the set-aside mails „⚠️ … nothing failed"),
|
||||||
|
> R-1289 (the vouch line names 0.344.0), R-1280 (a 25 h job REJECTED on demo-hp, a 30 min one ran), R-1281 (forged header
|
||||||
|
> 000 with a working control; the household's files 200; OnlyOffice fetches the document over `felhom-files`), R-1282,
|
||||||
|
> R-1278, R-1286, R-1283 — closed. **R-1285 back to OPEN:** in the normal two-job delivery the bundle runs in the OLD
|
||||||
|
> agent process, so the policy pass waits for the next bundle. R-1261 stays VERIFY until Tuesday's live line test (R-1250).
|
||||||
|
> New: R-1294 (demo-hp's weekly off-site clean-up refused by its own cap, 209 > 179). The golden waiver file is removed —
|
||||||
|
> the gate is green on the new golden alone. D50 pin `day0_agent` → 0.166.0.
|
||||||
|
|
||||||
|
> **2026-10-26 (attended release run) — D51 = Start; D52 = bake the golden today on v0.344.0.** (`09` §3 decisions 263,
|
||||||
|
> 264.) D51: the operator pressed Start on demo-hp's Kimai (07:57 local); the crash-loop hold lifted, both containers
|
||||||
|
> healthy, 0 restarts (R-1286). D52: the Sunday 2026-10-25 bake was missed (not in that night's brief); the golden waiver
|
||||||
|
> is renewed for 2026-10-26 only (`expires: 2026-10-27`) so the release commits can be pushed; the golden is baked on
|
||||||
|
> v0.344.0 the same day and the waiver is removed after the vouch. The weekly Sunday bake stays the rule; every
|
||||||
|
> Sunday-night brief names it.
|
||||||
|
|
||||||
|
> **2026-10-25 night (unattended, commit only) — the night's fixes, each red-proved; proven live on a BYO test VM (demo-hp
|
||||||
|
> VM 9372, customer Tester4, agent 0.165.0, controller rc.1–rc.3).** Controller (`a79a292`…`7ea942a`, rides v0.344.0):
|
||||||
|
> R-1261 the line hold holds EVERY running seeder, drops a removed app's record, names qBittorrent's slot (`up_limit` /
|
||||||
|
> `alt_up_limit` via `app/preferences` — `transfer/uploadLimit` follows the current speed mode, measured); R-1288 the
|
||||||
|
> restore of a set-aside unit sets the replaced unit aside AT the restore, no alarm (the brief's safest reading, `07`
|
||||||
|
> §8.2d changed); R-1282 the „later" media restart stored and resumed; R-1281 FileBrowser served on its own
|
||||||
|
> `felhom-files` network (traefik + OnlyOffice while linked), off traefik-public — the forged header from an app went
|
||||||
|
> 200 → no answer; R-1286 every hold reason says what it is (Kimai's was a crash-loop stop, not a failed restore). Hub
|
||||||
|
> (`f6328486`, v0.171.0): R-1287 a set-aside-only digest is a warning „nothing failed"; R-1289 suffixed versions never
|
||||||
|
> „the fleet's newest". Agent (`d912307`, `abcc1e6`, next agent release + vouch): R-1280 a signed window ≤ 24 h; R-1285
|
||||||
|
> the agent runs the wrapper's `bundle-policy` pass after every bundle install. Drill catalog `6733764`: R-1283 MeTube
|
||||||
|
> `optional` (a named ruling in the download-binds gate). New rows R-1291 (a held app has no lasting signal), R-1292
|
||||||
|
> (the 50 % rule follows the speed mode).
|
||||||
|
|
||||||
|
> **2026-10-25 (evening, attended) — D50 = A; installer 1.36.1 published.** (`09` §3 decision 262.) The registry keeps the
|
||||||
|
> newest 5 agent versions + the MinAgent floor + the day-0 agent; the agent CI asserts exactly these
|
||||||
|
> (`retention-policy.json` `pinned`, copied from hub state by the vouching session — CI holds no hub secret). The operator
|
||||||
|
> also pruned controller, golden and hub images by hand; every artifact still named (manifest agent/bundle/golden, the
|
||||||
|
> floor agent, tags v0.158.1+, controller 0.343.0/0.342.0/0.331.0, samba 1.2.0, the ISO) downloads
|
||||||
|
> (`audits/night-2026-10-25/part1-artifacts.txt`). Installer 1.36.1 (`installer-v1.36.1` → `79ee3e33`, served sha256
|
||||||
|
> `e18cdd17…`): the preflight words 401/404/409 again (R-1284).
|
||||||
|
|
||||||
|
> **2026-10-25 (attended release run) — D49 = A, D44/D45 = all picks, D46 = yes; released: hub 0.170.0, agent 0.165.0,
|
||||||
|
> controller 0.343.0, catalog `ebbf1f4`** (`09` §3 decisions 258–261; `REPORT-release-2026-10-26.md`). **D49 = A:** a
|
||||||
|
> restore whose app does not start keeps both copies and never pours back by itself (`07` §8.2e). **Fix A** (controller
|
||||||
|
> `restore_verdict.go`): „started" = running on every sample of 45 s, no `RestartCount` growth, every healthcheck
|
||||||
|
> `healthy`; the undo copy is released only after 3 more minutes of the same (10 s samples, 30 min watch, a newer restore
|
||||||
|
> ends the watch — `restoreBegunAfter`). **Fix B:** the `.fab` import pulls first (`Exporter.SetImagePreparer` →
|
||||||
|
> `PrepareRestoreImages`). **D47 follow-up:** a set-aside mails the operator (`backup_run_failures`, leg
|
||||||
|
> `unit-set-aside`) — its hub subject still says „failed to back up" (R-1287). Agent 0.165.0 vouched 18:38 CET; on a
|
||||||
|
> BYO host our SSH drop-in is removed only on the bundle run AFTER the one that installs the new wrapper (R-1285).
|
||||||
|
> Installer 1.36.1 on main, not published (R-1284: `http_code` printed the code twice). R-1250's live proof is a dated
|
||||||
|
> check on 2026-10-27 (the agent tests only on its due Tuesday 10:00–15:59).
|
||||||
|
|
||||||
|
> **2026-10-25 08:38 — operator rulings by starting the R-1269 brief: D47 and D48** (`09` §3 decisions 256, 257;
|
||||||
|
> `audits/restore-truth-2026-10-25/README.md`). **D47:** an existing recovery unit is never changed in part — its
|
||||||
|
> definition and data are written only together, by a backup of the same install; a unit an earlier install left is set
|
||||||
|
> aside whole and stays restorable (`07` §8.2d). **D48:** every restore path resolves and pulls every image its
|
||||||
|
> definition names before the first change to the app; a failed pull is a refusal with the app untouched (`07` §6.6).
|
||||||
|
> **Confirmed, no change:** `07` §6.6's version rule; R-698 and R-933. Built the same day, controller commits only (rides
|
||||||
|
> v0.343.0): `02daebe` (D47 / R-1269), `b680db8` (R-1270: a restore whose app does not start is never a success; the
|
||||||
|
> safest reading — no automatic pour-back, the undo copy kept, a late start turns it into a success), `2d71270` (D48),
|
||||||
|
> `bd9f4f7` (R-1271: a copy dated more than 10 minutes in the future is never „the newest").
|
||||||
|
|
||||||
|
> **2026-10-24 (afternoon, attended) — two DooPlex backup alarms — DONE.** One cause: the 2026-10-23 window reboot
|
||||||
|
> restarted `postgresql-2`; CNPG 1.29 sets the role `postgres` from the superuser secret, then from the
|
||||||
|
> `bootstrap.initdb` owner secret (`owner: postgres`), so `postgres-superuser` is in force at every start; the dump
|
||||||
|
> CronJob read `postgresql-superuser` → 28P01 from 16:00Z → `PostgresqlDumpStale`; the 00:20 Gitea push refused the
|
||||||
|
> stale dump → `DooplexGiteaOffsiteStale`. **Operator rulings 2026-10-24:** (1) fix the CronJob only (homelab
|
||||||
|
> `531023d`; the Cluster keeps two writers — homelab list); (2) apply the routing change by restarting Alertmanager
|
||||||
|
> (its `envsubst` init container makes `/-/reload` useless; homelab `22d7697`: critical 4 h, warning 12 h, one mail per
|
||||||
|
> backup group and per drive); (3) the R-1250 live proof may use DooPlex's k3s qBittorrent as the downloading peer
|
||||||
|
> (one private LAN torrent in its container `/tmp`, deleted with its files). R-1250 committed (agent `29bcf44`,
|
||||||
|
> `1e087bb`; controller `046b808`), not released. R-934 closed (the disk was pulled 2026-10-23). Catalog `7de554e`:
|
||||||
|
> the box walk sends the drive itself as HDD_PATH (R-839). Report: `REPORT-dooplex-alarms-2026-10-24.md`.
|
||||||
|
|
||||||
|
> **2026-10-24 (attended release run) — DONE.** Hub 0.168.0 (05:39Z) and 0.169.0 (07:51Z) deployed; agent 0.164.0
|
||||||
|
> released, delivered to the three test boxes and vouched by the operator; controller 0.342.0 the global floor (08:29Z,
|
||||||
|
> MinAgent 0.158.1). ep0 runs `felhom-iperf3` (tunnel only). First line tests: Tester 1 94.3, demo-hp 94.4, demo-felhom
|
||||||
|
> 92.6 Mbit/s; demo-felhom's qBittorrent limited to 46 Mbit/s. D40 in effect (dated check 2026-10-25, R-1243). Opened
|
||||||
|
> R-1249 (the start-up VACUUM), R-1250 (a seeding qBittorrent is never tested). Report `REPORT-attended-release-2026-10-24.md`.
|
||||||
|
|
||||||
|
> **2026-10-24 07:22 / 07:26 — operator rulings by starting the attended release brief (D40, D41, D42; the line test).**
|
||||||
|
> **D40 (R-1243) = stop the Longhorn backup of the hub's volume**; the encrypted 6-hourly copy to ep0 stays the hub's
|
||||||
|
> off-site copy; the operator deleted the two old plain Longhorn backups himself (UI); the Longhorn UI cannot take the
|
||||||
|
> volume out of the backup job, so the change is in a manifest. **D41 (R-1245) = 50 %** of the measured line for
|
||||||
|
> qBittorrent's default upload limit. **D42 (R-1246) = the family is told only about the torrent limit**;
|
||||||
|
> `app_unexplained_upload` / `app_unexplained_cpu` are operator-only. **Line measure (R-1247) = a weekly iperf3 upload
|
||||||
|
> test from the box's host to an iperf3 server on ep0, inside the WireGuard tunnel only** (operator 07:26; his design of
|
||||||
|
> 2026-10-23: „during a quiet window the box measures the upload bandwidth"); the passive floor stays as a lower bound.
|
||||||
|
> This is the one change on ep0 the run may make (Tier 2 — the operator's yes is starting the run). CC still stops
|
||||||
|
> before each hub deploy, before the global controller floor, and for the agent vouch.
|
||||||
|
> **Two more answers in the same run (AskUserQuestion, 2026-10-24 ~08:2x CEST):** (1) **the agent speaks iperf3 itself**
|
||||||
|
> (no host has the iperf3 program; nothing is installed on hosts) — agent `internal/linetest`; (2) **the live hub file's
|
||||||
|
> old token bytes are accepted, not cleaned** (the start-up VACUUM did not finish on the Longhorn disk, R-1249; a clean-up
|
||||||
|
> needs an 8–20 min hub stop, a trim and the volume's snapshots deleted; only root on DooPlex can read the volume, and it
|
||||||
|
> holds the key; every copy that leaves is clean) — written into `05` §16.2 with the rule „a hand copy of hub.db is made
|
||||||
|
> with VACUUM INTO, never cp/tar".
|
||||||
|
|
||||||
|
> **2026-10-23 19:22 — operator rulings by starting the weekend-night brief (R-1241's Felhom half; the upload watch).**
|
||||||
|
> **Part 1 = R-1241's Felhom half:** seal R-975 (done in hub 0.168.0, not deployed — `05` §16.2,
|
||||||
|
> `audits/hub-token-seal-2026-10-23/`); the hub's plain Longhorn copies are decide row R-1243 (CC's pick: take the hub
|
||||||
|
> volume out of the Longhorn backup; ep0's encrypted copy is the backup). **Part 2 = the upload watch, shaped by the
|
||||||
|
> operator:** the box measures its own upload line (first from what it already sees — the off-site backup's upload
|
||||||
|
> rate — a short test at a quiet hour only if needed); on a sustained high upload it attributes the bytes to an app's
|
||||||
|
> container and ASKS THE APP (qBittorrent's seeding, Plex's remote sessions); a mismatch is the alarm
|
||||||
|
> (`app_unexplained_upload`, the same shape for CPU, `app_unexplained_cpu`) — operator only. qBittorrent gets a default
|
||||||
|
> upload limit from the measured line; a limit the household set is respected. **Open for the operator, CC's pick in
|
||||||
|
> STATUS:** the default limit's size (CC: 50 % of the measured upload) and who is told (CC: the family only about the
|
||||||
|
> qBittorrent limit they can change; the unexplained alarms to the operator only — the R-388 rule).
|
||||||
|
|
||||||
|
> **2026-10-24 (night, unattended) — the restore and backup holes; operator ruling D43 = A**
|
||||||
|
> (`audits/restore-holes-2026-10-24/README.md`). **Ruling D43 = A (operator, 2026-10-24 evening, by starting the run):**
|
||||||
|
> the root-key-only sshd drop-in applies only in appliance mode; BYO hosts keep their SSH settings (`11` §1, `03`).
|
||||||
|
> Fixed on main, nothing released: controller `bb2809d`/`062f9b4` (R-1251: a pending put-back blocks every start — a
|
||||||
|
> restore hold `putback_pending`; the put-back runs first when the drive binds; a failed put-back holds; the copy synced
|
||||||
|
> before the marker names it), `518a135` (R-1252: reconcile only under a valid classification), `c9b4b82` (R-1253: a
|
||||||
|
> stop-first export refuses while another operation holds the marker); hub `dcf4f84c` (R-1254: escrow in one
|
||||||
|
> transaction; image built locally, not pushed); agent `c8d3c17` (R-1257: D43). Proven: R-1251 on Tester 1 (power cut,
|
||||||
|
> disk off, disk back), R-1252 and R-1253 on 9202. D44/D45 prepared: `audits/review-round-2-2026-10-24/D44-D45-picks.md`.
|
||||||
|
|
||||||
|
> **2026-10-24 (evening, unattended) — review round 2** (`audits/review-round-2-2026-10-24/README.md`). The 17 architecture
|
||||||
|
> documents checked against `main`: facts corrected at `file:line` (EDITS.md), rulings and promises left for the operator
|
||||||
|
> (R-1266, D45), undocumented decisions written as notes (R-1265, D44). New: **R-1251 (P1)** a cut restore of a drive-backed
|
||||||
|
> app is put back over newer data (shown with the real `Recover`); **R-1252 (P2)** a rejected backup block makes the
|
||||||
|
> second-drive reconcile delete the household's copies (reproduced on 9202). Controller `b01bb4b` (unreleased, next
|
||||||
|
> v0.343.0): a media restart skips household-stopped apps (R-721); the FileBrowser SSO header file appears whole.
|
||||||
|
|
||||||
|
> **2026-10-23 ~12:00 — operator rulings by starting the D39 brief (`09` §3 decision 255; R-1239 closed).** **D39 = B:** a
|
||||||
|
> `security_fix` ladder step — box walk on 9202 required, bench optional, taken by the night leg. **No operator word per
|
||||||
|
> incident:** a public advisory id (CVE or GHSA) + its https:// notice is the authority; the session that writes the step
|
||||||
|
> reports it to the operator after the push. The design (Claude.ai's recommendation) was taken as written; one fact in the
|
||||||
|
> brief was checked and held (Plex's box fixture exists but is a `NoRoute` stub, so `seed: "none"` is right). Built:
|
||||||
|
> catalog `3898593` (`ladder.py` `check_security` + `engine_majors_crossed` importing `check-engine-major.py`; the move gate
|
||||||
|
> accepts it; `check-test-record.py` WARNs on a head with no bench; `upgrade-test.py --write-ladder --security`;
|
||||||
|
> `box_walk.py --security` + `refresh_box_mirror()`), controller **0.341.0** (`legCandidate` takes `proven` or
|
||||||
|
> `security_fix`; global floor 11:24 UTC with MinAgent 0.158.1, the operator's yes in chat). Proof:
|
||||||
|
> `audits/security-lane-2026-10-23/`. Older controllers skip the step at night and take it on a hand press (proven).
|
||||||
|
|
||||||
|
> **2026-10-23 10:56 — operator ruling: Plex to the fixed version now, without the test record (Plex2Shell).**
|
||||||
|
> Verbatim: *„bump the version to the newest in the app template immediately (as there are no customers yet), I want the
|
||||||
|
> demo boxes on the newest version asap."* It skips `09` §3 decision 13's test record **once, for Plex only**. Done: catalog
|
||||||
|
> `6cc317d` (Plex 1.41.4.9463-630c9f557 → 1.43.5.11029-6bf8af3f0, CVE-2026-96652/-96654/-96655/-96656), pushed
|
||||||
|
> `--no-verify` past `check-test-record-move.py` (its refusal is in `app-catalog-felhom.eu/REPORT.md`); the only box running
|
||||||
|
> Plex (demo-hp 9201) was checked for the attack's signs (none) and updated by the Update press; the guide re-checked
|
||||||
|
> (`guide.checked_version` moved). Decision 13 itself is unchanged. Follow-ups: R-1238 (Plex's test record and fixture),
|
||||||
|
> R-1239 (a fast lane for security fixes — the operator's decision, D39). Evidence: `audits/plex-demo-boxes-2026-10-23/`.
|
||||||
|
|
||||||
|
> **2026-10-23 morning — the operator's „go" for the day brief: the releases, D31 as built, D32–D36 (decide rows in `STATUS-HISTORY.md`).**
|
||||||
|
> **Releases:** controller 0.340.0 (global floor 06:41 UTC, MinAgent 0.158.1); agent 0.163.0 (binary sha256 561b5316…, bundle 3afb88d9…, vouched by the operator 07:58 UTC; delivered to the three test boxes by signed jobs through a step bundle `0.163.0-step1` — the D4 drop-in is a new path, R-880); installer 1.36.0 (tag `installer-v1.36.0` → 83388c08, both webpage pins); hub 0.167.0 (sha256:177860a9…, 08:27 UTC); the website `0bb57a4f`.
|
||||||
|
> **D31** = as built (the re-issue reads the newest deletion of a KEY-HOLDING host; an install that got the password but no recovery package is rotated without a press). **D32** = yes: the sheet is names and places only, one A4, HU + EN; the operator prints and fills it. **D33** = yes: `scripts/release-copy/` nightly 01:40, newest 3 image tags (controller, hub, samba) + 5 agent versions, skopeo `dir:` (digest-true — why not `docker save`: a re-push re-compresses), ep0 `/srv/felhom-release-copy` under the one limited user `felhom-relcopy` (rrsync), cap 3 GB enforced by the sender, a read-only registry token of its own. **D34** = yes: hub snapshots at 02/08/14/20 (catch-up > 6 h), DooPlex push :30 after, limit 7 h + a wait for a snapshot being written (R-1234: 2 h was wrong), alarm 13 h; ep0 retention unchanged (keep-daily 14, keep-weekly 8). **D35** = yes, after the release week (R-1228, dated 2026-10-30). **D36** = yes, both sentences, published.
|
||||||
|
> **Added on the operator's word the same day (not soaked):** D34's hub half went into 0.167.0 (asked before the build). Record: `audits/day-2026-10-23/`, `REPORT-day-2026-10-23.md`.
|
||||||
|
|
||||||
> **2026-10-22 evening — operator rulings D29, D30, D31 (CC's picks accepted; the decide rows now in `STATUS-HISTORY.md`).**
|
> **2026-10-22 evening — operator rulings D29, D30, D31 (CC's picks accepted; the decide rows now in `STATUS-HISTORY.md`).**
|
||||||
> **D29** = no: the controller repository stays public for now (R-925). **D30** = the agent makes a new WireGuard key and
|
> **D29** = no: the controller repository stays public for now (R-925). **D30** = the agent makes a new WireGuard key and
|
||||||
> registers it on the existing peer path (R-1204). **D31** = the hub skips a deleted install that never made a key when it
|
> registers it on the existing peer path (R-1204). **D31** = the hub skips a deleted install that never made a key when it
|
||||||
|
|||||||
@@ -0,0 +1,72 @@
|
|||||||
|
# REPORT — 2026-10-26 afternoon: R-1295 fixed at the cause, R-1294 read, R-1285 / R-1292, R-1291 designed, releases
|
||||||
|
|
||||||
|
**Plain words first.**
|
||||||
|
1. **R-1295 cause:** PBS's proxy keeps permissions in a 5-second cache, and a new access key's grant does not clear it.
|
||||||
|
When other boxes are talking to ep0 at that moment, the new key is refused for a few seconds, and the old script
|
||||||
|
gave up after one quick retry. **Fixed at the cause on ep0** (the script now waits up to 15 s) **and in the hub**
|
||||||
|
(retries, and **a failure now mails you** — yes). Live: a fresh Tester4 install set up its whole-box off-site copy.
|
||||||
|
2. **Tester 2's reinstall is safe** from this failure: the fix is on ep0 and in the hub, and if anything still fails you
|
||||||
|
get a mail that says what to do.
|
||||||
|
3. **R-1294:** demo-hp's clean-up plan is **honest** (old copies only, every app keeps its copies), but it can never get
|
||||||
|
back under the safety cap by itself. **Your pick D54:** a one-time window with a raised cap (my pick).
|
||||||
|
4. **R-1292 fixed and released; R-1285 fixed, not released** (its live proof needs a box coming from 0.164.0).
|
||||||
|
5. **Released:** hub 0.172.0, controller 0.345.0 (every live box), felhom-tenantsync v1.4.0 on ep0. Not released: agent.
|
||||||
|
|
||||||
|
Safest readings taken alone (said first): R-1292's live test ran on Tester 1, not 9202 (9202 has no tested line, so the
|
||||||
|
rule never acts there); R-1285 has no live test today (a box coming from 0.164.0 needs a signed downgrade chain); the
|
||||||
|
fresh-install proof ran after the hub rc deploy, both on your yes.
|
||||||
|
|
||||||
|
## Each fix, before → after
|
||||||
|
|
||||||
|
- **R-1295 (hub v0.172.0 + tenantsync v1.4.0):** a new access key refused by the PBS cache → rolled back, no off-site copy,
|
||||||
|
only a log line → the script waits up to 15 s (the admin key too); the hub retries 3×, 7 s apart; a final failure mails
|
||||||
|
you („… could not be set up — save the customer config to retry"). Measured 11/30 → 0/30 under load; red-proved.
|
||||||
|
- **R-1292 (controller v0.345.0):** with alternative speed on, the 50 % rule changed the alternative limit or gave up →
|
||||||
|
it sets only the normal limit. Live on Tester 1; red-proved.
|
||||||
|
- **R-1285 (agent `b1bc63d`, not released):** the old agent installed the bundle, so the new policy pass waited a release
|
||||||
|
→ the new agent runs it once after its update commits. Tested in the real order; red-proved.
|
||||||
|
- **R-1294 (read only):** no change; D54.
|
||||||
|
- **R-1291 (design only):** `08` §10; D53.
|
||||||
|
|
||||||
|
## Versions, CI by commit
|
||||||
|
|
||||||
|
| What | Version / digest | Commit → CI |
|
||||||
|
|---|---|---|
|
||||||
|
| hub | 0.172.0, `sha256:7c8d129a…` (rc.1 `f6cb8f57…` for the proof) | `722013da` 2362 ✓, `232f42d2` 2366 ✓; manifests `ee38dee2` 2364 ✓ |
|
||||||
|
| ep0 | felhom-tenantsync v1.4.0, sha256 `751b3c26…` (v1.3.0 kept beside it) | `722013da` |
|
||||||
|
| controller | 0.345.0, `sha256:2dc2c0a5…` (rc.1 `1378e4e2…`) | `2619191` 2365 ✓, `6bfb784` 2368 ✓ |
|
||||||
|
| agent | — (not released) | `b1bc63d` 2363 ✓ |
|
||||||
|
|
||||||
|
Global floor 0.345.0 at 13:27:04Z (MinAgent 0.158.1), read back from the hub DB with its WAL and the hub log; the
|
||||||
|
per-box floors cleared. Every live box reports 0.345.0.
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
|
||||||
|
Before **212**, after **212**; opened **2** (R-1296 the hub drops ep0's refusal reason; R-1297 CI speed — your „both"),
|
||||||
|
closed **2** (R-1295, R-1292). R-1285 VERIFY; R-1294 and R-1291 wait for D54 / D53.
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- FILED: R-1296 — PBS's real refusal reason never reaches the hub log (found only by reading ep0's journal).
|
||||||
|
- FILED: R-1297 — felhom.eu CI ~13 min, one runner; ruled „both".
|
||||||
|
- NOT-A-FINDING: the `backup_tier_skipped` 6 min after the fresh install — before the box applied its descriptor; the hub
|
||||||
|
held the mail by design (box enrolled < 1 h); 15 min later the box took its key and made its first copy.
|
||||||
|
- NOT-A-FINDING: 9202's FileBrowser WARN on the move from 0.343.0 (twice now) — healed in 3 s, log only, every live box is
|
||||||
|
past that move.
|
||||||
|
- NOT-A-FINDING: R-1292 run 1 „FAIL" — my probe's quoting broke its reads (kept as evidence); run 2 with a fail-fast
|
||||||
|
instrument check passed.
|
||||||
|
- My mistake, fixed: removing Tester 1's test record the first time raced a path unit that restarted the controller; the
|
||||||
|
second time I stopped the path unit first.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
- **Machines:** VM 9373 (the R-1295 fresh install) and LXC 9374 (the throwaway PBS) destroyed with their disks; Tester 1's
|
||||||
|
two throwaway qBittorrents removed through the product, its rule record removed (it had none before), helper files
|
||||||
|
gone; 9202 back on 0.343.0 exactly as found (FileBrowser on `traefik-public`, no `felhom-files` network, the rc image
|
||||||
|
deleted — 9202 is not bound to the hub, so no floor moves it). Nothing of mine runs on demo-hp into Tuesday 10:00.
|
||||||
|
- **Product uninstall of VM 9373: not run** — tried root SSH with the original password (refused: day-0 replaced it with
|
||||||
|
the vaulted one) and DooPlex's key (never installed); the vault read is yours. The VM was destroyed instead.
|
||||||
|
- **Hub:** delete host `Tester4-b5ec39` (STATUS). Its first whole-guest copy stays in Tester4's ep0 namespace (a host
|
||||||
|
delete keeps the tenancy, by design).
|
||||||
|
- **ep0:** only the script swap you approved; v1.3.0 kept as `/usr/local/bin/felhom-tenantsync.v1.3.0`.
|
||||||
|
- **Secrets:** evidence scanned (the sec values + the bind token and pairing code): 0 hits.
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
# REPORT — attended release run 2026-10-24: the hub's tokens locked, the hub out of the open backup, the iperf3 line test, the watch released
|
||||||
|
|
||||||
|
Brief: „the hub's tokens locked, the hub out of the open backup, an iperf3 line test via ep0, then the upload watch
|
||||||
|
released" (operator rulings 07:22 / 07:26, `CONTEXT.md`). Read first: `05` §16.2, `06` §4.5 / §5, `07`, `08` §9, the
|
||||||
|
spike `audits/SPIKE-upload-watch-2026-10-23.md`, the felhom-build-deploy skill. Evidence: `audits/line-test-2026-10-24/`.
|
||||||
|
|
||||||
|
## Five lines
|
||||||
|
|
||||||
|
1. **Hub tokens:** sealed in hub 0.168.0 and still working (demo-hp is served the same tunnel and API tokens it runs; the
|
||||||
|
template fetch works; the next ep0 copy holds 0 of 8 token values). The live file keeps old bytes in free space — your
|
||||||
|
call: accepted, written down, with the rule „copy hub.db only with VACUUM INTO".
|
||||||
|
2. **The hub is out of the open backup:** the hub volume reads `default: disabled`; the folder holds no copy of it (your
|
||||||
|
two deletions worked; nothing else is left). Tonight's job is the last proof (dated check 2026-10-25).
|
||||||
|
3. **The line test:** demo-hp measured **94.4 Mbit/s** (Tester 1 94.3, demo-felhom 92.6) — the night backup said „at
|
||||||
|
least 10".
|
||||||
|
4. **Live where:** hub 0.169.0 and controller 0.342.0 run on all three boxes (the global floor); agent 0.164.0 on all
|
||||||
|
three (vouched); demo-felhom's qBittorrent is limited to 46 Mbit/s. Only these three boxes report to the hub.
|
||||||
|
5. **You still do:** nothing for this run. Two rows wait for a design (below).
|
||||||
|
|
||||||
|
## What I asked you, and what you said
|
||||||
|
|
||||||
|
- Deploy hub 0.168.0 → yes. Deploy hub 0.169.0 → yes. Global floor 0.342.0 → yes. Agent vouch → done.
|
||||||
|
- The live file's old bytes → accept and write down (my first estimate of „2 minutes" was wrong; the disk writes ~1.3
|
||||||
|
MB/s, so a clean-up means an 8–20 minute hub stop).
|
||||||
|
- No host has the iperf3 program → the agent speaks iperf3 itself (nothing installed on hosts).
|
||||||
|
|
||||||
|
## Versions, digests, CI (by commit)
|
||||||
|
|
||||||
|
| Repo | Commit | What | CI |
|
||||||
|
|---|---|---|---|
|
||||||
|
| felhom.eu | `00031150` | hub 0.168.0 manifest + rulings | 2275 success |
|
||||||
|
| felhom.eu | `3363ffa8` | D40: the PVC label | 2276 success |
|
||||||
|
| felhom-agent | `f7e1dcb` | the line test | 2277 success |
|
||||||
|
| felhom-controller | `ed1011b` | the tested line in the watch | 2278 success |
|
||||||
|
| felhom.eu | `1a788f33` | docs, register, ep0 unit, hub `line_test` model | 2279 success |
|
||||||
|
| felhom-controller | `42cda07` | the start-line wording | 2283 success |
|
||||||
|
| felhom-agent | `08ca678` | CHANGELOG released, REPORT | (docs) |
|
||||||
|
| felhom.eu | `01bf4e83` | hub 0.169.0 manifest | (see the final push) |
|
||||||
|
| felhom-controller | `bddc477` | CHANGELOG released, REPORT | (docs) |
|
||||||
|
|
||||||
|
- **Hub 0.168.0** `sha256:3bfdecaa…` (from `5277d859`), deployed 05:39Z. **Hub 0.169.0** `sha256:fb9b2489…` (from
|
||||||
|
`1a788f33`), deployed 07:51Z; 30-minute watch: only info events, 0 errors.
|
||||||
|
- **Agent 0.164.0**: binary `1b803d8b70f8afbe1546f6989f35b6b0e29c9ed63f6acb3843d3c04ef0d24b2b`, bundle
|
||||||
|
`1d7b54d39d0c1aff682ba0c0a7f91aea0f9407a10d2f5a8bbfdc858d32c13341`, tag `v0.164.0`; signed `agent_update` to the three
|
||||||
|
boxes, self-check 76/76; vouched (hub page + `Artifact manifest set` line).
|
||||||
|
- **Controller 0.342.0** `sha256:d002d3b7…` (from `42cda07`); rc.2 on 9202; per-box floors → healthy on all three, box
|
||||||
|
checks and https-only green; global floor 08:29:19Z, MinAgent 0.158.1, read back on the hub log and page.
|
||||||
|
- **ep0:** `felhom-iperf3.service` (iperf3 3.18-2+deb13u2, already installed since 2026-07-03), one nftables line;
|
||||||
|
no answer on the internet (`:22` as the control), an answer in the tunnel. CVEs: four fixed in this build; two open
|
||||||
|
„minor" server DoS issues bounded by tunnel-only, TCP-only and systemd caps (`scripts/ep0-iperf3/README.md`).
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
|
||||||
|
**Before 194, after 191; opened 2 (R-1249, R-1250), closed 5 (R-975, R-1244, R-1245, R-1246, R-1247).** R-1243 →
|
||||||
|
VERIFY (dated check 2026-10-25); R-1241 keeps only the homelab half.
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- The start-up VACUUM after a legacy seal cannot finish on the Longhorn disk; its comment said 2.8 s — FILED: R-1249.
|
||||||
|
- A qBittorrent that seeds all day keeps the uplink busy, so that box is never tested and gets no limit — FILED: R-1250.
|
||||||
|
- The live hub file's old token bytes — NOT-A-FINDING: accepted by the operator 2026-10-24 (`05` §16.2), every copy that
|
||||||
|
leaves is clean.
|
||||||
|
- Two pre-deploy hub snapshots in `/data/snapshots` held plain tokens — NOT-A-FINDING: they roll off (one already pruned
|
||||||
|
at 06:06Z; the other at the 12:00Z snapshot); same volume, same reader as above.
|
||||||
|
- The cross-repo wire gate refused the agent's new `line_test` field until the hub modelled it — fixed in the session
|
||||||
|
(hub 0.169.0 `sysfacts.LineTest`, the System page cell).
|
||||||
|
- My first ArgoCD sync used a wrong commit id — NOT-A-FINDING: it failed to fetch and changed nothing; redone.
|
||||||
|
- 9202 has no agent local API, so its controller cannot read a line test — NOT-A-FINDING: by design (scratch guest);
|
||||||
|
the qBittorrent proof moved to demo-felhom, where the product path runs.
|
||||||
|
- demo-hp's first watch minute could not resolve `plex` — NOT-A-FINDING: the known start-up race (2026-10-23).
|
||||||
|
- The family's line on demo-felhom's qBittorrent page was not seen live — NOT-A-FINDING: the household's password is not
|
||||||
|
CC's; the render is pinned by a test, and the text with the live value is in the evidence.
|
||||||
|
- Old rc controller images (other sessions) sit on 9202 — NOT-A-FINDING: a scratch guest; mine (rc.2) is removed.
|
||||||
|
- The hub's database is 530 MB, mostly report history (`host_reports` 305 MB, `reports` 128 MB) — NOT-A-FINDING here:
|
||||||
|
it is why a VACUUM is slow, recorded in R-1249.
|
||||||
|
|
||||||
|
## Teardown, three layers
|
||||||
|
|
||||||
|
- **Machine:** 9202 — the throwaway qBittorrent removed through the product (0 containers, 0 volumes), back on the
|
||||||
|
0.342.0 release, my rc.2 image removed. demo-hp's host — the probe binary deleted, nothing running.
|
||||||
|
- **Host / DooPlex:** the database copies I restored for the checks shredded; the 9202 password copy, the signed job
|
||||||
|
envelopes and the release clone shredded / removed. ep0 — the one planned change stays (`felhom-iperf3`); the backup
|
||||||
|
of the firewall file is `/etc/nftables.conf.bak-20261024-iperf3`.
|
||||||
|
- **Hub:** the three per-box floors cleared; the global floor 0.342.0 stays (the release).
|
||||||
|
|
||||||
|
## Instruction files, poster
|
||||||
|
|
||||||
|
No instruction file edited. System poster: a text-only change (ep0 also runs the tunnel-only line-test server) — the
|
||||||
|
facts file and the matching poster text edited in the same commit.
|
||||||
@@ -0,0 +1,67 @@
|
|||||||
|
# REPORT — every catalog app on a fresh box, the way a family uses it (2026-10-24 → 25 night, unattended)
|
||||||
|
|
||||||
|
**Plain words first.**
|
||||||
|
- 65 apps: **52 work, 10 work with a problem, 0 broken**; 2 are not offered for a new install (by design); Wanderer
|
||||||
|
cannot run on a box without DNS (known, R-739).
|
||||||
|
- **The three worst:** (1) after „remove — keep backups" and a reinstall, restoring that backup breaks the app — the
|
||||||
|
household's item is gone from view (Tandoor, AdventureLog; R-1269, P2); (2) that restore still says it worked (R-1270,
|
||||||
|
P2); (3) „use my kept data" loaded another box's newer-dated database for Paperless-ngx, Nextcloud and Immich, and
|
||||||
|
Immich never answered again (R-1271, P2). In all three the data is not deleted.
|
||||||
|
- **Fixes made in the drill catalog, not live:** n8n stops sending usage data out; OpenGist gets 256 MB (at 128 MB one
|
||||||
|
sign-in hit the limit 443 times and swapped 98 MB). Both proven on the sweep box pointed at the drill.
|
||||||
|
- **Morning:** say yes to the two catalog fixes (in Monday's release list); decide D46 (Homepage behind the family gate —
|
||||||
|
CC's pick: yes); delete hub host Tester4-aa9466.
|
||||||
|
|
||||||
|
**Safest readings taken (the brief allowed no question):** ISO 1.30.0 (the current public one, not 1.36.0); 12 GiB VM
|
||||||
|
(the brief's minimum; demo-hp had 14 free); the kept-data question exists only for drive-folder apps, so volume apps
|
||||||
|
were checked by a unit restore after the reinstall (that made a unit restore on every seeded app, not every fifth);
|
||||||
|
„Mentés most" is local-only by design; backup/storage findings filed in their own category, not „Apps & catalog". All in
|
||||||
|
the audit's head.
|
||||||
|
|
||||||
|
**Table and evidence:** `documentation/audits/catalog-sweep-2026-10-24/README.md` (the 65-row table, the findings, the
|
||||||
|
R-1269 proof, D22 put-back PASS on PrivateBin and Vikunja, the drill proofs, teardown).
|
||||||
|
|
||||||
|
**Architecture read:** `07-backup-architecture.md` §6.5 (kept data) and §8, `16-storage-physical-identity.md` §5 (the
|
||||||
|
internal place is the default drive — F3 NOT-A-FINDING), `10-localisation.md`.
|
||||||
|
|
||||||
|
**Two mistakes of this session, both caught and kept in the evidence:**
|
||||||
|
1. The first drill pass ran on the live templates: I had believed the drill repo public; the box's clone failed and the
|
||||||
|
controller kept its templates. Re-run through the box's local mirror (R-992's method) with a positive check before
|
||||||
|
any install (`DRILL-CONFIRMED`). Memory note written.
|
||||||
|
2. I reverted the OpenGist fix on a wrong reading („it grows into its limit"); an automated security review flagged the
|
||||||
|
lowered cap; a cgroup measurement proved the fix needed and it went back in.
|
||||||
|
|
||||||
|
**Commits:** drill catalog (never live) `a601950` (n8n), `fd23571` → `c420053` (wrong revert) → `405077f` + `492a17f`
|
||||||
|
(OpenGist). felhom.eu: this commit (audit + tools + evidence, register, STATUS, STATUS-HISTORY, capability map B1 +
|
||||||
|
detail). CI by hash: in the final report.
|
||||||
|
|
||||||
|
**Register:** at the sweep's start 190 (other sessions filed 18 since); before this commit 208, after 217; opened 9
|
||||||
|
(R-1269–R-1277), closed 0. Notes added to R-739 (Wanderer, no DNS) and R-1267 (anonymous volumes: OnlyOffice and RomM
|
||||||
|
too; 52 left after every app was removed with data).
|
||||||
|
|
||||||
|
**Observations:**
|
||||||
|
- FILED: R-1269 (reinstall's capture rewrites the kept unit), R-1270 (restore says ok while the app never starts),
|
||||||
|
R-1271 (kept data loads another box's future-dated database), R-1272 (Homepage has no gate — operator, D46), R-1273
|
||||||
|
(registry rate limit shown as „exit code 1"), R-1274 (restore refusal advice when tiers exist but have not copied),
|
||||||
|
R-1275 (drive names in Hungarian on English pages), R-1276 (empty parent folder after remove with data), R-1277 (box
|
||||||
|
fixtures drifted: OpenGist, Calibre-Web, Mealie, Papra, Rallly, Vaultwarden 1.37).
|
||||||
|
- FIXED (drill, morning list): n8n usage data (`a601950`); OpenGist memory (`405077f`, `492a17f`).
|
||||||
|
- NOTE on existing rows: R-739 (Wanderer), R-1267 (anonymous volumes).
|
||||||
|
- NOT-A-FINDING: F3 install default is the internal place (`16` §5, by design); F4 English guide labels (the app's own
|
||||||
|
words; the Hungarian copy is frozen by the copy gate); F5 Kimai in English (a Hungarian browser lands on `/hu/`); F6
|
||||||
|
AdventureLog 95 % (docker stats with cache; anon 75 %); database first-start lines in SparkyFitness and Dawarich (no
|
||||||
|
restart, item back); Radicale IP redirect (curl without Host); NZBGet/Transmission basic-auth prompt in screenshots (the
|
||||||
|
apps' own login); Vaultwarden stranger 200 (its own login page; sign-up closed).
|
||||||
|
|
||||||
|
**Teardown:** machine — VM 9360 and LXC 9361 (both this session's) destroyed with `--purge`, no disk left on
|
||||||
|
`nvme-scratch`; Tester 1, 9201, 9202, the thin pool and the ISO untouched; the box's logs copied off first. Host —
|
||||||
|
DooPlex: the session's secret files shredded. Hub — host Tester4-aa9466 left for the operator to delete. ep0 — listed,
|
||||||
|
nothing removed: the host's PBS link was set up, no whole-box copy went to it; the app copies went to the Tester4 restic
|
||||||
|
repository on the storage box.
|
||||||
|
|
||||||
|
**Secrets:** read by name through `sec`; the evidence was scanned for every known secret value (positive control seen) and
|
||||||
|
for token shapes; one Actual Budget session token (a throwaway app) was blanked; screenshots spot-checked (passwords sit
|
||||||
|
behind „Show", never pressed). No password, key or token is in this commit.
|
||||||
|
|
||||||
|
**System poster:** no fact on it changed. **Instruction files:** none edited. **No** release, deploy, floor, vouch,
|
||||||
|
publish or live catalog push.
|
||||||
@@ -0,0 +1,76 @@
|
|||||||
|
# REPORT — cloudflared 2026.10.3 on every box (R-1300), 2026-10-26 evening, attended
|
||||||
|
|
||||||
|
> A sibling of `REPORT.md`: the night session (`TASK-night-burndown-and-tuesday-2026-10-26.md`) owns `REPORT.md`
|
||||||
|
> tonight.
|
||||||
|
|
||||||
|
## In plain words
|
||||||
|
|
||||||
|
1. **We run 3 cloudflared, one in each box's guest** (Tester 1, demo-hp, demo-felhom). None on DooPlex. Tester4 has no
|
||||||
|
tunnel; Tester 2 has no box.
|
||||||
|
2. **The https-origin trap does not apply.** Each tunnel sends to `https://traefik` with „No TLS Verify" on, and traefik
|
||||||
|
shows its own self-signed certificate — not an eMudhra one.
|
||||||
|
3. **0.345.1 is on every box: 4 of 4 reporting controllers** (the three above + Tester4); none left behind.
|
||||||
|
4. **Longest tunnel gap: about 11 seconds** (demo-hp). Tester 1 about 2 s, demo-felhom about 4 s.
|
||||||
|
|
||||||
|
Decisions: the operator said yes to the build-gate exception (build from a pushed tag) and to the global floor.
|
||||||
|
|
||||||
|
## Versions, digests, CI
|
||||||
|
|
||||||
|
| What | Value |
|
||||||
|
|---|---|
|
||||||
|
| controller tag `v0.345.1` | commit `f3204024` = v0.345.0 `6bfb784` + one commit; CI job 2393 success (attempt 1) |
|
||||||
|
| controller `main` | `a1c65b3d` (same bump + `build.sh` tag path); CI job 2392 success (attempt 1) |
|
||||||
|
| controller image `0.345.1` / `:latest` | `sha256:7cfdf7e1f835628de9c7d536e0a60c06bad5bbe19e8133765cc0a6bb732dbff6` |
|
||||||
|
| cloudflared | `cloudflare/cloudflared:2026.10.3`, index `sha256:faa8e0a1475f137c910be6025f43c8e4092fdd35461b31e19fd1390657a1ff39` |
|
||||||
|
| MinAgent | 0.158.1 (unchanged, declared with the floor) |
|
||||||
|
| global floor | 0.345.1 at 20:08:39Z (`303 floor_set`; read back from the running hub: „Effective floor: v0.345.1 — source: DB") |
|
||||||
|
| per-box floors | set 20:05:08Z (Tester 1), 20:06:04Z (demo-hp, demo-felhom); cleared 20:39:54Z |
|
||||||
|
|
||||||
|
Tests and gates: controller gates `--fast` green on both commits; `go test ./...` green on the tag commit; the new
|
||||||
|
reconcile test and the four build-gate cases red-proved. felhom.eu gates green on push. **Methods:** box state by
|
||||||
|
`docker ps`/`docker logs` in each guest; tunnel state by the hub's Hosts page and by a public name with no app
|
||||||
|
(`cfprobe.<domain>`: 404 = tunnel up, 530 = down), probed every 0.5–1 s from DooPlex.
|
||||||
|
|
||||||
|
## Register
|
||||||
|
|
||||||
|
Rows before **213**, after **214**, opened **2** (R-1300, R-1301), closed **1** (R-1300).
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- The N100 host's clock is 14 s slow; chrony fails at every boot — **FILED: R-1301** (and `lived-in/n100.md`).
|
||||||
|
- I put a test run and a commit in one command on the detached tag clone (standing rule 1). The commit was local; the
|
||||||
|
test result was then read on its own (green) before anything was pushed — **NOT-A-FINDING: my slip, caught before
|
||||||
|
push, recorded here.**
|
||||||
|
- `noTLSVerify` on every tunnel route (R-510) — **NOT-A-FINDING: a leg that never leaves the guest** (cloudflared and
|
||||||
|
traefik alone on the /29 tunnel network); now written in `01` as a recorded exception, with the rule „never turn off
|
||||||
|
a certificate check to make an update work".
|
||||||
|
- cloudflared 2026.10.4 (today, VULN-176088: metrics debug endpoints allowlist) — **NOT-A-FINDING: our metrics bind to
|
||||||
|
the container's own loopback**; not taken because it is under a day old and the notice names 2026.10.3.
|
||||||
|
- 0.345.1 is built with 0.345.0's Go builder; `main` moved to Go 1.27.3 tonight (R-972) — **NOT-A-FINDING: R-972 owns
|
||||||
|
it; the next release from `main` carries it.**
|
||||||
|
- Single-sample `000` answers at 20:13:37 (both Tester 1 probes at once) and 20:17:13 (demo-hp) — **NOT-A-FINDING: a
|
||||||
|
client-side timeout, no 530; the hub and the boxes showed the tunnel up throughout.**
|
||||||
|
- `internal/stacks` tests took 450 s under the night session's load (each R-442 remove test 26 s) — **NOT-A-FINDING: CI
|
||||||
|
green; slow only under load.**
|
||||||
|
- Tester4's agent reads `0.167.0-night1 floor held` on the Hosts page — the night session's agent; not touched.
|
||||||
|
|
||||||
|
## What changed in documents
|
||||||
|
|
||||||
|
- `architecture/01-topology-and-trust.md` „Tunnel placement": the version, the rule, the recorded exception.
|
||||||
|
- `architecture/11-os-updates.md` C8: the new pin.
|
||||||
|
- `STATUS.md`: one line (released; `main` and Sunday's golden carry it). The golden 0.344.0 still has 2026.9.3; a new
|
||||||
|
box moves to the floor in its first minutes. No extra bake (operator did not ask).
|
||||||
|
- System poster: none of the three needed — no fact on the poster changed (cloudflared still runs in the guest).
|
||||||
|
- Controller: `build.sh` may push from the pushed tag `v<VERSION>` (operator's yes). Controller report:
|
||||||
|
`felhom-controller/REPORT-cloudflared-0.345.1.md`.
|
||||||
|
|
||||||
|
## For the operator
|
||||||
|
|
||||||
|
- **Nothing to do.** If you want a second look: Cloudflare dashboard → Networking → Tunnels shows each connector's
|
||||||
|
version (CC holds no Cloudflare token).
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
Machine: the image pulled on scratch guest 9202 for the version check was removed. Host: nothing. Hub: the three
|
||||||
|
per-box floors cleared; the global floor stays at 0.345.1 by design. Clones `felhom-controller-sec`,
|
||||||
|
`felhom-controller-mainsec` and `felhom.eu-cfsec` on DooPlex are my work copies (all pushed); removed at the end.
|
||||||
@@ -0,0 +1,103 @@
|
|||||||
|
# REPORT — day 2026-10-23: the releases and the DooPlex hour (the operator's „go")
|
||||||
|
|
||||||
|
Design read: `07-backup-architecture.md` §6.3/§8, `09-update-architecture.md` §3 decision 250, `03-host-agent.md`,
|
||||||
|
`05-hub-architecture.md` §16.3, `runbooks/total-loss-of-dooplex.md`, `audits/dooplex-survival-2026-10-22/PLAN.md`.
|
||||||
|
Evidence: `documentation/audits/day-2026-10-23/01`–`14`.
|
||||||
|
|
||||||
|
## Five lines
|
||||||
|
|
||||||
|
1. Released: controller 0.340.0 (global floor), agent 0.163.0 (vouched by the operator), installer 1.36.0, hub 0.167.0,
|
||||||
|
the website with D36's two sentences.
|
||||||
|
2. Root SSH from the LAN is key-only on all three test boxes; new installs get it from installer 1.36.0.
|
||||||
|
3. The hub database now leaves every 6 hours (restored back byte-identical); the released images and agent packages
|
||||||
|
leave nightly to ep0 (pulled back, every sha256 OK, one image loaded on a scratch box).
|
||||||
|
4. One false failure mail at 10:28 (my install of the 6-hourly copy) — filed R-1234, fixed and red-proved the same hour.
|
||||||
|
5. Register 191 → 184; the operator: print the sheet, Hosts page clean-up, D37 + D38.
|
||||||
|
|
||||||
|
## Per box — versions before → after, and the SSH checks
|
||||||
|
|
||||||
|
| Box | controller | agent | root-file bundle | `sshd -T` root | sshd reloaded | key login | felhom-sshd | swap = RAM (9201) |
|
||||||
|
|---|---|---|---|---|---|---|---|---|
|
||||||
|
| Tester 1 | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 (via 0.163.0-step1) | without-password | yes, 09:27:38 | ok | active, `sshd -t` ok | 4096/4096 |
|
||||||
|
| demo-hp | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 | without-password | yes, 09:19:44 | ok | active, `sshd -t` ok | 25898/25898 |
|
||||||
|
| demo-felhom | 0.339.0 → 0.340.0 | 0.162.0 → 0.163.0 | 0.160.0 → 0.163.0 | without-password | yes, 09:24:42 | ok | active, `sshd -t` ok | 12288/12288 |
|
||||||
|
| hub | 0.165.0 → 0.167.0 | | | | | | | |
|
||||||
|
|
||||||
|
Not proven (said plainly): the real root password over SSH (CC does not hold it — a dummy one is refused, but OpenSSH logs
|
||||||
|
only „Failed password"); a felhom-sshd connection (it admits only the operator's tunnel peer); the physical console (the
|
||||||
|
drop-in does not touch it; `passwd -S root` = P).
|
||||||
|
|
||||||
|
## Digests, sha256, CI by commit
|
||||||
|
|
||||||
|
| What | Value |
|
||||||
|
|---|---|
|
||||||
|
| controller 0.340.0 | `sha256:a8d1c6458da4ea9d667f928498656a291771e0b3756b20f2fcc8c1cf9d516621` — `caaa86e`, CI 2245 success |
|
||||||
|
| agent 0.163.0 binary / bundle | `561b5316cd511de9fa9d8b8be4553da7add684f715d3bea533ca330ff7f3763c` / `3afb88d9fbbc4adc16083fa52a98cd4974583c6378a77a1a6e0ed8ae6d1b80c8` — `1109092` CI 2246, `b4721ee` CI 2249 success |
|
||||||
|
| step bundle 0.163.0-step1 | `6cbc066902d5f3294765d5c7dc7113166c1dca25e994dbc36bfc5f139bb61fcf` |
|
||||||
|
| installer 1.36.0 (served = tag) | `de8bf5f32cbcda44c9976ba829f04ea151e3b30634b54d5fdd9eb6c4a0351cec` — `83388c08` CI 2247 success; pins `2aebf1bd` |
|
||||||
|
| hub 0.167.0 | `sha256:177860a996cb285d935777d6ffdf85725f6cf4489885f54fd3bf8135c68041a0` — built at `d306b8c2` (hub/ = `be7e97dc`, CI 2250 success; `d306b8c2` CI 2251 success); manifest `34979d5d` |
|
||||||
|
| website | `0bb57a4f` |
|
||||||
|
| D34 fix (R-1234) | `52af38df` |
|
||||||
|
| prometheus rules | homelab-manifests `1bbef892` |
|
||||||
|
|
||||||
|
CI of the last commits (`2aebf1bd`, `0bb57a4f`, `34979d5d`, `52af38df` and this close-out): see the final message.
|
||||||
|
|
||||||
|
## Parts
|
||||||
|
|
||||||
|
- **1 — controller 0.340.0:** 9202 smoke, per-customer floors, three boxes green, box walk + https-only rc 0, global floor
|
||||||
|
06:41 UTC (MinAgent 0.158.1), read back with the WAL (`01`–`06`).
|
||||||
|
- **2 — agent + installer:** release, a step bundle because the D4 drop-in is a new path (R-880), three signed jobs per
|
||||||
|
box (`07`); the operator's vouch read back from the DB and the live page (`10`); installer tag, both pins, a
|
||||||
|
webpage-only sync, the served script = the tag (`11`).
|
||||||
|
- **3 — hub 0.167.0:** built from a clean clone; on the operator's word it also carries D34's 6-hourly snapshot (asked
|
||||||
|
before the build; not soaked); hub-only sync; 30-minute watch: 0 hub events, 0 errors, all boxes reporting, no hub mail
|
||||||
|
(`14`).
|
||||||
|
- **4 — website:** D3/D5 patch + D36 HU + EN; site gates; published; read back live: new text present, old gone (control:
|
||||||
|
the previous commit), screenshots at 390 and 1366 px, no sideways scroll (`12-website-readback/`).
|
||||||
|
- **5 — D34:** retention unchanged, server-side on ep0 (`prune-operator-hubdb` 03:45, keep-daily 14, keep-weekly 8 — the
|
||||||
|
current day's up to 4 copies until the next prune). Hand run 501.6 MiB / 24.5 MiB uploaded in 4 s; the newest copy
|
||||||
|
restored from ep0 with the read-only token = the pod's snapshot by sha256 `a9ff4c8a…`, integrity ok, 4 hosts, holds
|
||||||
|
today's vouch. Alarm 13 h. Runbook updated (`13`).
|
||||||
|
- **6 — D33:** ep0 before: `/` 14 GB free (the data disk's root IS the PBS datastore — not used). One user
|
||||||
|
`felhom-relcopy` (locked, rrsync-confined: shell and `..` refused), one directory `/srv/felhom-release-copy`. Cap 3 GB,
|
||||||
|
enforced by the sender (ep0 has no quota). **skopeo, not `docker save`:** `dir:` keeps the registry's bytes, so a push
|
||||||
|
back keeps the digest; no Docker daemon. A read-only registry token of its own, minted in the Gitea pod straight into a
|
||||||
|
0600 file. First run 676 MB, 91 s; pulled back: 141 files OK, 9 images re-verified (digests match independent records),
|
||||||
|
controller 0.340.0 loaded on 9202 (ID = config digest). Alarm `FelhomReleaseCopyStale`. Restore steps in
|
||||||
|
`total-loss-of-dooplex.md` (`08`).
|
||||||
|
- **7 — D32:** the sheet is names and places only, HU + EN, one A4 (rendered: 1 page; a copy in the operator's home
|
||||||
|
folder); every place checked to exist. demo-hp: three superseded ISOs removed by name, `/` 8.5 → 14 GB free (`09`).
|
||||||
|
R-548: the box half done, the product fix proposed (R-921's pattern).
|
||||||
|
- **8 — close-out:** CONTEXT rulings, STATUS (decide rows D32–D36 → history, D37/D38 new, the do list, D35 dated
|
||||||
|
2026-10-30), REPORTs and CHANGELOGs; the system poster's facts file updated, its hub-DB label edited (text), and a new
|
||||||
|
arrow asked for in STATUS („System poster needs a refresh").
|
||||||
|
|
||||||
|
## Register — 191 → 184; opened 4, closed 11
|
||||||
|
|
||||||
|
Opened: R-1234 (closed the same hour), R-1235, R-1236, R-1237. Closed: R-49, R-982, R-985, R-991, R-1204, R-1205,
|
||||||
|
R-1225, R-1226, R-1227, R-1230, R-1234. Updated: R-1214 (released; dated check 2026-10-24), R-1187, R-1189, R-232,
|
||||||
|
R-548, R-923, R-924, R-1228 (D35, dated 2026-10-30), R-1183's check text (four snapshot slots).
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- The D34 first push refused the hub's own catch-up and mailed admin@ — FILED: R-1234 (closed, fixed `52af38df`).
|
||||||
|
- The release copy is proven to load, not to restore a registry; it follows the newest versions, not the floor/vouch; no
|
||||||
|
golden — FILED: R-1235.
|
||||||
|
- Gitea OOM-killed at its 1 GiB limit, the last at 06:41:10 UTC during the release — FILED: R-1236 (STATUS D37).
|
||||||
|
- The night sweep's mail: one-time test-box codes in one session record — FILED: R-1237.
|
||||||
|
- The hub DB snapshot is 501 MiB, not 24 MB — NOT-A-FINDING: 24 MB is the compressed upload, which is what ep0 stores.
|
||||||
|
- felhom-sshd does not answer from the LAN — NOT-A-FINDING: its firewall admits only the operator peer (H1, by design).
|
||||||
|
- A dummy root password logs „Failed password", not a policy refusal — NOT-A-FINDING: OpenSSH's behaviour under
|
||||||
|
`prohibit-password`; the policy is proven by `sshd -T` and the reload.
|
||||||
|
- The vouch left `artifact_wrapper_sha256` empty — NOT-A-FINDING: an optional field; the wrapper rides the vouched bundle.
|
||||||
|
- Guests 9202, 9401 (demo-hp) and a transient 990000 (Tester 1) have swap ≠ RAM — NOT-A-FINDING: outside the felhom pool.
|
||||||
|
- `known_hosts` holds an older key for 192.168.0.155 (demo-hp's guest address) — NOT-A-FINDING: guests are reached by
|
||||||
|
`pct exec`; nothing was changed.
|
||||||
|
- The D4 drop-in needed a step bundle — NOT-A-FINDING: the documented R-880 path for a new bundle path.
|
||||||
|
|
||||||
|
## Teardown — three layers
|
||||||
|
|
||||||
|
Machine: 9202 — the test image untagged, the tar removed. Host: demo-hp `/tmp` tar removed; three ISOs removed on purpose
|
||||||
|
(Part 7); DooPlex: the read-back copies and the hub-DB scratch restore shredded/removed. Hub: nothing provisioned; the
|
||||||
|
per-customer floors cleared after the global floor. Left on purpose: the ep0 user + directory, the DooPlex release-copy
|
||||||
|
job, the 6-hourly hub-DB timer, both alarms.
|
||||||
@@ -0,0 +1,162 @@
|
|||||||
|
# REPORT — 2026-10-28 (from midday, attended at the STOPs): the watch and Docker Hub; the four big security updates
|
||||||
|
|
||||||
|
Architecture read: `08-alarm-ladder.md` §11 (the watch); `09-update-architecture.md` §3 (decisions 255, 274, 275) and
|
||||||
|
§6.4 (the ladder); the catalog's `CLAUDE.md` (the test record) and its move gates.
|
||||||
|
|
||||||
|
## In plain words
|
||||||
|
|
||||||
|
1. **Why Docker Hub refused the watch:** speed, not volume. The night run asked 211 times in about 2 minutes from six
|
||||||
|
workers and got 15 refusals. Now the requests are spaced, and a refusal is retried. **No Docker Hub login was needed.**
|
||||||
|
2. **One CI night with 0 refusals:** not yet. Two daytime test runs had 0 refusals. Tonight's 03:40 run is the check
|
||||||
|
(R-1334 stays open until then).
|
||||||
|
3. **The four apps — all live, each a normal ladder step** (bench + box + undo proven):
|
||||||
|
- Wanderer v0.21.2
|
||||||
|
- Termix 26.10.5 (two steps: 2.8.0 → 2.9.2 → 26.10.5)
|
||||||
|
- Homepage v2.7.0
|
||||||
|
- Seerr v3.5.1 (a definition step: new image name, new start command)
|
||||||
|
4. **Wanderer is offered again: yes** (your word; D59 ended). Federation from other servers stays off (your word).
|
||||||
|
5. **What you do:** nothing today. Tomorrow morning I check that the night run had 0 Docker Hub refusals and then
|
||||||
|
close R-1334.
|
||||||
|
|
||||||
|
## Decisions (yours, at the STOPs)
|
||||||
|
|
||||||
|
- **Pushed, one STOP each:** Wanderer, Termix, Homepage, Seerr.
|
||||||
|
- **Wanderer offered again**, and **its inbound federation stays off** (`09` §3 decision 279).
|
||||||
|
- **The Docker Hub key was not asked for.** The fix without a login measured clean.
|
||||||
|
|
||||||
|
## Part 1 — R-1334 (evidence: `documentation/audits/dockerhub-bigsteps-2026-10-28/part1/`)
|
||||||
|
|
||||||
|
**The cause, measured.** Docker Hub's tag-list API (`hub.docker.com`) refused requests that came in a burst:
|
||||||
|
|
||||||
|
| Run | Requests | Time | Refused |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Night 2446 | 211 | ~2 min | 15 |
|
||||||
|
| Dry 2450 | 224 | ~3.5 min | 0 |
|
||||||
|
|
||||||
|
No scheduled job on DooPlex calls Docker Hub at that hour (k3s cron jobs and CI schedules, read only: Renovate runs
|
||||||
|
on Saturdays; the others are Longhorn backups and Postgres dumps). Other traffic from the same address was not measured.
|
||||||
|
|
||||||
|
**The fix.**
|
||||||
|
- `fe677bae`:
|
||||||
|
- Docker Hub requests are spaced: one per second across the workers.
|
||||||
|
- A 429 is retried after its `Retry-After`. The image is NOT CHECKED only after the 4th try.
|
||||||
|
- Tag lists are kept for 24 hours per registry path, on the CI runner. Two names of one image fetch once.
|
||||||
|
- Test W14: 7 checks, red first.
|
||||||
|
- `8644450d`: the image reader also retries a registry 429. ghcr.io had refused two blob reads when two dry runs came
|
||||||
|
15 minutes apart. Its retry checks were red first.
|
||||||
|
|
||||||
|
**Measured after.** Dry runs 2458 and 2459:
|
||||||
|
- 0 Docker Hub 429s and 0 retries. 210 requests over about 7 minutes.
|
||||||
|
- The only NOT CHECKED images are 6 private DooPlex ones (by design) and `minio/minio:latest`, which Docker Hub no
|
||||||
|
longer serves (noted on R-1328 for the homelab session).
|
||||||
|
- The two CI runners each keep their own tag cache. So the cache helps a re-run on the same runner. The spacing is
|
||||||
|
what fixes the night.
|
||||||
|
|
||||||
|
## Part 2 — the four big steps (evidence: `documentation/audits/dockerhub-bigsteps-2026-10-28/<app>/`)
|
||||||
|
|
||||||
|
**The path, by the rules.** No box runs any of the four apps (checked on all three boxes and on 9202). There is still
|
||||||
|
no pin move without a step: the catalog's test record and its move gates require it. `09` did not say so, so it now
|
||||||
|
has a `[DESIGN]` note. So each app is a normal ladder step. In every case below the undo was proven on 9202: the new
|
||||||
|
version, with its checks broken on purpose, ended `undone`, and the data read back on the old version.
|
||||||
|
|
||||||
|
### Wanderer v0.20.0 → v0.21.2 — catalog `1825247`
|
||||||
|
|
||||||
|
- **Template, changed at v0.20.0 first:**
|
||||||
|
- PocketBase is internal (no public address, no database-subdomain question at install). Upstream says the db must
|
||||||
|
not be reachable from outside.
|
||||||
|
- The sign-up lock now also covers `PUT /api/v1/user`. v0.20.0 already had that route, and the old lock missed it.
|
||||||
|
- No `POCKETBASE_PROXY_SECRET`, so inbound federation is rejected. There is no federation switch, and this is the
|
||||||
|
only way to close it.
|
||||||
|
- **Fixture:** a user and a private list.
|
||||||
|
- **Proof:**
|
||||||
|
- Bench: `proven`.
|
||||||
|
- Box 9202: update in 78 s.
|
||||||
|
- Undo: back to v0.20.0 in 28 s.
|
||||||
|
- **Memory:** Meilisearch fills whatever limit it gets: 256 → 384 MB, still 88.8 %. That is filed as R-1338.
|
||||||
|
|
||||||
|
### Termix 2.8.0 → 2.9.2 → 26.10.5 — catalog `e0b9594`, `589c957`
|
||||||
|
|
||||||
|
- **Two pushes**, because 2.9 must start once before 26.x.
|
||||||
|
- **Fixture:** the existing one.
|
||||||
|
- **Proof, per hop:** bench, box and undo all proven.
|
||||||
|
- **D58, measured on 9202.** Seven wrong logins, each with a different forged `X-Forwarded-For`:
|
||||||
|
- **From the LAN:** Traefik replaces the header. The 6th try got HTTP 429 on 2.8.0, 2.9.2 and 26.10.5.
|
||||||
|
- **Through the tunnel's own address** (a test container at cloudflared's address, with the chain Cloudflare
|
||||||
|
delivers): the 6th try got 429 on all three.
|
||||||
|
- So the bypass was not reachable on our boxes, before the update either. 26.10.5 adds the app's own guard
|
||||||
|
(`trust proxy: loopback`).
|
||||||
|
- **A loopback-only login route in 26.10.5:** off in our image. `NODE_ENV=production`, and nginx sets `X-Real-IP` on
|
||||||
|
every request.
|
||||||
|
|
||||||
|
### Homepage v1.13.2 → v2.7.0 — catalog `1a254d4`
|
||||||
|
|
||||||
|
- **Fixture:** a new one. It reads the dashboard's own services and bookmarks (written at first start) and plants
|
||||||
|
nothing.
|
||||||
|
- **Proof:** bench, box and undo all proven. The family gate still answers 302.
|
||||||
|
|
||||||
|
### Seerr: Jellyseerr 2.7.3 → Seerr v3.5.1 — catalog `9ef06a7`
|
||||||
|
|
||||||
|
- **A definition step:**
|
||||||
|
- The new image, plus `init: true`.
|
||||||
|
- The container starts as root only to hand `/app/config` to UID 1000. Then it runs Seerr as `node` (smoke-tested
|
||||||
|
first).
|
||||||
|
- **Fixture:** a new one. It brings a throwaway Jellyfin for Seerr's first sign-in and stores an application title.
|
||||||
|
- **Proof:**
|
||||||
|
- Bench: `proven`. Its bare abort loses the seed on the converted data, as Jellyfin did yesterday.
|
||||||
|
- Box 9202: `proven`.
|
||||||
|
- Undo: Seerr ran its migration, failed its check, and the box's undo copy brought the data back.
|
||||||
|
|
||||||
|
**Tools added in the catalog.**
|
||||||
|
- `box_walk.py --to-definition`, with 8 checks in `test_box_walk_drill_move.py`.
|
||||||
|
- The Wanderer, Homepage and Seerr fixtures.
|
||||||
|
- The bench's Wanderer override was removed, because the template does it now.
|
||||||
|
- The undo tool: `audits/dockerhub-bigsteps-2026-10-28/undo_proof2.py`. Its broken probe now uses port 1, because a
|
||||||
|
single-page app answers 200 on any path.
|
||||||
|
|
||||||
|
## Commits and CI (all green by hash)
|
||||||
|
|
||||||
|
| Repo | Commit | What | CI job |
|
||||||
|
|---|---|---|---|
|
||||||
|
| felhom.eu | `fe677bae` | Docker Hub spacing, retry, tag cache | gates 2457; dry runs 2458 and 2459 |
|
||||||
|
| felhom.eu | `8644450d`, `3c0a33c1` | image-reader retry; evidence | 2460 |
|
||||||
|
| catalog | `1825247` | Wanderer | 2461 |
|
||||||
|
| catalog | `e0b9594` | Termix hop 1 | 2462 |
|
||||||
|
| catalog | `589c957` | Termix hop 2 | 2463 |
|
||||||
|
| catalog | `1a254d4` | Homepage | 2464 |
|
||||||
|
| catalog | `9ef06a7` | Seerr | 2467 |
|
||||||
|
| felhom.eu | this close-out | docs, register, evidence | in the final check |
|
||||||
|
|
||||||
|
## Register — 229 before, 226 after
|
||||||
|
|
||||||
|
| Opened (1) | Closed (4) |
|
||||||
|
|---|---|
|
||||||
|
| R-1338 — Meilisearch memory | R-1335 — Wanderer |
|
||||||
|
| | R-1311 — Termix |
|
||||||
|
| | R-1320 — Homepage |
|
||||||
|
| | R-1317 — Seerr |
|
||||||
|
|
||||||
|
R-1334 is VERIFY; it closes after tonight. R-1328 has a note added (minio).
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
| Observation | Result |
|
||||||
|
|---|---|
|
||||||
|
| Meilisearch fills its memory limit | FILED: R-1338 |
|
||||||
|
| `minio/minio:latest` gives 401 on Docker Hub | FILED: noted on R-1328 (the homelab list) |
|
||||||
|
| Three security-review notices: „image downgrade" in the drill | NOT-A-FINDING: the drill is the test copy and must stand at FROM before a walk; the live catalog is at the new versions |
|
||||||
|
| The update accepts a container whose Docker check is still „starting" | NOT-A-FINDING: by design the declared probe decides (`09` §3); the undo tool now breaks the probe on port 1 |
|
||||||
|
| `test-record-move` judges a whole push as one move, so two hops refused together | NOT-A-FINDING: the rule works as built; Termix went out as two pushes |
|
||||||
|
| The `memory_tight` mark needs a raised limit and a new measurement | NOT-A-FINDING: the gate did its job; done for Wanderer |
|
||||||
|
| The Termix header test, first try: no gate cookie, and the shell expanded `$x` too early | NOT-A-FINDING: my test tool; fixed and re-run, and every try is kept |
|
||||||
|
| The Termix header test with a forged-only chain skipped the limit on every version | NOT-A-FINDING: Cloudflare always appends the real address; with the real chain every version locks |
|
||||||
|
| Termix 2.9.2's route dropped for a while right after the first install (502, then 404) | NOT-A-FINDING: the test now waits for health; 2.9.2 is only a stop on the way to 26.10.5 |
|
||||||
|
| Termix 26.10.5's loopback auto-login route | NOT-A-FINDING: off in production, and nginx sets `X-Real-IP` |
|
||||||
|
| Secret scan of the evidence found `"password": "wrong-"` | NOT-A-FINDING: the deliberately wrong test password, a code literal |
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
- **Machine:** 9202 is on the LIVE catalog (`repoint-9202-live.txt`, cache `9ef06a7`). It runs only its standing apps.
|
||||||
|
Every test app was removed through the product. The throwaway Jellyfin containers were removed on the bench and
|
||||||
|
on 9202.
|
||||||
|
- **Bench:** 9401 is stopped, with 0 containers.
|
||||||
|
- **Host and hub:** nothing provisioned, no change.
|
||||||
@@ -0,0 +1,111 @@
|
|||||||
|
# REPORT — two DooPlex backup alarms, every alarm the mail route delivers, and R-1250 (2026-10-24, attended)
|
||||||
|
|
||||||
|
Brief: „two DooPlex backup alarms — find why, fix, and check every other alarm the new mail route now delivers".
|
||||||
|
Read first: `runbooks/total-loss-of-dooplex.md`, `runbooks/RUNBOOK-hub-db-offsite-backup.md`,
|
||||||
|
`audits/dooplex-survival-2026-10-22/PLAN.md`, R-232, R-1227, R-1241, the two homelab rules and the Alertmanager
|
||||||
|
route, homelab HM-040…HM-049 and the window record; design `08` §9. Evidence: `audits/dooplex-alarms-2026-10-24/`.
|
||||||
|
|
||||||
|
## Three lines
|
||||||
|
|
||||||
|
1. **Both alarms had one cause, and both are fixed.** Yesterday's reboot restarted the database, and it now takes the
|
||||||
|
password of the other of its two secrets; the dump job could not log in, and the Gitea off-site push then refused
|
||||||
|
the old dump. The job now reads the right secret. Cleared: the dump alarm 10:44Z, the Gitea alarm 10:51Z.
|
||||||
|
2. **Nothing else fires now.** In 7 days only the pulled disk's two alarms fired long (silenced by you since 10-12,
|
||||||
|
ended by the pull); the rest were minutes long or are known.
|
||||||
|
3. **Fewer mails:** critical every 4 h (was 1 h), warning every 12 h (was 4 h), all backup alarms in one mail.
|
||||||
|
|
||||||
|
## What I asked you, and what you said
|
||||||
|
|
||||||
|
- The dump fix → **the CronJob only** (the database's own definition untouched).
|
||||||
|
- The mail change needs an Alertmanager restart, not a reload → **restart**.
|
||||||
|
- The R-1250 proof needed a downloader on another machine → **use DooPlex's k3s qBittorrent**.
|
||||||
|
|
||||||
|
## The alarm table (7 days, Prometheus `ALERTS`; nothing firing at 10:51Z)
|
||||||
|
|
||||||
|
| Alarm | Severity | When (UTC) | True? | Cause | Fix / row |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| `DooplexGiteaOffsiteStale` | critical | 10-24 00:57 → 10:51 | true | the push refused a 12 h old dump | the dump fix; R-232 note |
|
||||||
|
| `PostgresqlDumpStale` | warning | 10-23 22:17 → 10-24 10:44 | true | dump job login failed (28P01) | homelab `531023d` |
|
||||||
|
| `DiskOfflineUncorrectable`, `DiskUncorrectableSectors` (Z6A07P2G) | critical | whole window (≥ 145 h) → 10-23 12:02 | true | the dying HDD4 | silenced by you 10-12; pulled 10-23 — R-934 closed |
|
||||||
|
| `DiskDisappeared` (Z6A07P2G) | critical | 10-23 12:17 → 18:02 | true | the pull | expected; cleared |
|
||||||
|
| `DiskInventoryChanged` | warning | 10-23 12:17 → 20:17 | true | the pull | expected; cleared |
|
||||||
|
| `LonghornVolumeDegraded` (two Plex volumes) | warning | 10-23 08:22 → 11:22 | true | the HM-040 rebuild | cleared |
|
||||||
|
| `PodRestartingFrequently` (felhom-webpage) | warning | 10-19 09:02 → 10:07 | true | git-syncs stopped on errors | R-1163 (closed 10-19) |
|
||||||
|
| `WanUploadDegraded`, `WanDownloadDegraded`, `WanUpstreamPacketLoss`, `WanBufferbloat` | warning/info | minutes, 10-17 → 10-20 | true | short WAN dips (the flood days) | none needed |
|
||||||
|
| `AuthentikHighLatency` | warning | 10-21 08:12, 5 min | true | a short spike | none needed |
|
||||||
|
|
||||||
|
## The mail change (homelab `22d7697`, applied by a restart, read back from the running Alertmanager)
|
||||||
|
|
||||||
|
| | before | after |
|
||||||
|
|---|---|---|
|
||||||
|
| critical | every 1 h | every 4 h |
|
||||||
|
| warning | every 4 h | every 12 h |
|
||||||
|
| other (info) | every 4 h | every 12 h |
|
||||||
|
| backup alarms | one mail stream per alarm | one mail for the group, every 4 h |
|
||||||
|
| a disk | one stream per alarm | one mail per drive, daily (unchanged) |
|
||||||
|
| resolved mail | `send_resolved: true` | unchanged; the mail counter rose once after each clear (10:47, and by 10:57) — that they read „RESOLVED" is not seen: the mailbox is not mine |
|
||||||
|
|
||||||
|
## Part 4 — R-1250, committed and proven
|
||||||
|
|
||||||
|
Agent `29bcf44` + `1e087bb`, controller `046b808` (commit only; release in the next train). A seeding box: the agent
|
||||||
|
says „waiting for quiet"; the controller records the torrent app's limit, holds it at 1 KiB/s, asks for one test,
|
||||||
|
puts the exact limit back. **Proven on demo-hp with a real seed** (10 Mbit/s from 9202 to DooPlex's qBittorrent):
|
||||||
|
refused without the hold (11.1 Mbit/s); with it the uplink fell to 0.02–0.16 Mbit/s, 94.2 Mbit/s, 56.8 s, the limit
|
||||||
|
back to 1 249 280 B/s; a kill in the middle → the start-up check put it back. The brief said „a nested VM": NOT done —
|
||||||
|
a nested VM would need its own tunnel to ep0 to run the real test; demo-hp's host with probes did the same with the
|
||||||
|
real tunnel. The pinned agent client is unit-tested only.
|
||||||
|
|
||||||
|
## Commits, CI, undo
|
||||||
|
|
||||||
|
| Repo | Commit | What | CI |
|
||||||
|
|---|---|---|---|
|
||||||
|
| homelab-manifests | `531023d` | the dump CronJob's secret | no CI in this repo |
|
||||||
|
| homelab-manifests | `22d7697` | the mail routes, the dump alarm's label | no CI |
|
||||||
|
| homelab-manifests | `9f399fe` | REPORT | no CI |
|
||||||
|
| felhom-agent | `29bcf44`, `1e087bb` | R-1250 | 2286, 2288 success |
|
||||||
|
| felhom-controller | `046b808` | R-1250 | 2287 success |
|
||||||
|
| app-catalog-felhom.eu | `7de554e`, `5e196e1` | the box walk: HDD_PATH, the leftover check | 2289 success, (final push) |
|
||||||
|
| felhom-agent / felhom-controller | `5e05627` / `321df1f` | REPORT, README | (final push) |
|
||||||
|
| felhom.eu | this commit | register, STATUS, CONTEXT, `08` §9, evidence | (final push) |
|
||||||
|
|
||||||
|
Undo: `audits/dooplex-alarms-2026-10-24/README.md` § Undo. Backups of the three homelab files: `*.before` there.
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
|
||||||
|
**Before 191, after 190; opened 0, closed 1 (R-934 — the disk was already pulled).** R-232: a dated note and a dated
|
||||||
|
check (2026-10-25, tonight's sweep). R-1250: NARROWED (fixed on main, waits for the release train).
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- The database's two secrets for one role stay (only the CronJob moved) — homelab list item 1 (not a Felhom row).
|
||||||
|
- A stale dump stops the Gitea + secrets off-site push for as long as it lasts — NOT-A-FINDING: the designed fail-closed
|
||||||
|
order (`PLAN.md`: the dump first, for consistency); the alarm caught it inside its 26 h, and it now comes in one mail
|
||||||
|
with its cause.
|
||||||
|
- Alertmanager cannot reload a ConfigMap change (the brief assumed it could) — NOT-A-FINDING: the homelab manifest's
|
||||||
|
design (`envsubst` init container); you chose the restart; the fact is in CC's memory and the homelab list.
|
||||||
|
- The agent said „waiting for quiet" before it had watched a minute — fixed in the session (`1e087bb`, found live).
|
||||||
|
- The box walk sent a folder inside the drive as HDD_PATH (refused since R-839) — fixed in the session (`7de554e`).
|
||||||
|
- The box walk called the template folder a leftover after every remove — fixed in the session (`5e196e1`).
|
||||||
|
- R-934 said the failed disk was still in DooPlex; STATUS still listed R-1214's done check — fixed (row closed, line removed).
|
||||||
|
- CC's memory said Prometheus has no config-reloader; it has one since 2026-10-12 — fixed (memory note).
|
||||||
|
- `govulncheck` warns GO-2026-6809 (crypto/x509, stdlib) in the agent and the controller — NOT-A-FINDING here: the Go
|
||||||
|
toolchain row R-972 owns it.
|
||||||
|
- My Gmail connector's mailbox gets no Alertmanager mail — NOT-A-FINDING: it is another mailbox than `SMTP_TO`; 60 sent,
|
||||||
|
0 failed in 7 days. The Alertmanager config did not change yesterday (last 10-12).
|
||||||
|
- `kubectl exec -i` never closed stdin for `pg_restore` — NOT-A-FINDING: my method; the host's own `pg_restore` read the dumps.
|
||||||
|
- The 9202 ISO torrent never uploaded (behind NAT, 18 leechers / 1080 seeds) — NOT-A-FINDING: why the LAN peer was used.
|
||||||
|
- I overwrote homelab-manifests' REPORT.md, which held another session's HM-050 report from today — NOT-A-FINDING: the
|
||||||
|
repo's overwrite rule; HM-050's entry stays in its CHANGELOG.
|
||||||
|
|
||||||
|
## Teardown, three layers
|
||||||
|
|
||||||
|
- **Machine:** 9202 — qBittorrent removed through the product (no container, no volume; the template folder stays, as for
|
||||||
|
every app); the test files, the forwarder and the probe removed. DooPlex's qBittorrent — the test torrent and its
|
||||||
|
files deleted (by hash), nothing else touched.
|
||||||
|
- **Host:** demo-hp — the agent probe stopped and deleted, no iperf3 left; the installed agent untouched. DooPlex — the
|
||||||
|
9202 password copy and the helper's session files shredded; the clones removed.
|
||||||
|
- **Hub:** nothing provisioned.
|
||||||
|
|
||||||
|
## Instruction files, poster
|
||||||
|
|
||||||
|
No instruction file edited. No poster fact changed (the mail intervals are not on it).
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
# REPORT — installer 1.36.1 published; the night's fixes (2026-10-25 evening → 2026-10-26 morning)
|
||||||
|
|
||||||
|
**Plain words first.**
|
||||||
|
1. **Every package that is still named is there.** You pruned old controller, agent and golden packages by hand; nothing a
|
||||||
|
box or the installer can still ask for is missing (the agent 0.165.0 + bundle, the 0.158.1 floor agent, golden
|
||||||
|
0.331.0, the agent tags from 0.158.1 up, controller 0.343.0 / 0.342.0 / 0.331.0, samba 1.2.0, the ISO).
|
||||||
|
2. **Installer 1.36.1 is live, and its three messages were seen:** a wrong customer → „not found (404)", a wrong
|
||||||
|
passphrase → „passphrase REJECTED (401)", a customer whose box is live → „already has a live box (409)" (R-1167's
|
||||||
|
sentence, which could never show before).
|
||||||
|
3. **The agent CI is green with D50 = A** (the newest 5 versions + the MinAgent floor + the day-0 agent).
|
||||||
|
4. **R-1261 is fixed and proven before Tuesday** — on a test VM with two seeders: both held, 94.8 Mbit/s measured, both
|
||||||
|
limits back; a speed-mode switch mid-test never left 1 KiB/s. It reaches households with Monday's controller release.
|
||||||
|
5. **You decide:** Kimai on demo-hp — press Start, or remove it (STATUS, D51). And the Monday release list (STATUS).
|
||||||
|
|
||||||
|
## Each fix, before → after
|
||||||
|
|
||||||
|
- **R-1261 (controller `a79a292`):** one seeder held, a removed app's record kept forever, qBittorrent's limit put back
|
||||||
|
in whatever slot its mode showed → every seeder held, the record dropped, the held slot named. Live on the rig.
|
||||||
|
- **R-1287 (hub `f6328486`, v0.171.0):** a set-aside mailed „🔴 1 of 1 apps failed to back up" → „⚠️ … a backup of an
|
||||||
|
earlier install was set aside — nothing failed", a warning. Test on the real dispatch path.
|
||||||
|
- **R-1288 (controller `b5863a9`):** a set-aside restore's replaced unit was set aside at the NEXT backup, with a second
|
||||||
|
alarm → set aside at the restore, no alarm. Live on the rig (one set-aside mail only).
|
||||||
|
- **R-1281 (controller `7ea942a`):** an app container signed in to FileBrowser with a forged header (200) → no answer;
|
||||||
|
the household's file manager still 200; a FileBrowser left on traefik-public is moved off by the next round. Live.
|
||||||
|
- **R-1289 (hub `f6328486`):** a dev build no box ran was „the fleet's newest" → test builds never count.
|
||||||
|
- **R-1282 (controller `b5863a9`):** the 03:30 media restart was forgotten by a controller restart → stored, resumed once.
|
||||||
|
- **R-1280 (agent `d912307`):** a signed window of any length → at most 24 h (verifier, root wrapper, felhom-opsign).
|
||||||
|
- **R-1285 (agent `abcc1e6`):** a BYO host lost our SSH file one update late → the agent runs the new wrapper's policy
|
||||||
|
pass after every bundle install.
|
||||||
|
- **R-1283 (drill catalog `6733764`):** MeTube's downloads `excluded` → `optional` (a named ruling in the gate).
|
||||||
|
- **R-1286 (controller `838c834`):** Kimai's crash-loop hold read „a failed restore whose rollback also failed" → every
|
||||||
|
hold reason says what it is. The cause: the crash-loop guard during the 10-14 power drill.
|
||||||
|
|
||||||
|
## Teardown — three layers
|
||||||
|
|
||||||
|
- **Machine (rig VM 9372 on demo-hp):** the product's own `--uninstall` ran first (rc 0, „UNINSTALL complete — removed:
|
||||||
|
guest 9201, the felhom-agent …", `audits/night-2026-10-25/rig/uninstall-console.txt`); the host journal was copied off
|
||||||
|
before (`rig/rig-host-journal.txt`). Then `qm destroy 9372 --purge 1`: `qm list` shows no 9372, its folder on
|
||||||
|
`nvme-scratch` is gone, the plain-PVE ISO is removed from demo-hp, the ISO work folder on DooPlex is removed, the old
|
||||||
|
host key is removed from `known_hosts`.
|
||||||
|
- **Host (demo-hp):** guest 9202's test tracker, downloader container, test files and image are removed (0 containers,
|
||||||
|
0 files, port 6969 closed). `nvme-scratch` is a standing storage (it holds 341, 9202, 9401) — this session did not add
|
||||||
|
it, so it stays. `local-lvm` (the thin pool) was never used: 48 %.
|
||||||
|
- **Hub:** host `Tester4-84bbe7` stays on the Hosts page — a host delete is the operator's (STATUS, „Also Monday"). The
|
||||||
|
R-1285 test package was deleted from the registry (204, then 404).
|
||||||
|
- **Secrets:** the scratch session, CSRF and header files, the dashboard password file, the rig root password file, the
|
||||||
|
test bundle and the test agent build are shredded. The evidence was scanned against the four known values (through
|
||||||
|
`sec`) and four token shapes: 0 hits (one shape hit is code that reads a file). The `sec` entries
|
||||||
|
`RIG_2026_10_25_DASH_PW` and `RIG_2026_10_25_SETUP_CODE` remain; `sec` has no delete, and the box they opened is gone.
|
||||||
@@ -0,0 +1,155 @@
|
|||||||
|
# REPORT — night 2026-10-28 → 29 (unattended)
|
||||||
|
|
||||||
|
## Safest readings taken (decided by CC unattended — operator may reverse)
|
||||||
|
|
||||||
|
1. **An unused restore copy is deleted after 14 days** (R-1256 H8; `07` §6.3 [DESIGN] 2026-10-28; CONTEXT). Options:
|
||||||
|
14 days, for ever (as before), or 3 days. A held or waiting copy is never touched.
|
||||||
|
2. **H4 and H5's failed-stop branch stay on main with test proofs only.** Rule 1 asks for a live proof by morning or a
|
||||||
|
revert.
|
||||||
|
- H4 runs only on an off-site restore, and 9202 has no off-site target. A fault injected there races the write-back.
|
||||||
|
It can leave an operator-only hold (R-379), which is not ours to clear.
|
||||||
|
- H5's failed stop needs a `docker stop` that fails on demand.
|
||||||
|
- Both fixes fail safe: they hold or keep, and never pour.
|
||||||
|
- Reading: keep them, and say so first. The live proof is owed to an attended session. Or the operator reverts those
|
||||||
|
two branches of controller `342a394`.
|
||||||
|
3. **The catalog changes are two prepared patches, not a push** (the brief: drill only, no live catalog push).
|
||||||
|
`documentation/audits/night-2026-10-28/catalog-morning/`. The catalog working tree is clean again.
|
||||||
|
4. **Plex's `catalog_since` stays 2026-10-23** in the prepared patch. The ladder writer set it to the day it ran. But the
|
||||||
|
date means "the pin last moved", and the pin did not move tonight (it moved on 10-23, the one `--no-verify` push).
|
||||||
|
Put back by hand, and written in the patch README.
|
||||||
|
5. **STATUS was at its 50-line limit.** Yesterday's one-item do-list (the homelab list, R-1328) became item 5 of the
|
||||||
|
morning list. Its words did not change.
|
||||||
|
|
||||||
|
## Five lines
|
||||||
|
|
||||||
|
1. Restore edges: 4 of 4 fixed (H4, H5, H6, H8), each test red first. Proven live: 2 (H5's busy volume, H8's sweep).
|
||||||
|
2. Escrow edges: R-1255 (hub: the carry, the set-aside name, the purge) and R-1254 (the ceremony half, agent +
|
||||||
|
controller) fixed on main, tests red first. No live proof: that needs a hub deploy, and no hub deploy runs at night.
|
||||||
|
3. OOM proven: **yes** — on Tester 1, one `app_oom` "seen by docker,cgroup" and one mail.
|
||||||
|
4. Plex route: **works** — unclaimed, from Plex's own container. Proven on the bench and on box 9202 (1.41.4 → 1.43.5,
|
||||||
|
the section read back with the same uuid).
|
||||||
|
5. Claims walked: 2 walked, 2 proven, 0 failed. Not walked: 31 of 55 (was 32).
|
||||||
|
|
||||||
|
## Morning release list (each one is yours; nothing was released)
|
||||||
|
|
||||||
|
1. **Hub vX:** R-1255 (felhom.eu `1ee6d5a5`).
|
||||||
|
2. **Agent vY + your vouch:** R-1254 (`5ddd124`) and R-528 (`6f933d6`). Then `pinned.day0_agent` (D50).
|
||||||
|
3. **Controller vZ:** R-1256 (`342a394`), R-1254's controller half (`e33cffa`), R-528 (`eed23e7`), and an unknown app
|
||||||
|
answers 404 (`20c5877`). MinAgent unchanged (0.158.1): an older agent answers 404, and the controller asks again
|
||||||
|
after 10 minutes.
|
||||||
|
4. **Catalog:** the two prepared patches. R-1338 (the memory figure; Wanderer re-measured at 27.7 %, not tight), and
|
||||||
|
the Plex fixtures + the Plex ladder entry (R-1238, proven on both venues).
|
||||||
|
5. Owed to an attended session: the live proof of H4 and H5's failed stop, or a revert of those two branches.
|
||||||
|
|
||||||
|
If nothing is done: the fixes stay on main only. No box changes.
|
||||||
|
|
||||||
|
## [DESIGN] lines
|
||||||
|
|
||||||
|
- R-1256: `documentation/architecture/07-backup-architecture.md` §6.3, "[DESIGN] 2026-10-28 night — four restore
|
||||||
|
edges", plus two [FACT] paragraphs (H5, H8 live).
|
||||||
|
- R-1255: `03-host-agent.md`, "[DESIGN] 2026-10-28 night — WHICH retained row is carried".
|
||||||
|
- R-1254: `03-host-agent.md`, "[DESIGN] 2026-10-28 night — a lost or late answer".
|
||||||
|
- R-528: `08-alarm-ladder.md`, "[DESIGN] 2026-10-28 night — the host's counter beside Docker's flag", with its [FACT]
|
||||||
|
of the Tester 1 proof.
|
||||||
|
|
||||||
|
## What was proven live
|
||||||
|
|
||||||
|
- **H5, scratch 9202, 01:33 UTC:** Vikunja's first volume was held by a created-only container. The household's
|
||||||
|
restore was pressed through the page. The result was "Failed to remove volume vikunja_vikunja_data before the restore
|
||||||
|
… (not written into)". The restore failed, and the put-back poured the live copy of both volumes. The app ran, with
|
||||||
|
no hold. Projects from before and after the restore point read back.
|
||||||
|
- **H8, scratch 9202, 03:29 UTC:** three copies were planted by hand. The first sweep removed the two older than 14
|
||||||
|
days and kept the young one (8 ms). Then the young one was removed by hand.
|
||||||
|
- **R-528, Tester 1, 2026-10-28 23:27 UTC:** a Radicale capped at 128 MB. The host counter went 0 → 1, with one
|
||||||
|
`app_oom` "seen by docker,cgroup" and one operator mail.
|
||||||
|
- **R-1238 Plex box walk, 9202, 03:45–03:59 UTC:**
|
||||||
|
- Deployed through the setup gate at 1.41.4.
|
||||||
|
- A library section was seeded from Plex's own container. Its front door answers 401 to the proxy.
|
||||||
|
- The household's Update ran in 41 s.
|
||||||
|
- The section read back on 1.43.5 with the same uuid.
|
||||||
|
- The 10-minute soak peaked at 10.6 %, with no kill. The verdict was `proven`.
|
||||||
|
|
||||||
|
## Claims walked (household first; front doors only)
|
||||||
|
|
||||||
|
| Claim | Result | Evidence |
|
||||||
|
|---|---|---|
|
||||||
|
| fail.customer-self-restore | **PROVEN-LIVE** (the route). Stays `partial`: no one other than the operator has done it yet. | `audits/night-2026-10-28/part5-claims/selfrestore-walk.log` |
|
||||||
|
| fault.selfheal | **PROVEN-LIVE**, built → walked. A crashed redis ran again in 1 s, and the probe read the app healthy. A guest reboot brought everything back in under 2 minutes. No event, no mail. | `…/part5-claims/selfheal/WALK.md` |
|
||||||
|
|
||||||
|
Not walkable tonight, with the reason for each:
|
||||||
|
|
||||||
|
- backup.sikeres (the run half): needs an off-site target, and 9202 has none.
|
||||||
|
- backup.fill-warning: 9202's drive is a shared 938 GB host disk, which cannot be filled safely.
|
||||||
|
- The app-crash mail and fault.operator-email: 9202 has no hub, and a hub deploy is not a night's work.
|
||||||
|
- backup.restore-proof: needs every tier on both machines. Its block is R-366. CC's hub token reads no event list.
|
||||||
|
|
||||||
|
## Commits and CI (by commit)
|
||||||
|
|
||||||
|
| Repo | Commit | CI run id / # | Result |
|
||||||
|
|---|---|---|---|
|
||||||
|
| felhom.eu (hub) | `1ee6d5a5` R-1255 | 2470 / #1331 | green |
|
||||||
|
| felhom.eu | `8a3df79f` design notes | 2476 / #1332 | green |
|
||||||
|
| felhom.eu | `d8752de9` R-1338 evidence | 2477 / #1333 | green |
|
||||||
|
| felhom.eu | `0abed37f` + `6a6dd94d` H5, the self-restore walk, row states | 2478 / #1334 | green |
|
||||||
|
| felhom.eu | `7a6b9ff1` 07 [DESIGN] + facts, fault.selfheal walked | 2482 / #1337 | green |
|
||||||
|
| felhom.eu | close-out (this report, STATUS, CONTEXT, rows, Plex box evidence, patches) | see the end of the final report | — |
|
||||||
|
| controller | `342a394` R-1256 | 2469 / #532 | green |
|
||||||
|
| controller | `e33cffa` R-1254 | 2472 / #533 | green |
|
||||||
|
| controller | `eed23e7` R-528 | 2474 / #534 | green |
|
||||||
|
| controller | `20c5877` unknown app → 404 | 2475 / #535 | green |
|
||||||
|
| controller | `a815b37` docs | 2483 / #536 | green |
|
||||||
|
| agent | `5ddd124` R-1254 | 2471 / #283 | green |
|
||||||
|
| agent | `6f933d6` R-528 | 2473 / #284 | green |
|
||||||
|
| agent | `91bd96c` docs | 2479 / #285 | green |
|
||||||
|
| catalog | none pushed (2 prepared patches) | — | — |
|
||||||
|
| drill | `41b37ad`, `37736f0`, then their reverts `8aaf198` + `3b66d35` (the tree equals its start) | — | — |
|
||||||
|
|
||||||
|
## Teardown (three layers)
|
||||||
|
|
||||||
|
- **Machine — scratch 9202 (04:14 UTC):**
|
||||||
|
- The controller is back on 0.346.1 and healthy. The test image `0.347.0-rc.n1028` is removed (a local tag, never
|
||||||
|
pushed).
|
||||||
|
- The live catalog is back; the config is byte-identical to the saved copy.
|
||||||
|
- The drill mirror is back at its start (`320cff5d`).
|
||||||
|
- No throwaway app, no planted restore copy, no volume made tonight. The containers are as found: the controller,
|
||||||
|
the file manager, paperless (3) and traefik.
|
||||||
|
- **Machine — Tester 1 (23:30 UTC):** controller 0.346.1, agent 0.167.0 (sha256 `28bdd282…`), the test app removed.
|
||||||
|
- **Host — demo-hp:** the bench 9401 is stopped. No new guest, and nothing on the thin pool.
|
||||||
|
- **Hub:** nothing provisioned.
|
||||||
|
|
||||||
|
## Register
|
||||||
|
|
||||||
|
Before: 225. After: 226. Opened: 1 (R-1339). Closed: 0.
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- FILED: R-1339 — at boot the controller logs "drive …/userdata/<app> not mounted", but the drive IS mounted. No data
|
||||||
|
is lost; the log line misleads a diagnosis.
|
||||||
|
- NOT-A-FINDING: the Plex bench soak got 401s. An unclaimed Plex refuses the harness's request. There were 0 kills, and
|
||||||
|
the peak was 12 % of 2 GiB.
|
||||||
|
- NOT-A-FINDING: Plex's front door answers 401 to the box walk. Plex does not take the proxy's address as local. The
|
||||||
|
fixture asks from Plex's own container, which is the measured route (R-1238).
|
||||||
|
- NOT-A-FINDING: 9202 drops `app_start_failed` and `controller_started`, because it has no hub. This is by design on a
|
||||||
|
scratch box.
|
||||||
|
- NOT-A-FINDING: the ladder writer sets `catalog_since` to the day it runs. It is made for a pin move, and this was the
|
||||||
|
one entry written after a move made by hand (R-1238's history). Put back in the patch.
|
||||||
|
- NOT-A-FINDING: the security review flagged the drill's Plex downgrade. It was on purpose: drill only, scratch 9202,
|
||||||
|
and reverted.
|
||||||
|
- NOT-A-FINDING: the 03:45 push was refused once. The nightly security watch pushed its own state during the gate run.
|
||||||
|
A rebase, then the push went in.
|
||||||
|
- NOT-A-FINDING: the security watch's daily list has a new Brotli decoder hole in many images. That is its normal job,
|
||||||
|
with its own mail. A catalog image move would be a live catalog push, which the night fences.
|
||||||
|
- NOT-A-FINDING: 9202 holds 359 Docker volumes, all made before 2026-10-28 16:00. None is from tonight. This is scratch
|
||||||
|
hygiene from earlier sessions, not a product defect.
|
||||||
|
- Fixed (small): two dead bench Wanderer tokens were redacted from committed evidence. They stay in the history of
|
||||||
|
`d8752de9`; the throwaway app is gone.
|
||||||
|
- Fixed (small): a stale R-356 sentence in the self-restore claim's note.
|
||||||
|
|
||||||
|
## Other
|
||||||
|
|
||||||
|
- Instruction files: none edited.
|
||||||
|
- System poster: no listed fact changed. The restore copy is not on the poster, and nothing was released.
|
||||||
|
- Architecture read: `07-backup-architecture.md` (restore), `03-host-agent.md` (escrow), `08-alarm-ladder.md` (OOM),
|
||||||
|
`09-update-architecture.md` §6.4 (the ladder).
|
||||||
|
- Secrets: the night's passwords stayed in the scratchpad. Every evidence folder was scanned; the only hits are setting
|
||||||
|
names (`cf_tunnel_token`, `PLEX_CLAIM`, "password from settings.json"), never values.
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
# REPORT — Plex to the fixed version in the catalog and on the demo/test boxes (2026-10-23 day, Plex2Shell)
|
||||||
|
|
||||||
|
**Plain words first.**
|
||||||
|
- The catalog now gives Plex 1.43.5.11029-6bf8af3f0 (catalog `6cc317d`, pushed without the update test, on your ruling).
|
||||||
|
- One box ran Plex: demo-hp. It showed no sign of the attack. CC pressed Update; it runs the new Plex now.
|
||||||
|
- You have one decision (D39): may a security fix reach the boxes at night without the full update test?
|
||||||
|
|
||||||
|
## Per box
|
||||||
|
|
||||||
|
| Box | Plex before → after | Signs check | Box walk |
|
||||||
|
|---|---|---|---|
|
||||||
|
| demo-hp 9201 | 1.41.4.9463-630c9f557 → 1.43.5.11029-6bf8af3f0 | clean (NOT-A-FINDING; logs reach back only to 00:32 UTC today — CPU history and files cover 10-16 → 10-22) | green: all containers healthy, 0 errors, 0 warnings since the press |
|
||||||
|
| demo-hp 9202, demo-felhom 9201, tester-1 9201 | no Plex | — | — |
|
||||||
|
| Tester4 | hub host DOWN, VM gone | — | — |
|
||||||
|
| Tester 2 | no host on the hub | — | — |
|
||||||
|
|
||||||
|
Detail and evidence: `documentation/audits/plex-demo-boxes-2026-10-23/README.md`.
|
||||||
|
|
||||||
|
## Records
|
||||||
|
|
||||||
|
- Register 184 → 186: opened R-1238 (P2, Plex has no test record or fixture), R-1239 (P2, a fast lane for security
|
||||||
|
fixes — the operator's decision); closed none.
|
||||||
|
- `CONTEXT.md`: the 10:56 ruling. `STATUS.md`: decide row D39. No „broken" line: Part 2 was finished before the
|
||||||
|
records were written, so the line the brief asked for would have been added and removed in the same commit.
|
||||||
|
The „Updated" paragraph was left as the day session wrote it (it is that session's summary; this run adds one
|
||||||
|
decide row only).
|
||||||
|
- Architecture read: `09-update-architecture.md` §3 decisions 13 and 52; `13-threat-model.md` (Plex row);
|
||||||
|
`runbooks/target-selection.md`. Decision 13 is unchanged; no poster fact changed (none of the three poster actions).
|
||||||
|
- `--no-verify` was used ONCE, for catalog `6cc317d`, on the ruling. The `check-test-record-move.py` refusal is verbatim
|
||||||
|
in `app-catalog-felhom.eu/REPORT.md`.
|
||||||
|
|
||||||
|
## Fixed without a row
|
||||||
|
|
||||||
|
- The memory index (`.claude-memory/MEMORY.md`, outside git) had grown to 201 lines / 25 735 bytes at 11:09 (another
|
||||||
|
session), over the 200 / 25 600 ceiling, so `instructions_gate` failed and so did two script suites that run it.
|
||||||
|
Lines 71–72 (two memories about the deleted credentials file, both superseded by `sec`) became one line: 200 lines,
|
||||||
|
25 590 bytes. `repo_gates.py --fast` then OK. The index sits right at the ceiling; the next added line trips it again.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
Machine: Plex stays on the new version; helper scripts removed from guest and host. Host: nothing provisioned.
|
||||||
|
Hub: reads only.
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
# REPORT — did the Plex attack on DooPlex reach Felhom's secrets? (2026-10-23)
|
||||||
|
|
||||||
|
## Plain words first
|
||||||
|
|
||||||
|
1. **Verdict: NOT REACHED.** Two open doors led from the hacked Plex to Felhom's secrets. Neither was used in the attack
|
||||||
|
days: the backup folder got no new connection at all, and Longhorn made no backup or disk outside its normal plan.
|
||||||
|
2. **The shared backup folder: closed to the home network and the tailnet** (your yes). It is **still open to every app
|
||||||
|
on DooPlex** — Longhorn needs that, so the real fix is the homelab session's.
|
||||||
|
3. **What you do next:** give the homelab session the hand-over list (`documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md`,
|
||||||
|
last section). Nothing needs to be rotated.
|
||||||
|
|
||||||
|
## The Part 4 table
|
||||||
|
|
||||||
|
In `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md` § Part 4. Short form: every Felhom secret on
|
||||||
|
DooPlex has exposure **none**. The ones a hacked pod *could* have reached: the hub's unencrypted nightly copies (with the
|
||||||
|
unsealed Cloudflare/tunnel/git tokens, R-975) through the NFS export or a Longhorn backup, and the operator signing keys
|
||||||
|
through the NFS file-handle escape (documented, not exercised). Sealed hub columns need a key that is not in the copies;
|
||||||
|
escrow blobs need each household's recovery code; the k8s API, the kubelet and the hub refuse a caller with no credential
|
||||||
|
(measured).
|
||||||
|
|
||||||
|
## The paths and the evidence per path
|
||||||
|
|
||||||
|
| path | existed? | evidence of use (window 10-17 04:00Z → 10-23 08:04Z) |
|
||||||
|
|---|---|---|
|
||||||
|
| NFS export `longhorn-pvc` (`*`, `no_root_squash`, `no_subtree_check`; `secure` void in pods) | yes | **unused** — `node_nfsd_connections_total` flat at 288 for 13 days, 3902 samples, no gap; positive control: today's three probe connections moved it to 4. nfsd is TCP-only. Reads only in Longhorn's backup hours |
|
||||||
|
| Longhorn API :9500, no login | yes | **no sign of use** — 0 backups first seen off-schedule (1-min scrape); every night wrote to the local store; no unexplained volume; the backup target is the 2025-11-29 one. Manager logs: lost (reboot) |
|
||||||
|
| hub `hub:8080` direct | yes, but no secret without a credential | 0 requests from 10.42.0.52 in `/data/access-log` (from 10-18 10:25Z; positive control: the probe's IP is logged). 10-17 04:00Z → 10-18 10:25Z: no record |
|
||||||
|
| k8s API, kubelet | no (401) | — |
|
||||||
|
| SSH, Samba on DooPlex | login needed | 0 sshd lines from the pod network (control: 61 from the LAN); no Samba client file for 10.42.0.52 (positive control: the probe made one) |
|
||||||
|
| Vaultwarden | login needed; vault encrypted | logs lost (reboot) |
|
||||||
|
|
||||||
|
**Evidence lost before this session started:** the 13:20Z maintenance reboot erased the pre-reboot container logs of
|
||||||
|
longhorn-manager, longhorn-ui, the ingress, the hub, Gitea, Vaultwarden, Grafana and Prometheus. Reproduced
|
||||||
|
independently from records that survived: Prometheus history (nfsd, Longhorn, cAdvisor, node network), the hub's own
|
||||||
|
access log, the journal (boot -1), Samba per-client files, CoreDNS error lines, the Alertmanager silence list. The
|
||||||
|
incident session's pre-reboot greps (10.42.0.52 in longhorn-manager, CNPG, Gitea, authentik: 0) are cited as theirs.
|
||||||
|
|
||||||
|
## For the homelab session
|
||||||
|
|
||||||
|
The seven items at the end of the audit README: the export still open to pods (and `secure` is void there); Longhorn API
|
||||||
|
and UI with no login; an unattributed Longhorn delete of the old Plex volume at 11:24:15Z; **~1.5 TB sent out by the Plex
|
||||||
|
pod 10-18 → 10-22** (not in INCIDENT.md) plus `hand.genddos.st` DNS lookups on 10-19; the reboot that erased the logs;
|
||||||
|
the monitoring stack and Headlamp reachable from every pod; a `python3 main.py` listener on :8443. No HM numbers minted;
|
||||||
|
`homelab-manifests` not touched.
|
||||||
|
|
||||||
|
## What this session changed
|
||||||
|
|
||||||
|
- **DooPlex `/etc/exports`** (operator yes, ~15:18Z): `longhorn-pvc` from `*` to `10.42.0.0/16`, `exportfs -ra`; Longhorn
|
||||||
|
target `available: true` at 15:20:10Z. Backup `/etc/exports.bak-2026-10-23-plex2shell`. The only change on DooPlex.
|
||||||
|
- **Probe pod** `default/cc-p2s-probe`: created 15:19:46Z, removed 15:33:44Z; before/after listings in the evidence. No
|
||||||
|
other object created. The incident's NetworkPolicies untouched.
|
||||||
|
- **felhom.eu:** the audit folder; R-1241 and R-1242 in `OPEN-ITEMS.md`; `13-threat-model.md` (attacker j, one boundary
|
||||||
|
row); `STATUS.md` (verdict in „What works", the P1 in „What is broken", one „do" item; the old „Updated" paragraph moved to
|
||||||
|
`STATUS-HISTORY.md`).
|
||||||
|
- Nothing rotated. No secret value written anywhere (evidence scanned for secret shapes; the only 64-hex string is the
|
||||||
|
sha256 of `/etc/exports`).
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
|
||||||
|
Before 186, after 188; opened 2 (R-1241 P1, R-1242 P3), closed 0.
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- Hacked pod → hub backups + signing keys through the NFS export and the Longhorn API — FILED: R-1241.
|
||||||
|
- Demo NAS export on the same disk with `no_subtree_check` — FILED: R-1242.
|
||||||
|
- The hub answers pods without the geo filter — NOT-A-FINDING: the geo filter is not authentication; every no-token path refused.
|
||||||
|
- demo-felhom's Proxmox UI and SSH reachable from any DooPlex pod — NOT-A-FINDING: the door of R-982; added as attacker j in `13` §3.
|
||||||
|
- Pod logs erased by the reboot — NOT-A-FINDING for Felhom (the hub's access log persisted, R-1140); handed to the homelab session.
|
||||||
|
|
||||||
|
## Notes for the next session
|
||||||
|
|
||||||
|
- Teardown, three layers: machine — probe pod removed, export change kept on purpose (operator yes); host — none; hub —
|
||||||
|
none (read-only access-log aggregation in the running pod).
|
||||||
|
- System poster: no fact on it changed (none of the three applied).
|
||||||
|
- `unproven.py`: not run — no capability claim changed.
|
||||||
@@ -0,0 +1,106 @@
|
|||||||
|
# REPORT — the Monday release, run on Sunday 2026-10-25 (attended)
|
||||||
|
|
||||||
|
**Plain words first.**
|
||||||
|
1. **Fix A and fix B are proven.** A restore whose app does not start says so and keeps both copies (D49 = A); the old
|
||||||
|
copy goes only after 3 minutes of stable running. A `.fab` import that cannot download its version refuses and
|
||||||
|
touches nothing.
|
||||||
|
2. **Hub 0.170.0 is live.** Tester 1's recovery package went to the hub in one transaction; no alarm mail after.
|
||||||
|
3. **Agent 0.165.0 is vouched** (18:38:49 CET). It runs on all four hosts; on a BYO host our SSH file is removed — one
|
||||||
|
bundle run late (R-1285).
|
||||||
|
4. **Controller 0.343.0 is on every box:** 4 of 4 hosts (Tester 1, demo-hp, demo-felhom, the BYO test host) and the
|
||||||
|
scratch guest 9202. None stayed behind.
|
||||||
|
5. **The catalog is pushed:** n8n sends no usage data, OpenGist has 256 MiB, Homepage is behind the family gate.
|
||||||
|
|
||||||
|
**The D47 upgrade count:** Tester 1 3 of 3 apps got an install id, demo-felhom 5 of 5, demo-hp 11 of 12 (kimai is held
|
||||||
|
stopped since 2026-10-14 and the start-up step skips a held app — R-1286). **0 units set aside on any box.**
|
||||||
|
|
||||||
|
**The dated checks:** R-232 — DONE (last night's DooPlex database copy reached the restic sweep, snapshot `3973450f`,
|
||||||
|
folder `20261024-220001`, no alarm mail since 2026-10-23 22:20Z). R-1183 — partial today (one hub snapshot file per
|
||||||
|
slot, `07:00Z` and `13:00Z`; the hub restarted at 15:50Z for 0.170.0, so the morning log is gone); the full read stays
|
||||||
|
due 2026-10-26.
|
||||||
|
|
||||||
|
## What ran, in order
|
||||||
|
|
||||||
|
- **Part 1, fix A (R-1270 follow-up):** one helper, „started" = running on every sample of a 45 s window, no
|
||||||
|
`RestartCount` growth, every healthcheck `healthy`; the undo copy released after 3 more minutes (10 s samples, 30 min
|
||||||
|
watch; only a restore begun after the watch ends it). Found live and fixed on the way: docker's `-f` template failed on
|
||||||
|
every app without a healthcheck (now the JSON is read). Live on 9202 rc.6: an image that exits after 20 s → „még nem
|
||||||
|
indult el", copy kept, one operator line; the original unit back → released 3 min later.
|
||||||
|
- **Part 2, fix B (R-1279):** the D48 helper before step 1 of the import. Live: a bundle copy with a missing tag →
|
||||||
|
refused with the D48 sentence, the app running, its volumes untouched.
|
||||||
|
- **Part 3, hub 0.170.0 (R-1254):** asked → yes → manifest bump, hard refresh, sync of `Deployment/hub` alone,
|
||||||
|
Synced/Healthy, digest `sha256:6478498d…`. Tester 1 ran the escrow ceremony through the page's endpoints (the new code
|
||||||
|
stored by the operator as a CC secret); the hub logged „the new package is on the hub". No alarm mail after the
|
||||||
|
deploy (the mailbox as the second channel; a bind mail to Tester4 arrived, the control).
|
||||||
|
- **Part 4, agent 0.165.0 (R-1257, R-1250):** `release-agent.sh` from a clean clone; signed jobs to the three boxes
|
||||||
|
(76/76, the appliance drop-in in force by `sshd -T`). **No BYO host existed, so one was built** on demo-hp: VM 9371, a
|
||||||
|
plain Proxmox VE 9.2 from an answer-file ISO, Felhom installed with `--mode byo` as Tester4 (the operator deleted the
|
||||||
|
old host and stored the retrieval passphrase as a CC secret). Our 0.164.0 drop-in planted first; after 0.165.0 still
|
||||||
|
there; a second config bundle removed it, wrote nothing, `sshd -T` read the owner's own setting. Vouched by the
|
||||||
|
operator.
|
||||||
|
- **Part 5, controller 0.343.0:** rc.6/rc.7 on 9202; the set-aside alarm added and red-proved (Part 5.2); `0.343.0`
|
||||||
|
pushed after CI 2330; per-box floors → the three boxes at 17:17Z; box checks: 0 ERROR lines, all containers up,
|
||||||
|
https-only green; the D47 count above; on Tester 1 a unit restore and a set-aside restore — the item back, the copy
|
||||||
|
released after 3m0s each. Asked → yes → global floor 17:53:42Z (MinAgent 0.158.1, read back in the hub log; the BYO
|
||||||
|
host moved 18 s later); per-box floors cleared; a 30-minute watch: the hub sent 0 alarms (`global-floor-watch.txt`) and no alarm mail arrived (the mailbox as the second channel).
|
||||||
|
- **Part 6, catalog:** Homepage's family gate tried on the drill on 9202 (stranger 302/401, household 200); asked →
|
||||||
|
yes → `ebbf1f4` pushed; demo-hp read it 8 minutes later.
|
||||||
|
- **Part 7, D44/D45 = all picks:** confirm lines tagged, text-follows-code changes made (old words struck through), the
|
||||||
|
four code picks became R-1280–R-1283; R-1265 and R-1266 closed.
|
||||||
|
- **R-1250:** released, but its live proof cannot run before the agent's next due window (Tuesday 10:00–15:59 local; all
|
||||||
|
boxes measured on 2026-10-24) — a dated check on 2026-10-27 with the operator's yes for DooPlex's qBittorrent as peer.
|
||||||
|
|
||||||
|
## Versions, digests, CI by commit
|
||||||
|
|
||||||
|
- **Hub 0.170.0** `sha256:6478498d…`.
|
||||||
|
- **Agent 0.165.0** binary `e9d04a47ee85222ce6c472633ecdcebe1baa88d6836848f9a7da6435a9adb7d3`, bundle
|
||||||
|
`8dea1f34d77ba40aa44c5461c3728c2ca94bab46854a6bd8b86191f520dfe85a`; agent CI by commit: `c8d3c17` 2306 ✓, `4a1a849` 2325 ✓, `4697f8a` 2326 ✓, `6b6ba0c` 2334 ✗ (R-1290 — the registry, not the code).
|
||||||
|
- **Controller 0.343.0** `sha256:bb3da72a054c45751e3baf6e4bad43d1561a952717c5f426d65a5aeed0413a3b` from `e2a9761`.
|
||||||
|
CI: `7720439` 2319 ✓, `2149a13` 2321 ✓, `35232de` 2322 ✓, `1ee0353` 2324 ✓, `3a1f09d` 2327 ✓, `935e624` 2328 ✓,
|
||||||
|
`e2a9761` 2330 ✓, `fe98377` 2333 ✓.
|
||||||
|
- **Catalog** `ebbf1f4` 2332 ✓, `cd0cf48` (REPORT).
|
||||||
|
- **felhom.eu** `ade357a1` 2329 ✓, `36ef8dcd` 2331 ✓, this commit (CI by hash in the final message).
|
||||||
|
- **Installer 1.36.1** on main, NOT published (R-1284).
|
||||||
|
|
||||||
|
## Register
|
||||||
|
|
||||||
|
Before **219**, after **219**; opened **11** (R-1280–R-1290), closed **11** (R-1251, R-1252, R-1253, R-1269, R-1270,
|
||||||
|
R-1271, R-1279, R-1257, R-1272, R-1265, R-1266). R-1254: the hub half closed in its row, the ceremony half open. R-1250:
|
||||||
|
VERIFY with a dated check. `unproven.py --summary`: no number moved (35 of 55 not walked).
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- FILED: R-1284 — the installer's preflight printed every refusal code twice („401401"), so its 401/404/409 sentences
|
||||||
|
never showed (R-1167's 409 sentence included); fixed on main (1.36.1, red-proved), the publish is the operator's.
|
||||||
|
- FILED: R-1285 — BYO: our drop-in is removed one bundle run late (the old wrapper plans the run that installs the new one).
|
||||||
|
- FILED: R-1286 — demo-hp's kimai held since 2026-10-14, no row named it.
|
||||||
|
- FILED: R-1287 — a set-aside mails „1 of 1 apps failed to back up" (red); nothing failed.
|
||||||
|
- FILED: R-1288 — after a set-aside restore, the next backup sets the reinstall's own unit aside too (a second copy and
|
||||||
|
alarm).
|
||||||
|
- FILED: R-1289 — the hub's „fleet's newest controller" can be a dev build no box runs (`0.343.0-dev.r1251a`).
|
||||||
|
- FILED: R-1280–R-1283 — the four D44 code picks.
|
||||||
|
- FILED: R-1290 — the agent's CI is RED since ~18:00Z: four old agent packages (0.159.0–0.160.0) were removed from the
|
||||||
|
registry by something outside the repos, and the `published` gate still expects the newest 10 (`6b6ba0c`, job 2334).
|
||||||
|
Not the release's code (job 2326 on the release commit was green); the keep rule is the operator's.
|
||||||
|
- NOT-A-FINDING: the BYO host's first bundle read 71/76 (`pve:store-grant:felhom-pbs`) — the PBS storage was being added;
|
||||||
|
76/76 two seconds later and on every run after.
|
||||||
|
- NOT-A-FINDING: the `.fab` import status reads `"ok": true` beside its `error` — `ok` is the HTTP envelope, the job's
|
||||||
|
result is `error` and the steps.
|
||||||
|
- NOT-A-FINDING: 9202's alarms read „DROPPED … notifier disabled" — 9202 has no hub; the line is the positive observable
|
||||||
|
that the event was raised (one per case).
|
||||||
|
|
||||||
|
## Teardown (three layers)
|
||||||
|
|
||||||
|
- **Machine:** every throwaway app removed through the product (9202: Vikunja, Tandoor, Homepage; Tester 1: Tandoor —
|
||||||
|
no container, volume, `app.yaml` or set-aside left). 9202 back on the released 0.343.0, its drill mirror and my saved
|
||||||
|
`controller.yaml` copy removed, my `0.343.0-rc.3…7` images removed (older rc images of earlier sessions left as found).
|
||||||
|
Tester 1 keeps its new escrow package on purpose (the old recovery code no longer works; the new one is a CC secret).
|
||||||
|
- **Host:** the BYO VM uninstalled through the product's `--uninstall` (rc 0), then VM 9371 destroyed with its disk; its
|
||||||
|
ISO removed from demo-hp's `local`; demo-hp lists only VM 341 and its old ISO, as found. DooPlex: only the hub deploy;
|
||||||
|
the BYO ISO work folder deleted; the session's password, session and job files shredded.
|
||||||
|
- **Hub:** the three per-box floors cleared; the global floor 0.343.0 and the vouched manifest stay (the release). The
|
||||||
|
host record `Tester4-f967d4` stays until the operator deletes it (session only — in STATUS); customer Tester4 is kept.
|
||||||
|
|
||||||
|
**Secrets:** used only through `sec`; the evidence scanned for every stored value used (0 hits) and the BYO root
|
||||||
|
password (0 hits); one CSRF token blanked in a saved page snippet. **System poster:** no fact changed. **Instruction
|
||||||
|
files:** none edited.
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
# REPORT — 2026-10-26 night → 2026-10-27 afternoon: the night's fixes, the line test, one release block
|
||||||
|
|
||||||
|
**Safest reading taken alone:** none needed today — every release step had the operator's yes in the session.
|
||||||
|
|
||||||
|
## Plain words first
|
||||||
|
|
||||||
|
1. **The night's fixes:** all released today. Proven live: kept data (R-1220), the reinstalled box's off-site tier
|
||||||
|
(R-1232), app-file export/import (R-1187 a, b, g), held-app reports (R-1298, D53). Not live: R-1219 (handler test, as
|
||||||
|
the brief allowed) and R-1218 (the operator waived its live proof — no product path reaches the case on a hub box).
|
||||||
|
2. **The line test:** ran — but only after a same-day fix. As built, the hold was refused four times (qBittorrent keeps
|
||||||
|
sending ~14 s after it is slowed). With the fix: **94.5 Mbit/s**, the seeder's own limit back **exactly**.
|
||||||
|
3. **Released:** hub 0.173.0, agent 0.167.0 (not vouched yet), controller 0.346.0 (global floor 15:37 CET).
|
||||||
|
4. **D55 grant seen:** yes — the window ran 03:18 UTC, 223 of 223 removed (373 → 150), no alarm.
|
||||||
|
5. **The operator still does:** vouch agent 0.167.0; revoke the token `cc-d55-window`; delete hub host
|
||||||
|
`Tester4-b5dbbd`.
|
||||||
|
|
||||||
|
## Versions, digests, CI by commit
|
||||||
|
|
||||||
|
| | Version | Image / sha256 | From | CI |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| hub | 0.173.0 | `sha256:458bfb03e614db4b7abd6442e6af15c652ba4b9b4509fe2b0efe00ac979b8f52` | `29134c52` (manifest `bbc3acb8`) | 2418, 2419 ✓ |
|
||||||
|
| agent | 0.167.0 | agent `28bdd2826f45f3767880fcff4d3105afb7247c09c0ced99d78b3a978d8d826f0`, bundle `9d0344164b22634159984bfbcdaf77c53e6d2423812973ed8b4f2135a7ed1268` | `54d3a10` (tag `v0.167.0`) | 2400, 2420 ✓ |
|
||||||
|
| controller | 0.346.0 | `sha256:3d910b039fc52f08366c657774d9664320d6913fdebdd3deb2bbbb01d7eb9dd3` | `b1b9be3` (fix `710ceac`) | 2421, 2423 ✓ |
|
||||||
|
| controller rc | 0.346.0-rc.1 | `sha256:bfa091e9…` | `710ceac` | 2421 ✓ |
|
||||||
|
|
||||||
|
## Each fix, before → after
|
||||||
|
|
||||||
|
- **R-1250** (`710ceac`): four holds refused „not quiet" (4.0–4.2 Mbit/s) → held, slowed after 22 s, 94.5 Mbit/s, limit
|
||||||
|
5898240 back exactly; the 50 % rule kept the normal slot (0.15 % move). `audits/linetest-2026-10-27/`.
|
||||||
|
- **R-1232**: `due:false` (a foreign archive counted) → `due:true / absent`, first whole-box copy at once (night).
|
||||||
|
- **R-1220 / R-1187 / R-1298**: see `audits/night-fixes-2026-10-26/`.
|
||||||
|
- **R-1218** hub: a key-less upload over a keyed package → 409 (red-proved); controller: stage when the FILE exists
|
||||||
|
(red-proved). Live proof waived (`09` §3 decision 270).
|
||||||
|
- **R-1285**: the bundle pass now runs right after the commit in the normal delivery (all three boxes) — the BYO removal
|
||||||
|
itself is still unproven (stays VERIFY).
|
||||||
|
- **R-1174** (dated check): the status gate refuses since today — a hand run convicted a copy, rc 1.
|
||||||
|
|
||||||
|
## Release steps
|
||||||
|
|
||||||
|
Hub: yes 09:30 CET → build, manifest, sync of Deployment/hub only, Synced/Healthy, the digest matches, the „Held apps"
|
||||||
|
list renders, 30 min: 0 events, 0 WARN, reports flowing, no alarm mail. Agent: yes → release-agent.sh from a clean clone,
|
||||||
|
signed updates (demo-felhom 10:26, Tester 1 10:32, demo-hp 13:49 after its test). Controller: rc.1 on 9201 and Tester 1
|
||||||
|
(9202 was session 2's) → 0.346.0 → per-box floors 15:09 → all three „update successful" → box walk green (start logs 0
|
||||||
|
errors, every app healthy, https-only green, held apps 0 = 0) → yes → global floor 15:37 → 30-min watch clean → per-box
|
||||||
|
floors cleared 16:09 (DB reads empty). `audits/release-2026-10-27/`.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
Machine: the throwaway qBittorrent on 9201 removed through the product (shared library folder and older backups kept —
|
||||||
|
older than the test), forwarder and seed gone. Host: nothing provisioned on demo-hp today. DooPlex: the test torrent,
|
||||||
|
its data and folder removed, 326 torrents as before. Hub: per-box floors cleared; `Tester4-b5dbbd` (last night) waits for
|
||||||
|
the operator. Tester 1's off-site switch: the operator switched it off and on again; the box never saw it (R-1312).
|
||||||
|
|
||||||
|
## Register
|
||||||
|
|
||||||
|
Before 223 open → after 214. Closed 11 (R-1174 and R-1250, R-1261, R-1218, R-1219, R-1220, R-1232, R-1187, R-1298,
|
||||||
|
R-972, R-1291). Opened 1: R-1312 (the hub's off-site switch did not reach the box).
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- FILED R-1312 — the hub page's off-site switch does not change the box's saved target.
|
||||||
|
- NOT-A-FINDING: the line-test slot is not 10:00 — each host has its own slot minute (demo-hp 95, Tester 1 213,
|
||||||
|
demo-felhom 20), by design; the brief's 09:25 start was early, not wrong.
|
||||||
|
- NOT-A-FINDING: the off-site push WARNs „radarr/sonarr/prowlarr/qbittorrent: not deployed — unit-only" on demo-hp —
|
||||||
|
designed (`offbox_capture.go`, decision §2.4: kept units of removed apps).
|
||||||
|
- NOT-A-FINDING: the 50 % rule did not re-set qBittorrent after the new test — the line moved 0.15 %, inside its 20 % band.
|
||||||
|
- The 0.345.1 container's log was dropped when the rc replaced it; the four refusal lines were recovered from this
|
||||||
|
session's earlier reads, and the agent journal holds them independently (said in the evidence).
|
||||||
|
- The nightly CI run 2412 failed on the two due checks, by design; one operator mail at 03:24 UTC.
|
||||||
|
- A pushed commit message (`ae3db89a`) names the new row „R-1300"; the register uses R-1302 (the number was taken).
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
# REPORT — release run 2026-10-29 (attended): D62 + D63, then hub, agent, controller, catalog
|
||||||
|
|
||||||
|
## Readings taken
|
||||||
|
|
||||||
|
1. **D62's place: A** (the restore page's kept-copies list), the brief's default; no link on the app's page.
|
||||||
|
2. **D63's fault switch: a `-tags faultinject` test build** (the brief's first choice for both). Why not a read-only folder
|
||||||
|
for H4: it races the compose write. Why not a SIGTERM-ignoring container for H5: `docker stop` then kills it and
|
||||||
|
succeeds, so the stop never FAILS. The switch fails the exact step, on demand, once.
|
||||||
|
3. **H4's version split** was staged honestly: the snapshot held the catalog's 3.8.1; just before the restore the live
|
||||||
|
compose was set to the TRUE 3.8.0 (its own digest) and restarted. A first try that edited only the tag (keeping 3.8.1's
|
||||||
|
digest) was dropped — the backup's R-645 guard rightly refused it.
|
||||||
|
4. **The test boxes' leftovers:** the empty anonymous volumes this session's backup presses left (R-1267's class, 21 in all)
|
||||||
|
were removed by name; nothing was pruned.
|
||||||
|
|
||||||
|
## Five lines
|
||||||
|
|
||||||
|
1. **H4 and H5's failed stop: proven live** on Tester 1 (08:10 and 08:18 UTC). Nothing reverted.
|
||||||
|
2. **Hub 0.175.0: live** (09:39 CET); one escrow ceremony on Tester 1 end to end; 30 minutes with no unexpected alarm.
|
||||||
|
3. **Agent 0.168.0: vouched** (10:21:56 CET), on all three test boxes; the OOM line proven on the release.
|
||||||
|
4. **Controller 0.347.0: on every box** — the global floor since 10:31 CET (MinAgent 0.158.1); the three test boxes and 9202
|
||||||
|
run it.
|
||||||
|
5. **Catalog: pushed** (`313eaf6`); Tester 1 read it.
|
||||||
|
|
||||||
|
## Versions, digests, CI by commit
|
||||||
|
|
||||||
|
| Repo | Commit | CI run / # | What |
|
||||||
|
|---|---|---|---|
|
||||||
|
| felhom.eu (hub) | `19afefab` | 2485 / #1339 green | `restore_copy_deleted` allowed + operator-only (D62) |
|
||||||
|
| felhom.eu | `bf69bb91` | 2487 / #1340 green | `09` §3 decisions 281/282, `07` §6.3 [DESIGN] — **hub 0.175.0 built from here** (`sha256:642e94b6…`) |
|
||||||
|
| felhom.eu | `41f8f670` | 2488 / #1341 | D63 evidence, R-1340, `07` [FACT] |
|
||||||
|
| felhom.eu | `d6352cd8` | — | hub manifest → 0.175.0 (synced: Deployment/hub only) |
|
||||||
|
| felhom.eu | `650aa0cf` | — | release evidence, R-1341, rows moved to their sections |
|
||||||
|
| controller | `e3fce09` | 2486 / #537 green | D62 + D63 — rc.1 (`b6640c0a…`) |
|
||||||
|
| controller | `fd75c7c` | 2491 / #538 green | v0.347.0 heading — **0.347.0 built from here** (`sha256:c79b06e6…`) |
|
||||||
|
| agent | `91bd96c` | 2479 / #285 green | **0.168.0 built from here** (agent `1e70196c…d2ca`, bundle `3415c034…8cb8`, tag `v0.168.0`) |
|
||||||
|
| agent | `f4e63a5`, `6b0b093` | — | CHANGELOG; `pinned.day0_agent` = 0.168.0 |
|
||||||
|
| catalog | `3671802`, `ddf20f4`, `313eaf6` | — | R-1338, R-1238 (the night's patches), CHANGELOG/REPORT |
|
||||||
|
|
||||||
|
The close-out commits' CI runs are read at the end of the session (the final message names them).
|
||||||
|
|
||||||
|
## What was proven live (evidence: `documentation/audits/d63-2026-10-29/`)
|
||||||
|
|
||||||
|
- **H4** (test build): snapshot 3.8.1, live 3.8.0 → „comes back at the snapshot's version" → FAULT after the compose write →
|
||||||
|
„the live definition is written back before the start" → start; the app on 3.8.0, no hold, both seeds read back.
|
||||||
|
- **H5's failed stop** (test build): a busy volume „not written into"; the put-back's stop FAILED → „nothing is put back; the
|
||||||
|
copy stays …; holding the app"; the app page's hold sentence; the held copy's button off with its reason.
|
||||||
|
- **D62**: on the test build (the list line, wrong name 400, right name 200, gone; the old hub answered the event 400), and
|
||||||
|
end to end on the release build + hub 0.175.0 (the hub stored `restore_copy_deleted`).
|
||||||
|
- **Hub**: a ceremony on Tester 1 — claimed, two words typed back, „the new package is on the hub"; the hub logged the DR
|
||||||
|
bundle and the blob with the same fingerprint; the box stays `escrowed`, no stale-package warning.
|
||||||
|
- **Agent**: three signed jobs, each committed, self-check 76/76, sha256 matching; on Tester 1 a hog in a 128 MB Gokapi —
|
||||||
|
host counter 1, one `app_oom` „seen by docker,cgroup", one operator mail.
|
||||||
|
- **Controller**: walk green on all three (controller healthy, all containers up, 0 refusals, 0 errors, login and file
|
||||||
|
manager 200, https-only); a local restore of Gokapi on Tester 1 (2/2 volumes, running, front door 200).
|
||||||
|
- **Catalog**: Tester 1's sync — „catalog moved 0baf8f9 → 313eaf68", plex and wanderer updated.
|
||||||
|
|
||||||
|
## Teardown (three layers)
|
||||||
|
|
||||||
|
- **Machine — Tester 1:** throwaway Radicale and Gokapi removed through the product; 21 empty anonymous volumes removed by
|
||||||
|
name (268, as before); the empty `restore-undo` folder removed; the test image removed; controller 0.347.0, agent
|
||||||
|
0.168.0 (the releases). **Left: Radicale's restore hold record — the operator's to clear** (STATUS, the command).
|
||||||
|
Recovery code re-minted and stored (`TESTER1_RECOVERY_CODE_2026_10_29`); the 10-25 one opens only older packages.
|
||||||
|
- **Machine — scratch 9202:** controller 0.347.0 (rc.1 then the release); its old images removed; 359 volumes as before.
|
||||||
|
- **Hosts:** demo-hp, felhom-pve, Tester 1's host run agent 0.168.0. No guest made.
|
||||||
|
- **Hub:** 0.175.0; the three per-box floors cleared at 11:07 CET after the global floor's 30-minute watch; nothing else.
|
||||||
|
|
||||||
|
## Register
|
||||||
|
|
||||||
|
Before 226. After 226. Opened 4 (R-1340, R-1341, R-1342, R-1343). Closed 4 (R-1238, R-1338, R-1256, R-1255). R-528 is WATCHING (released
|
||||||
|
and proven; its dated check 2026-11-12 closes it). R-1254 stays NARROWED (released; the lost-answer branch has tests only).
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- FILED: R-1343 — the operator's hold clear at 11:42 CET was written back by the running controller's next settings
|
||||||
|
save; the CLI's restart hint does not say „inside the guest, at once" (it was run on the host, where that unit skipped
|
||||||
|
itself — no harm). The safe form is clear + restart in one command.
|
||||||
|
|
||||||
|
- FILED: R-1342 — at 10:39 CET Tester 1 mailed the operator „Gokapi did not start … the pre-restore copy is kept": false.
|
||||||
|
My test deleted the copy by hand during the 30-minute stable-start watch (a gap in D62: that watch is an operation on the
|
||||||
|
app) and then removed the app (the watch did not end). Both fixed on main with tests red first; not released (one release
|
||||||
|
per session) — it rides the next controller release.
|
||||||
|
|
||||||
|
- FILED: R-1340 — the mixed-hold operator mail says „database restore failed … rolled back … files and volumes are the
|
||||||
|
snapshot's" for any cause; in H5 nothing moved and the app has no database.
|
||||||
|
- FILED: R-1341 — the OOM WARN repeats every 30 s while the container lives (the event goes once).
|
||||||
|
- NOT-A-FINDING: the box writes Plex's compose with the ladder entry's digest — the exact-digest pin, by design.
|
||||||
|
- NOT-A-FINDING: Tester 1 reports Radicale as held after its removal — the hold record exists; the hub learns held apps by
|
||||||
|
design (D53).
|
||||||
|
- NOT-A-FINDING: the first seeds after a restart got 401 — the harness sent an empty password from a new process (the
|
||||||
|
generated login lives in the deploying process); fixed in the driver, not the product.
|
||||||
|
- NOT-A-FINDING: setup tries pushed one `backup_run_failures` event (Radicale not protected off-site) — caused by the
|
||||||
|
first, discarded version split; the R-645 guard was right.
|
||||||
|
- NOT-A-FINDING: the first agent watch read `None` for `host_id` — the key is nested; read with grep.
|
||||||
|
- NOT-A-FINDING: `GET /configuration/artifacts` with CC's token → 403 — session only, by design.
|
||||||
|
|
||||||
|
## Other
|
||||||
|
|
||||||
|
- Architecture read: `07-backup-architecture.md` §6.3/§8, `03-host-agent.md` (escrow), `05-hub-architecture.md`,
|
||||||
|
`10-localisation.md`, `09-update-architecture.md` §3.
|
||||||
|
- Instruction files: none edited. System poster: no listed fact changed (a retention age inside a box is not on it).
|
||||||
|
- Secrets: every evidence file scanned — 0 hits for the hub token, the hub key, Tester 1's password, the throwaway apps'
|
||||||
|
passwords; the recovery codes never left memory except into `sec put` (file shredded by it).
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
# REPORT — release run 2026-10-29 (afternoon, attended): restart a box's controller and clear a held app from the hub
|
||||||
|
|
||||||
|
## Five lines
|
||||||
|
|
||||||
|
1. **Restart works with the controller stopped: yes.** Stopped by hand on Tester 1, one press restarted it 5 s after
|
||||||
|
the stop. On the release builds one press read „done" on the hub 16 s after the press; the fourth press in 15
|
||||||
|
minutes was refused.
|
||||||
|
2. **Stuck check: alarm only, for now.** So far 631 probes on Tester 1 and demo-hp: median 57–61 ms, max 163 ms, no failed
|
||||||
|
probe, no 5-s timeout. The night chain is not in it yet; the log runs to 2026-10-30 07:00 UTC, then CC decides (R-1344).
|
||||||
|
3. **Clear stays after a save: yes.** On the release builds: cleared 3 s after the press, still gone after the next
|
||||||
|
settings save 6.5 minutes later; the hub's list emptied.
|
||||||
|
4. **Released:** hub 0.176.1, agent 0.169.0 (vouched), controller 0.348.0 (the global floor since 14:48 CET).
|
||||||
|
5. **What you do:** nothing is blocked. One new decide row (D65: a „Reboot guest" button — CC says not now). Tester4 is
|
||||||
|
off since 10-27; it updates when you start it.
|
||||||
|
|
||||||
|
## Readings taken
|
||||||
|
|
||||||
|
1. **The restart is the agent's, not the controller's** — a stopped or stuck controller cannot restart itself. It uses
|
||||||
|
`felhom-controller-bootstrap.service`, the same unit as the supervisor (R-523), not the host's bootstrap unit.
|
||||||
|
2. **The brake counts operator restarts with the supervisor's**, and a 4th restart in 15 minutes opens a 30-minute
|
||||||
|
refusal for presses (the supervisor's own pause is separate).
|
||||||
|
3. **Part 2 stopped at the alarm.** A day run cannot hold the night chain (the whole-guest vzdump), so the „no false fire"
|
||||||
|
condition is not shown yet. Building the restart now would introduce a mechanism nobody has measured.
|
||||||
|
4. **The clear goes through the running controller's socket**, never a file edit under a running controller — the
|
||||||
|
file edit was R-1343's cause.
|
||||||
|
|
||||||
|
## Versions, digests, CI by commit
|
||||||
|
|
||||||
|
| Repo | Commit | CI job | What |
|
||||||
|
|---|---|---|---|
|
||||||
|
| felhom.eu (hub) | `3d0c907b` | 2505 green | the two buttons, the agent channel, `controller_unresponsive`, D64 in `04`/`09` — **hub 0.176.0** (`sha256:53372198…`) |
|
||||||
|
| felhom.eu | `682015db` | 2506 green | manifest → 0.176.0 (Deployment/hub only) |
|
||||||
|
| felhom.eu (hub) | `d0dbc6f1` | 2510 green | restart button for every guest (0.176.0 gated it on a version the agent does not report); R-1345 |
|
||||||
|
| felhom.eu (hub) | `2ccf180b` | 2511 green | a token press named by its token — **hub 0.176.1** (`sha256:c105cb9a…`) |
|
||||||
|
| felhom.eu | `c1f3a163` | 2512 green | manifest → 0.176.1 |
|
||||||
|
| felhom.eu | `e4d91043` | 2516 | `03`/`05`/`08` [DESIGN], R-1346…R-1348, evidence |
|
||||||
|
| agent | `637c92c` | 2504, 2507 green | **0.169.0 built from here** (`fc066de6…`, bundle `3ca33f9c…`, tag `v0.169.0`) |
|
||||||
|
| agent | `db25eb0` | 2513 green | CHANGELOG; `pinned.day0_agent` = 0.169.0 |
|
||||||
|
| controller | `5b3b541`, `6077ac0`, `99fac06` | 2508, 2509, 2514 green | clear_hold, the socket, R-1340/R-1341, the fallback, the second R-1340 producer |
|
||||||
|
| controller | `2a0a17a` | 2515 green | v0.348.0 heading — **0.348.0 built from here** (`sha256:672fc1c1…`); rc.2 `1d96f60c…` on 9202 |
|
||||||
|
|
||||||
|
The close-out commits' CI is read at the end of the session.
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
|
||||||
|
- **Closed (4):** R-1343, R-1342, R-1340, R-1341 — each released in 0.348.0 and proven live.
|
||||||
|
- **Opened (7):** R-1344 (stuck controller: alarm; restart after the measurement), R-1345 (token scopes cannot be edited),
|
||||||
|
R-1346 (hub pages slow; the vouch's 504), R-1347 (update-held flag after a restore lift), R-1348 (restore_mixed set at
|
||||||
|
start-up sends no mail), R-1349 (guest reboot from the hub — needs your word), R-1350 (`app_start_failed` 30 s after
|
||||||
|
a clear).
|
||||||
|
- **Register:** 226 at the start of the brief → 229.
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- **F2 (an `app_start_failed` after a clear) is real, filed as R-1350.** The 12:44 clear showed none in 18 minutes; the
|
||||||
|
14:57 clear did, 30 s later — the hub's cooldown swallowed the mail. The text is true; the timing is noise.
|
||||||
|
- **A hub history line from rc.1** reads „held after an off-site restore failed its database replay" for a local restore.
|
||||||
|
That is the stored result of a 13:17 UTC clear, written before `99fac06` fixed the second producer: NOT-A-FINDING
|
||||||
|
(history, not live text).
|
||||||
|
- **Hub speed (R-1346):** the hub itself answers its pages in 0.01–0.7 s and the registry in ~0.1 s, measured in the
|
||||||
|
cluster; the 504 comes from the path in front of the hub.
|
||||||
|
- **Tester 1 tidy:** the test app Gokapi removed through the product (no container, no app file, no volume); 9
|
||||||
|
PrivateBin scratch volumes from today's backup presses removed by name (277 → 268; the night's 6 belong to R-1267);
|
||||||
|
the empty `restore-undo` folder removed. Radicale is not installed and holds nothing.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
Machine: Tester 1 as above; 9202 runs rc.2 (the scratch box is not linked to the hub, so no floor moves it; it stays there until the next test).
|
||||||
|
Host: nothing provisioned. Hub: no test customer or host made; the per-box floors cleared after the 30-minute watch.
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
`04` §6.1 [DESIGN] D64 + the agent's list; `09` §3 decision 283; `03` [DESIGN] operator restart + R-1344; `05` [DESIGN]
|
||||||
|
the two buttons; `08` table rows `controller_unresponsive`, `restore_copy_deleted`. The system poster: no fact changed
|
||||||
|
(no machine, path, backup tier or time).
|
||||||
|
|
||||||
|
Evidence: `documentation/audits/release-2026-10-29b/`.
|
||||||
@@ -0,0 +1,66 @@
|
|||||||
|
# REPORT — release run 2026-10-29 (evening, attended): „Reboot guest" from the hub (D65 = A), R-1350, R-1346
|
||||||
|
|
||||||
|
The brief was dated Friday 2026-10-30; it started Thursday 15:54 CET. Part 2 (the stuck-controller decision, R-1344)
|
||||||
|
needs the health log to 2026-10-30 07:00 UTC, so it is NOT in this run — it is the next session's first step.
|
||||||
|
|
||||||
|
## Five lines
|
||||||
|
|
||||||
|
1. **The reboot button works, and its guards refused when they should: yes.** On Tester 1, one press rebooted the guest
|
||||||
|
(up after 67 s, the controller answered after 79 s); a press during an off-site run was refused („the controller is
|
||||||
|
busy"); a second press inside the hour was refused. On the release builds: 66 s / 78 s, every app back, and the
|
||||||
|
household's dashboard said „A Felhom ügyfélszolgálata 20:05-kor újraindította a dobozt."
|
||||||
|
2. **The stuck-controller decision: not taken yet (alarm only).** The night log ends 2026-10-30 07:00 UTC. The logger
|
||||||
|
runs (1 730 lines at 18:26 UTC, not read tonight); today's planned restarts and reboots are listed in R-1344 so
|
||||||
|
tomorrow's reading leaves them out.
|
||||||
|
3. **R-1350 fixed: yes.** After the operator's hold clear the app stayed stopped and no „app is not running" alarm came
|
||||||
|
in 6.5 minutes (before: 30 s); the dead-app scan read „0 currently down".
|
||||||
|
4. **Where the vouch page's time goes: the hub's own database step of the save** — 38.1 s of 38.3 s in „fleet check +
|
||||||
|
save"; the registry took 135 ms, the page loads in 6 ms (in the cluster) / 65–255 ms (public name). ingress-nginx's
|
||||||
|
default 60-s read timeout turns a save over 60 s into the 504. Not fixed tonight (it needs one more hub release).
|
||||||
|
5. **Released:** hub 0.177.0, agent 0.170.0 (vouched), controller 0.349.0 (the global floor since 19:24 CET).
|
||||||
|
|
||||||
|
## Versions, digests, CI by commit
|
||||||
|
|
||||||
|
| Repo | Commit | CI job | What |
|
||||||
|
|---|---|---|---|
|
||||||
|
| agent | `ca15a31` | 2520, 2524 green | `reboot_guest`, its guards, the busy-file read, `GET /guest/operator-reboot` — **0.170.0 built from here** (`da8d12a0…`, bundle `03f2ff64…`, tag `v0.170.0`) |
|
||||||
|
| agent | `6578a8b` | 2525 green | v0.170.0 heading; `pinned.day0_agent` = 0.170.0 |
|
||||||
|
| controller | `b69e90b` | 2521 green | busymark, the household line, R-1350 — rc.1 `sha256:3bd08409…` |
|
||||||
|
| controller | `94049a7` | 2526 green | v0.349.0 heading — **0.349.0 built from here** (`sha256:c5341883cd58410ea631fc0c76262d0902a731a24c1096df551e7d35df1ea877`) |
|
||||||
|
| felhom.eu (hub) | `636022dc` | 2522 green | the button, the 30-min reboot expiry, the slow-request line, the vouch timing — **hub 0.177.0** (`sha256:727eeec76bf728c00f9a7e645ad26fddff31addc5bca9f8465098e264eb90ef9`) |
|
||||||
|
| felhom.eu | `8cc555be` | 2523 green | manifest → 0.177.0 (Deployment/hub only) |
|
||||||
|
|
||||||
|
The close-out commits' CI is read at the end of the session.
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
|
||||||
|
- **Before:** 229. **After:** 228. **Opened (1):** R-1351 (FileBrowser boot race, P4). **Closed (2):** R-1349 (guest
|
||||||
|
reboot from the hub), R-1350 (alarm after a hold clear). **Updated:** R-1346 (the numbers), R-1344 (planned events).
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- **FILED: R-1351** — at the release-build reboot two FileBrowser start paths ran `compose up` in the same second; one
|
||||||
|
logged two `[ERROR]` lines („container name already in use"); FileBrowser then ran healthy. Not seen on the first
|
||||||
|
reboot an hour earlier.
|
||||||
|
- **FILED: R-1346** (numbers added) — beyond the vouch: `/apps` 25–30 s, `/customers/tester-1` 7–9 s, `/system` 5.8 s in
|
||||||
|
the access log of the last days.
|
||||||
|
- **NOT-A-FINDING: the off-site run's „not deployed — offsite push is unit-only" warnings on Tester 1** — existing,
|
||||||
|
documented behaviour for apps with no recovery unit; unrelated to this run.
|
||||||
|
- **NOT-A-FINDING: the agent mutation „no swap guard" survived** — the reboot checked the swap twice (its own check and
|
||||||
|
`jobBlocker`'s). The duplicate was removed and `jobBlocker`'s check was proven by its own mutant.
|
||||||
|
- **NOT-A-FINDING: two controller mutants and one hub-side check first „failed" by not compiling** — re-made so they
|
||||||
|
compile; each then failed its test for the right reason.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
Machine: Tester 1 — the throwaway Gokapi removed through the product (no container, no app.yaml, no volume); the
|
||||||
|
controller on 0.349.0 by the floor. 9202 runs rc.1 (not linked to the hub; stays until the next test). Host: nothing
|
||||||
|
provisioned. Hub: no test customer or host; the per-box floors cleared after the 30-minute watch (clean: three boxes
|
||||||
|
healthy on 0.349.0, 0 error/held/refused lines).
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
`04` §6.1 [DESIGN] D65; `09` §3 decision 284; `03` [DESIGN] the reboot path; `05` [DESIGN] the button, the expiry, the
|
||||||
|
slow-request line; `08` [DESIGN] R-1350. The system poster: no fact changed (no machine, path, backup tier or time).
|
||||||
|
|
||||||
|
Evidence: `documentation/audits/release-2026-10-29c/`.
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
# REPORT — release run 2026-10-30 (day, attended): the stuck-controller decision, the slow hub, the site on Cloudflare
|
||||||
|
|
||||||
|
Read first: `03-host-agent.md` (the supervisor), `05-hub-architecture.md` (the store, the snapshot, the pages),
|
||||||
|
`01-topology-and-trust.md`, `runbooks/total-loss-of-dooplex.md`, `runbooks/iso-release-gate.md`,
|
||||||
|
`runbooks/publish-train-rules.md`, `audits/dooplex-survival-2026-10-22/PLAN.md` (G8); rows R-1344, R-1346, R-1228, R-1326.
|
||||||
|
|
||||||
|
## Five lines
|
||||||
|
|
||||||
|
1. **The stuck-controller decision: RESTART.** The night's log (4 733 probes, Tester 1 and demo-hp, 2026-10-29 11:12 →
|
||||||
|
10-30 07:00 UTC, through both vzdumps, the DB dumps, tier-2, the off-site push, a restore test and two updates)
|
||||||
|
had no failed probe and no 5-s timeout; the slowest was 303 ms. So agent 0.171.0 restarts a controller that runs but
|
||||||
|
does not answer at 10 unhealthy sweeps (alarm at 6) — proven on Tester 1: frozen 09:24:50 → restarted 09:31:22 UTC.
|
||||||
|
2. **The hub's slowness:** five statements read whole tables from the network disk, with 256 MiB for a 570 MB database.
|
||||||
|
Four indexes, one hint, one transaction and 1 GiB fixed it: your vouch save 38 255 ms → **132 ms**; `/apps` 25–30 s →
|
||||||
|
0.45 s (3.4 s cold); `/system` 5.8 s → 0.45 s.
|
||||||
|
3. **The site on Cloudflare: switched** at 10:02:35 UTC on your yes. preview.felhom.eu and then felhom.eu/www: 30 of 30
|
||||||
|
pages, the installer byte-identical to `installer-v1.36.1`, a preflight from a fresh ISO install passed with both.
|
||||||
|
4. **A new install still needs DooPlex for** the hub (claim, config, first report) and Gitea (the agent package, the
|
||||||
|
golden) — by design (D35). It no longer needs DooPlex to fetch the installer or show the site.
|
||||||
|
5. **What you do:** discard the unclaimed appliance (uuid `d3529d2d…`, my deleted proof VM) on the Hosts page. Nothing
|
||||||
|
else is blocked; on 2026-11-06 I ask whether to retire DooPlex's old website copy.
|
||||||
|
|
||||||
|
## Versions, digests, CI by commit
|
||||||
|
|
||||||
|
| Repo | Commit | CI job | What |
|
||||||
|
|---|---|---|---|
|
||||||
|
| agent | `bd18347` | 2532 green | R-1344 restart — **0.171.0 built from here** (sha256 `c888ca7b…202a5`, bundle `95f977f6…3d2b`, tag `v0.171.0`) |
|
||||||
|
| agent | `84726c9` | (close-out) | v0.171.0 heading; `pinned.day0_agent` = 0.171.0 (vouched 10:47:36 CET) |
|
||||||
|
| felhom.eu | `be538975` | (no own job; covered by 2533) | hub: `controller_unresponsive_restart`, the alarm text follows the agent |
|
||||||
|
| felhom.eu | `a13c69b0` | (covered by 2533) | hub 0.178.0: indexes, hint, one-transaction manifest, slow-statement line; evidence |
|
||||||
|
| felhom.eu | `f1ab634c` | 2533 green | R-1228 dated check removed — **hub 0.178.0 built from here** (`sha256:9c15c510…d925f`) |
|
||||||
|
| felhom.eu | `cea8a907` | 2535 green | manifest: hub 0.178.0, memory 1 GiB (Deployment/hub only, Synced/Healthy) |
|
||||||
|
| felhom.eu | `bd74a613`, `53107f1b`, `e0e55bf8` | read at the close-out | `api.felhom.eu` ingress; the publish script + the form endpoint; the ingress narrowed to `/api/contact` + 20 req/min |
|
||||||
|
|
||||||
|
Cloudflare: Pages project `felhom-site` (2 uploads), custom domains `preview.felhom.eu`, `felhom.eu`, `www.felhom.eu`;
|
||||||
|
DNS: `preview` and `api` added, `felhom.eu` and `www` switched (undo ids in `p3-cloudflare/switch-undo.txt`).
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
|
||||||
|
Before 228, after 228. **Closed (2):** R-1344, R-1346. **Opened (2):** R-1352 (the alarm mail arrives after the
|
||||||
|
restart mail when both fall between two reports, P4), R-1353 (statements still over 1 s, P4). **Narrowed:** R-1228
|
||||||
|
(switched; DooPlex copy until 2026-11-06, DUE-CHECKS).
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- **FILED: R-1352** — live, the hub mailed `controller_unresponsive_restart` and `controller_unresponsive` in the same
|
||||||
|
second, restart first.
|
||||||
|
- **FILED: R-1353** — the new line named a background deadline check reading days of full host-report JSON (up to
|
||||||
|
9.2 s cold), the start-up prune of `events`/`app_log_issues` (1–1.8 s), and the customer page's newest-report read.
|
||||||
|
- **NOT-A-FINDING: the first probe run slowed the live hub** (one host report took 25 s) — my own probe's random reads
|
||||||
|
on the shared Longhorn volume; stopped at once, and the measurement itself is the finding (R-1346, fixed).
|
||||||
|
- **NOT-A-FINDING: the hub was down ~2 min at the deploy** — the one-time index build (103 s) inside the 180-s startup
|
||||||
|
probe; R-1326's start-up is otherwise unchanged (the index check is four tiny queries).
|
||||||
|
- **NOT-A-FINDING: ArgoCD shows `Ingress/contact-mailer` OutOfSync** — the live object matches git field by field; not
|
||||||
|
synced (outside this run's change).
|
||||||
|
- **NOT-A-FINDING: demo-hp's local whole-guest backup skipped for space** (09:41 CET) — R-548.
|
||||||
|
- **NOT-A-FINDING: a 70-s 522 on felhom.eu at the switch** — the time Cloudflare took to activate the custom domains;
|
||||||
|
one 522 on the first full pass, then 30 of 30 twice.
|
||||||
|
- **NOT-A-FINDING: the full install (after the bind) was not run** — the bind is a session-only operator act and the
|
||||||
|
install would create Tester4's tunnel and off-site account; the preflight (both addresses) and the byte identity
|
||||||
|
carry the proof.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
Machine: VM 3461 destroyed on demo-hp, no disk left on `nvme-scratch`; the DB copies (tmpfs on DooPlex and
|
||||||
|
`/data/r1346-probe` in the hub pod) deleted, the access-log copy shredded, the probe binary removed from the pod.
|
||||||
|
Host: nothing else provisioned. Hub: one unclaimed appliance entry (uuid `d3529d2d…`) — the operator's to discard.
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
`03` [DESIGN] R-1344 restart; `05` [DESIGN] R-1346 + the restart event; `01` [DESIGN] the site on Cloudflare (row +
|
||||||
|
paragraph); `runbooks/total-loss-of-dooplex.md` (what a new install still needs); `runbooks/publish-train-rules.md` §6;
|
||||||
|
the system poster's facts (website on Cloudflare, mailer on DooPlex) — the drawing needs a refresh (STATUS line).
|
||||||
|
Instruction files corrected (stale facts, D35): `CLAUDE.md` „To publish / To roll back" (+ the upload script);
|
||||||
|
`.claude/rules/website.md` „Deploy" (git-sync is the way back, the script publishes); `skills/felhom-build-deploy`
|
||||||
|
„Website" (same). `unproven.py --summary`: not walked 31 of 55, unchanged.
|
||||||
|
|
||||||
|
Evidence: `documentation/audits/release-2026-10-30/` (`p1-r1344/`, `p2-r1346/`, `p3-cloudflare/`).
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
# REPORT — release run 2026-11-03 (attended): the Traefik fix first, then the golden, agent, hub, catalog
|
||||||
|
|
||||||
|
## Five lines
|
||||||
|
|
||||||
|
1. **Every box runs controller 0.350.0 with Traefik v3.7.17:** yes. The global floor at 08:18:16 CET; the three boxes moved at 08:05 CET on per-box floors; 9202 by hand.
|
||||||
|
2. **The load test (9202 only, from the LAN):** v3.7.16 grew from 18.7 to 79.0 MiB after 920 connections and did not release it (stopped at 11 s by the bound). v3.7.17 stayed near 25 MiB for 25,116 connections and the whole 300 s. Proven by behaviour.
|
||||||
|
3. **Golden 0.350.0 installed fresh and vouched:** yes, 10:18:10 CET. The install was the closest path, not the hub's: the public installer with one added line (the hub refuses an unvouched golden, R-297).
|
||||||
|
4. **Agent 0.172.0 vouched:** yes, same save. **Hub 0.179.0 live:** yes, 09:05 CET.
|
||||||
|
5. **Catalog pushed:** yes, `f4c5c79`; Tester 1 read it.
|
||||||
|
|
||||||
|
## Versions and digests
|
||||||
|
|
||||||
|
| What | Version | Digest / sha256 | Built from |
|
||||||
|
|---|---|---|---|
|
||||||
|
| Controller | 0.350.0 (rc.1 before it) | `9ee53908e6d95f0340c41bf4eaba954280084a8e6927de59779440e3551df261` (rc.1 `cbf4d42f…`) | `5c336d82` (rc.1 `13e34d4`) |
|
||||||
|
| Traefik | v3.7.17 | image id `a72cb2f4a2109f1ae52261c86d56b7d33a27dc5c4e211ce3c7835b772338b94f` | |
|
||||||
|
| Golden | 0.350.0 | `082e2e040de5615300da3ec8bc48fe6761ee0e6375dcc732e5cba82086e5e06b` | `build-golden.sh` v3.2.0 |
|
||||||
|
| Agent | 0.172.0 | `2c0222cd01713f65c627d6d17c18b4a8b2b1e0c39881db0df7ccb17de48e24c7`, bundle `cc34b0bc…` | `ecbb0cc` |
|
||||||
|
| Hub | 0.179.0 | `9c52c1c3427dcc493e007ec94d021802eab7dd267ea2a199639ff047b2767011` | `57d46201` |
|
||||||
|
|
||||||
|
Evidence: `documentation/audits/release-2026-11-03/` (each part has a README), `documentation/tests/golden-0.350.0-2026-11-03/`.
|
||||||
|
|
||||||
|
## Said plainly
|
||||||
|
|
||||||
|
- **My mistakes.** The first fresh install did not use the test installer: I set the variable in a place the bootstrap blanks, and I read the result only at the end. Then I told the operator a 45-minute wait would open the reinstall; it does not (R-1159), and 12 retries got the same refusal. It cost one operator act (the host delete) and about 90 minutes.
|
||||||
|
- **Scripted on purpose:** the uninstall's typed confirmation went through `ssh -tt`, on my own test VM only.
|
||||||
|
- **Proof limits.** R-1351, R-1323 and R-1352 have test proof only: their conditions did not fire. R-1347 and R-1348 were proven on the test image of the same code; the release build ran a plain restore. The file manager sign-in was not driven on the N100, which is lived-in; its anonymous page answered 200. R-1302 was staged: a plain first start on the appliance path reaches neither line.
|
||||||
|
- **ep0.** I changed nothing there by hand. The fresh installs made the hub re-issue Tester4's PBS token on ep0, twice: the product's own cascade on a customer's new host.
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- FILED (appended): R-548 — demo-hp's local backup tier is refused since 10-30, the disk has too little room.
|
||||||
|
- FILED (appended): R-1361 — the night walk's lost `db_dump_completed` became a false `expected_dbdump_missed` mail on Tester 1.
|
||||||
|
- NOT-A-FINDING: `Tester4: passphrase_keys_refused`, `host_stale`, `node_stale`, `offsite_delivery_stuck` mails — CC's own removal of a test install and its retries; the guard was right.
|
||||||
|
- NOT-A-FINDING: Traefik's repo digest on 9202 differs from the other boxes — the image id is the same on all; 9202 pulled the tag on the night of 11-02, the others this morning. The tag's index was not read.
|
||||||
|
- NOT-A-FINDING: `FELHOM_INSTALL_URL` in a unit drop-in is ignored — the bootstrap reads it only from its env file, by design.
|
||||||
|
- NOT-A-FINDING: ArgoCD's `Ingress/contact-mailer` OutOfSync — the live object matches git field by field (the same reading as 2026-10-30).
|
||||||
|
- NOT-A-FINDING: the slow-statement WARN lines on the hub — R-1353's class, already a row.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
VM 9383 destroyed with its disks; drill VM off, back on `virgin`; bench 9401 stopped; 9202 on 0.350.0, its `radicale` hold left for the operator; per-box floors cleared; DooPlex's clones and scratch secrets removed. **Hub:** host `Tester4-e286d3` waits for the operator, from about 11:00 CET.
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
# REPORT — the 2026-10-26 release: hub 0.171.0, agent 0.166.0, controller 0.344.0, catalog, golden 0.344.0
|
||||||
|
|
||||||
|
**Plain words first.**
|
||||||
|
1. **The push is unblocked.** The waiver covered one day; the three waiting felhom.eu commits went out (CI 2352 ✓ on
|
||||||
|
`7ad7c996`, which carries `b00d5c61` and `1550b1da`). The waiver is now removed: the golden gate is green on the new
|
||||||
|
golden alone.
|
||||||
|
2. **Hub 0.171.0 is live.** A set-aside now mails „⚠️ … nothing failed" as a warning (seen in your mailbox), and the
|
||||||
|
vouch line names a release version as „the fleet's newest".
|
||||||
|
3. **Agent 0.166.0 is vouched** (10:04:55 CET) and runs on the three test boxes; a 25-hour signed job is refused on a box.
|
||||||
|
4. **Controller 0.344.0 runs on every live box** (3 of 3; the global floor 09:07:47 UTC). Tester-2 has no box (R-862).
|
||||||
|
5. **Golden 0.344.0 is baked and vouched** (10:26:23 CET); a fresh install from the public ISO came up on 0.344.0 with no
|
||||||
|
update step.
|
||||||
|
|
||||||
|
## The release-build checks (one line each)
|
||||||
|
|
||||||
|
- **R-1287** — Tester 1 set-aside → mail „⚠️ tester-1: a backup of an earlier install was set aside — nothing failed",
|
||||||
|
warning (hub log + mailbox; the 10-25 mails of the same event read 🔴 „failed"). Closed.
|
||||||
|
- **R-1289** — the agent vouch logged „behind the fleet's 0.344.0" while `0.344.0-rc.3` reports sit in the same table. Closed.
|
||||||
|
- **R-1280** — demo-hp: 25 h job REJECTED, 30 min job ran; `felhom-opsign -ttl 25h` refuses. Closed.
|
||||||
|
- **R-1285** — the policy pass is wired (seen under 0.166.0), but in the normal delivery the bundle runs in the OLD agent
|
||||||
|
process, so a BYO host still loses our drop-in one release late. **Back to OPEN.**
|
||||||
|
- **R-1281** — FileBrowser only on its own networks on all three boxes; forged header from four apps → 000 (control 404/301);
|
||||||
|
household files 200, no cookie 401; OnlyOffice linked: joins `felhom-files`, editor config 200, the document fetched 200.
|
||||||
|
The file manager restarted once at the update. Closed.
|
||||||
|
- **R-1261 / R-1250** — the line hold is in 0.344.0; the live test is Tuesday 10:00–16:00 (dated check stays). R-1261 VERIFY.
|
||||||
|
- **R-1282** — „Később" survives a controller restart (start-up line + settings.json). Closed.
|
||||||
|
- **R-1278** — the install page shows the kept-backup hint and link; none before or after (Tester 1). Closed.
|
||||||
|
- **R-1286** — Kimai has its install id (07:01:55Z); the hold sentences are released. Closed.
|
||||||
|
- **R-1288** — released (proven on the night's rig). Closed.
|
||||||
|
- **R-1283** — catalog `73c165d` pushed; Tester 1 synced at once, MeTube's folder `optional`. Closed.
|
||||||
|
- **D47** — 0 set aside after the update; every app has an install id.
|
||||||
|
|
||||||
|
## Versions, digests, CI by commit
|
||||||
|
|
||||||
|
| What | Version / digest | Commit → CI job |
|
||||||
|
|---|---|---|
|
||||||
|
| hub | 0.171.0, `sha256:742d2e69ce75d0d96598e8ee9a9acb11e677a3663348571e34abbd3819c7c46e` | `7ad7c996` → 2352 ✓ (code `f6328486` → 2342 ✓); manifest `6efb4760` |
|
||||||
|
| agent | 0.166.0, binary `3f22c65b…bfea9c5c`, bundle `093772c1…5156003a3` | `aa69114` → 2351 ✓; record `b890b57` → 2356 ✓ |
|
||||||
|
| controller | 0.344.0, `sha256:9435165dab85b1200d94157ec49d8be065889139e1831a5476dc68f4b9474b8a` | `0a59044` → 2355 ✓ |
|
||||||
|
| catalog | `73c165d` | gates green locally (catalog CI runs on push) |
|
||||||
|
| golden | 0.344.0, `767a90fe07ed47c85a3e08a16b85926e30531b65608f9856aa25855b4dd53210` (637,293,448 bytes) | `documentation/tests/golden-0.344.0-2026-10-26/` |
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
|
||||||
|
Before **219**, after **212**; opened **2** (R-1294 demo-hp's off-site clean-up refused by its own cap; R-1295 PBS DR
|
||||||
|
provisioning rolled back at a re-enrolled install, log only); closed **9** (R-1278, R-1280, R-1281, R-1282, R-1283, R-1286,
|
||||||
|
R-1287, R-1288, R-1289). R-1285 back to OPEN.
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- FILED: R-1294 — demo-hp's weekly off-site clean-up refused (209 > 179) at 03:17Z; each refused week grows the next plan.
|
||||||
|
- FILED: R-1295 — at the fresh install the PBS DR auto-provision for Tester4 rolled back (`self_check_failed`); no mail.
|
||||||
|
- R-1285 reopened (delivery order), see above.
|
||||||
|
- NOT-A-FINDING: the vouch's `wrapper_sha=false` — the PBS-DR wrapper was never vouched (empty before and after).
|
||||||
|
- NOT-A-FINDING: 9202's rc start logged one WARN while FileBrowser was re-created in the same second; joined 2 s later, log
|
||||||
|
only; the release boxes logged none.
|
||||||
|
- NOT-A-FINDING: the 07:23Z self-bind mail to tester4@ — the operator's delete of `Tester4-84bbe7`, as designed.
|
||||||
|
- NOT-A-FINDING: the 02:57Z DooPlex sweep found `RIG_2026_10_25_SETUP_CODE` in last night's session record — that box is
|
||||||
|
destroyed, so the code opens nothing; the sweep worked as designed.
|
||||||
|
- NOT-A-FINDING: R-1282's status API read „LOST" on Tester 1 — the API shows `later_at` only on a box with media apps;
|
||||||
|
the stored value and the start-up line both kept it (instrument, recorded in the evidence).
|
||||||
|
- Deviation from the brief, said once: the fresh-install proof ran AFTER the golden vouch, because an install refuses any
|
||||||
|
golden but the vouched one (R-297); the vouch was reversible and the proof passed.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
- **Machines:** fresh-install VM 9373 on demo-hp destroyed with its disks (nvme-scratch); Tester 1's throwaway Tandoor (×2)
|
||||||
|
and OnlyOffice (×2) removed through the product, nothing left; 9202 back on 0.343.0 with FileBrowser on `traefik-public`
|
||||||
|
and no `felhom-files` network, the rc image deleted; the drill VM back to `virgin` (CT 9100 purged).
|
||||||
|
- **Host:** demo-hp's thin pool never used; `nvme-scratch` (standing) untouched otherwise.
|
||||||
|
- **Hub:** host `Tester4-eb7371` (the fresh-install VM) stays for the operator to delete (STATUS); per-box floors cleared.
|
||||||
|
- **Secrets:** bake token/user, bind token, pairing code shredded; the pairing code blacked out in the committed
|
||||||
|
screenshot; evidence scanned (6 sec values + the 3 one-time values: 0 hits). New sec entry `REL_2026_10_26_ROOTPW`
|
||||||
|
(the test VM's typed root password, replaced at day-0; the VM is gone).
|
||||||
|
- One test side effect: at 2026-10-27 03:30 the „Később" timer fires on Tester 1, which has no media app — it restarts nothing.
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
# REPORT — the restore and backup holes, D43 = A, and the D44/D45 picks (2026-10-24 night, unattended)
|
||||||
|
|
||||||
|
**Plain words first.**
|
||||||
|
- A power cut during a restore of an app on a drive no longer brings old data back later (R-1251, the P1). Proven on
|
||||||
|
Tester 1 with a real power-off and a drive that came back late.
|
||||||
|
- One wrong line in an app's backup settings no longer deletes its second-drive copy (R-1252). Proven on 9202.
|
||||||
|
- An export no longer overwrites a running restore's record (R-1253, proven on 9202); the hub saves a recovery package
|
||||||
|
in one step (R-1254, tested, not deployed).
|
||||||
|
- BYO hosts keep their own SSH settings (your D43 = A, R-1257). Tested in the bundle's harness.
|
||||||
|
- The review's 32 notes have one line each with a pick: answer „all picks" or name the ones you disagree with.
|
||||||
|
- Nothing is released. Monday: controller v0.343.0, agent v0.165.0 + your vouch, hub v0.170.0.
|
||||||
|
|
||||||
|
**Evidence and tables:** `documentation/audits/restore-holes-2026-10-24/README.md`; the picks:
|
||||||
|
`documentation/audits/review-round-2-2026-10-24/D44-D45-picks.md`.
|
||||||
|
|
||||||
|
**Commits:** controller `bb2809d`, `062f9b4`, `518a135`, `c9b4b82`, `e1613ae`; felhom.eu `dcf4f84c` (hub) and this
|
||||||
|
commit (documents, register, STATUS, CONTEXT, evidence); felhom-agent `c8d3c17`, `4a1a849`. CI by hash: in the final
|
||||||
|
report.
|
||||||
|
|
||||||
|
**Register:** before 207, after 208; opened 1 (R-1268), closed 0. R-1251/R-1252/R-1253/R-1257 → VERIFY (fixed on main,
|
||||||
|
waits for the release); R-1254 → NARROWED (the hub half fixed); R-1265/R-1266 carry the picks; R-1267 extended.
|
||||||
|
|
||||||
|
**Documents edited:** `07` (the R-1251 rule after D22's paragraph; the R-1252 rule in §8.2a; a pointer in §6.5), `05`
|
||||||
|
(the escrow transaction), `03` and `11` §1 (D43 = A). **System poster:** no fact on it changed (none of the four).
|
||||||
|
**Instruction files:** none edited.
|
||||||
|
|
||||||
|
**Machines:** Tester 1 and 9202 put back as found except leftover empty anonymous volumes of the R-1267 class (named in
|
||||||
|
the audit). demo-hp: nothing provisioned; the catalog sweep's VM untouched. DooPlex, ep0, household boxes, the hub: not
|
||||||
|
changed. No release, deploy, vouch, publish or catalog push.
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
# REPORT — a backup is never changed by a new install, a restore downloads first, a restore tells the truth (2026-10-25, unattended, commit only)
|
||||||
|
|
||||||
|
**Plain words first.**
|
||||||
|
- **R-1269 is fixed and proven:** a reinstall no longer writes into the old backup. The old backup is set aside whole,
|
||||||
|
listed on the restore page, and restores with its own password (Tandoor and AdventureLog on 9202: the account back).
|
||||||
|
- **D48 download-first on every restore path that starts a definition (the unit restore and with it the second
|
||||||
|
drive's unit and whole restore, both kept-data loads and a set-aside unit; the off-site restore; the second drive's
|
||||||
|
file restore; the no-unit fallback).** An unpullable version is refused and the app is never stopped (proven).
|
||||||
|
- **R-1270:** a restore whose app does not start says so — never „it worked"; the old copy is kept (proven).
|
||||||
|
- **R-1271:** cause — the other box's off-site copy was dated in the future (its clock), and „newest wins" took it;
|
||||||
|
now a copy more than 10 minutes in the future never counts as the newest (proven on a forged copy).
|
||||||
|
- **One question for you:** D49 in STATUS — after a restore whose app does not start, keep the data and wait (CC's
|
||||||
|
pick, done) or pour the old data back by itself.
|
||||||
|
|
||||||
|
**Safest readings taken** (also at the head of `documentation/audits/restore-truth-2026-10-25/README.md`): an install
|
||||||
|
identity field, not the data stamps; the set-aside unit is a D25 kept copy (remove-with-backups deletes it, as it does
|
||||||
|
other kept copies); a „partial write" means one half into another install's unit; R-1270 keeps the data and the undo
|
||||||
|
copy (no new retention rule); D48 tells a deleted version from a busy registry by Docker's answer; 10 minutes of clock
|
||||||
|
skew; the `.fab` import left to its own row.
|
||||||
|
|
||||||
|
**Architecture read:** `07` §6.3, §6.5, §6.6, §8 (D20/D22, §8.2a D25); `03` (drive identity); `09` §3 decisions 14,
|
||||||
|
40, 53; rows R-1269–R-1271, R-1206, R-698, R-933, R-1267.
|
||||||
|
|
||||||
|
**Each proof, before (`e1613ae`, 9202 on 0.342.0) → after (9202 on 0.343.0-rc.2):**
|
||||||
|
- R-1269: the kept unit's `app.yaml` rewritten (`cbfcb538…` → `e19d2610…`), restore broken → set aside byte for byte,
|
||||||
|
restored, the account back, 0 password errors (Tandoor, AdventureLog).
|
||||||
|
- R-1270: `ok: true` over a `starting` app → `ok: false`, `not_started`, the honest sentence, the undo copy kept.
|
||||||
|
- D48: the first pull after the stop (code order) → an unpullable pin refused, the container's start time unchanged.
|
||||||
|
- R-1271: the future copy chosen (sweep) → the own copy chosen; alone, the future copy marked and said.
|
||||||
|
|
||||||
|
**Commits.** Controller: `02daebe` (D47) CI 2311 ✓, `b680db8` (R-1270) 2312 ✓, `2d71270` (D48) 2313 ✓, `bd9f4f7`
|
||||||
|
(R-1271) 2314 ✓, `1f9c863` (the kept list without off-site) 2315 ✓, `1f82959` (remove deletes the undo copy) 2316 ✓,
|
||||||
|
`e8d2151` (REPORT). felhom.eu: this commit (`07` §6.5/§6.6/§8.2d/§8.2e, `09` §3 decisions 256–257, CONTEXT, register,
|
||||||
|
STATUS, STATUS-HISTORY, the audit with its evidence and tools) — CI by hash in the final message. Test images
|
||||||
|
`0.343.0-rc.1` (`bd9f4f7`) and `0.343.0-rc.2` (`1f9c863`) in the registry; nothing released.
|
||||||
|
|
||||||
|
**Teardown.** Machine: 9202 only; my throwaway apps removed through the product; two empty folders and one undo copy of
|
||||||
|
mine removed by hand after checking; my rc images removed from its Docker; back on 0.342.0; the apps found running still
|
||||||
|
run. Host: the session's 9202 password and session files shredded. Hub: provisioned nothing. ep0: nothing.
|
||||||
|
|
||||||
|
**Register:** before 217, after 219; opened 2 (R-1278, R-1279), closed 0; R-1269, R-1270, R-1271 → VERIFY (fixed on
|
||||||
|
main, wait for the release); notes on R-1206 (cross-reference) and R-1243 (its dated check done, read-only); R-232's
|
||||||
|
dated check moved one day (this brief: no DooPlex work).
|
||||||
|
|
||||||
|
**Observations:**
|
||||||
|
- FILED: R-1278 (a volume app's reinstall does not offer its kept backup).
|
||||||
|
- FILED: R-1279 (the `.fab` import removes the app's volumes before its pull — D48 does not cover it).
|
||||||
|
- FIXED without a row (found live, red-proved): the kept-copies list was hidden on a box without off-site (`1f9c863`);
|
||||||
|
a kept undo copy outlived remove-with-backups (`1f82959`).
|
||||||
|
- NOT-A-FINDING: Immich's 404 after the sweep's load — the copy brought its own address (`photos.`), as every unit
|
||||||
|
restore does; the app was healthy there.
|
||||||
|
- NOT-A-FINDING: „immich: definition REFUSED by the allow-list" on 9202 — present before this run's image; an earlier
|
||||||
|
session's Immich, not this change.
|
||||||
|
- NOT-A-FINDING: the D48 refusal reads „Visszaállítás sikertelen: …nem lehet letölteni… Az alkalmazáshoz nem
|
||||||
|
nyúltunk" — the page's failure frame around an honest sentence; true as written.
|
||||||
|
|
||||||
|
**Secrets:** used only through `sec`; the evidence scanned for the box password and the session tokens (the CSRF token
|
||||||
|
in two saved page snippets was the positive control and is blanked); units compared by sha256 only. **System poster:**
|
||||||
|
no fact changed. **Instruction files:** none edited.
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# REPORT — review round 2 (2026-10-24, unattended): the documents against the code, data loss and security
|
||||||
|
|
||||||
|
**Plain words first.**
|
||||||
|
- The 17 design documents were checked against the code: 493 claims, 121 out of date. About 130 places are corrected or
|
||||||
|
marked with a dated note; no rule, fence or promise was changed.
|
||||||
|
- About 30 things the code does that no document named are now written down for you to confirm (D44), and seven places
|
||||||
|
where a document's ruling or promise says something the code does not do are waiting for you (D45).
|
||||||
|
- The worst data-loss finding: a power cut during a restore of an app on a drive can later pour old data over the
|
||||||
|
household's new data (R-1251, P1). Proven with the real recovery code; it needs a fix in an attended run.
|
||||||
|
- Also proven, on the scratch box: one wrong line in an app's backup settings deletes that app's second-drive copy
|
||||||
|
(R-1252, P2).
|
||||||
|
- The worst security finding: the „root logs in by key only" SSH rule also reaches a host the household owns (BYO),
|
||||||
|
where we promised to change nothing (R-1257, D43). Nothing serious was found in the hub's or the controller's new routes.
|
||||||
|
|
||||||
|
**Evidence and the full tables:** `documentation/audits/review-round-2-2026-10-24/README.md`; every document edit with
|
||||||
|
its `file:line` in `EDITS.md`; the reviewers' candidates in `reviewers/`; the two proofs in `proofs/`.
|
||||||
|
|
||||||
|
**Commits (CI read by hash):** controller `b01bb4b` (two small fixes, red-proved; unreleased) — CI 2294 / #480 success;
|
||||||
|
controller `2449223` (REUSE.md) — 2298 / #481 success; felhom-agent `f5c3d77` — 2296 / #261 success and catalog
|
||||||
|
`227a544` — 2297 / #256 success (the rule-file alignment); felhom.eu `77015997` (documents batch 1) — 2295 / #1253
|
||||||
|
success; felhom.eu `876d9236` (batch 2, the register, STATUS, CONTEXT) — 2299 / #1254 success.
|
||||||
|
|
||||||
|
**Register:** before 190, after 207; opened 17 (R-1251–R-1267), closed 0.
|
||||||
|
|
||||||
|
**Instruction files edited (facts only, named here):** `.claude/rules/unprompted-work.md` in felhom-agent, the catalog
|
||||||
|
and the workspace root — the missing „A version number is pushed once" paragraph (before 95 lines, md5 3b071ad2; after
|
||||||
|
byte-identical to the other two, b1891bd4); `felhom.eu/.claude/rules/live-probes.md:41` — `/opt/felhom/stacks/<name>` →
|
||||||
|
`/opt/docker/stacks/<name>/app.yaml` (the real path, and the folder stays after a clean remove).
|
||||||
|
|
||||||
|
**System poster:** the facts file and the html changed together (text only, the hub-DB push times); no new drawing.
|
||||||
|
|
||||||
|
**Machines:** 9202 used for the R-1252 proof and put back (one difference: Immich's new empty volume, R-1267). DooPlex,
|
||||||
|
ep0, household boxes and the hub: not changed. No release, deploy, vouch or publish.
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
# REPORT — 2026-10-28 (day, attended at the STOPs): HTTP2Shell steps live, the runtime cache fills itself, the 20 apps
|
||||||
|
|
||||||
|
Architecture read: `09-update-architecture.md` §3 (decisions 255, 275–278) and §6.4 (the security-fix lane);
|
||||||
|
`08-alarm-ladder.md` §11 (the watch).
|
||||||
|
|
||||||
|
## In plain words
|
||||||
|
|
||||||
|
1. Last night's three steps are live: **yes** — Home Assistant 2026.10.4 and Wanderer hidden (`b5b78f6`), Immich v3.3.5
|
||||||
|
(`0158b50`). demo-hp (9201) read the new commit and listed Home Assistant's step as due.
|
||||||
|
2. The runtime cache fills itself in CI: **yes** — no DooPlex change. With two entries taken out of the cache, a CI run
|
||||||
|
read Home Assistant 2026.10.4 (zstd) in 87 s and RomM 5.4.1 (gzip) in 25 s, and took audiobookshelf from the cache.
|
||||||
|
The whole job took 3 min 34 s.
|
||||||
|
3. Of the 20 apps: **10 are live** as tested security steps. **4 need a big step.** **6 have no fixed release yet.**
|
||||||
|
4. The operator: give the homelab session its list (unchanged since the morning). Nothing else is waiting on you.
|
||||||
|
|
||||||
|
## Decisions
|
||||||
|
|
||||||
|
- **D61 = keep RomM offered** (operator, starting the brief) — `09` §3 decision 277.
|
||||||
|
- **zstd in CI through a hash-pinned `zstandard` wheel, not `zstd` in the runner image** (the brief's „say which and
|
||||||
|
why") — decision 278. The runner image would be a homelab change, a rebuild and a k3s rollout on DooPlex, which the
|
||||||
|
brief forbids. A wrong or missing wheel gives NOT CHECKED, never a pass.
|
||||||
|
|
||||||
|
## The 20 apps — `documentation/audits/runtime-fixes-2026-10-28/TABLE.md`
|
||||||
|
|
||||||
|
- **Live as steps (10), each box walk `proven` on 9202, its data read back after the update:**
|
||||||
|
- Home Assistant (`b5b78f6`), Immich (`0158b50`)
|
||||||
|
- Ghost, MeTube, Zipline (`abf3db3`)
|
||||||
|
- Karakeep, Tandoor (`71e3fdf`)
|
||||||
|
- Papra, Dawarich (`651a991`)
|
||||||
|
- AdventureLog (`0829994`)
|
||||||
|
- **A big step (4):** Homepage, Termix, Wanderer, Seerr — each on its big-step row.
|
||||||
|
- **No fix (6):** ActualBudget, cal.com, Calibre-Web, Paperless, RomM, SparkyFitness — each on its row, with its exposure.
|
||||||
|
- **No bench test ran:** none of these apps has a bench fixture.
|
||||||
|
- **AdventureLog failed twice before it passed:**
|
||||||
|
- After an update its backend needs about 2 min 45 s to start. With a 30 s start time, Docker marked it unhealthy
|
||||||
|
for about a minute.
|
||||||
|
- A record every 10 seconds shows this. With 300 s the services go from „starting" straight to „healthy".
|
||||||
|
- The step carries that change. Run 2 was inconclusive: 9202's copy of the drill was old.
|
||||||
|
|
||||||
|
## Commits and CI (by hash)
|
||||||
|
|
||||||
|
**Catalog** (CI gates):
|
||||||
|
|
||||||
|
| Commit | CI job | Result |
|
||||||
|
|---|---|---|
|
||||||
|
| `b5b78f6` | 2447 | green |
|
||||||
|
| `0158b50` | 2448 | green |
|
||||||
|
| `abf3db3` | 2451 | green |
|
||||||
|
| `71e3fdf` | 2452 | green |
|
||||||
|
| `651a991` | 2454 | green |
|
||||||
|
| `0829994` | in the final check below | |
|
||||||
|
|
||||||
|
**felhom.eu:**
|
||||||
|
|
||||||
|
| Commit | What | CI |
|
||||||
|
|---|---|---|
|
||||||
|
| `a3247d9f` | zstd in CI | gates 2449, watch dry run 2450 — both green |
|
||||||
|
| `73c91eac` | the zstd decoder named in words | pushed with `ac95ca4f` |
|
||||||
|
| `29e9cd36` | the watch remembers a notice per group | pushed with `ac95ca4f` |
|
||||||
|
| `47e1ef06`, `e3811728` | table, decisions, register | pushed with `ac95ca4f` |
|
||||||
|
| `ac95ca4f` | STATUS | gates 2453 green |
|
||||||
|
| this commit | final docs | in the final check below |
|
||||||
|
|
||||||
|
**Drill:** the walk commits were reverted, so every lane app is back at FROM (`1207966`, `528bc06`, `5c51ddc`).
|
||||||
|
|
||||||
|
## Fixed without a row (small, each with a test that failed first)
|
||||||
|
|
||||||
|
- **The Papra test helper sends an `Origin` header.**
|
||||||
|
- Papra 26.x refuses a sign-up without one (`MISSING_OR_NULL_ORIGIN`).
|
||||||
|
- Proven live: the sign-up was refused (403) before the fix and works (200) after.
|
||||||
|
- **The box-walk drill mover moves two services that share one image.**
|
||||||
|
- Dawarich's app and its worker use the same image. The walk stopped with „FROM ref not found".
|
||||||
|
- Test: `test_box_walk_drill_move.py`. Proven live on Dawarich.
|
||||||
|
- **The walk loads the drill into the box before it installs the old version.**
|
||||||
|
- My revert of the drill never reached 9202. The walk's own check stopped the run as inconclusive.
|
||||||
|
- **The nightly watch remembers a notice per group** (`dooplex:<image>|<id>`).
|
||||||
|
- Before, the catalog's RomM and DooPlex's RomM shared one memory entry. A new notice on both reached only one of
|
||||||
|
them.
|
||||||
|
- Old entries are copied once, so the change sends no repeat mails. Test W13: 4 of 5 checks failed before the fix.
|
||||||
|
- **The RUNTIME log line names the zstd decoder in words**, not as a Python module object.
|
||||||
|
- **Two code comments** no longer say the cache is filled on DooPlex.
|
||||||
|
|
||||||
|
## Corrected
|
||||||
|
|
||||||
|
- **Calibre-Web does load Brotli.** This morning's table said it does not.
|
||||||
|
- The morning check used the process list. Inside the container, the app's memory map could not be read.
|
||||||
|
- Read from the guest instead (`/proc/<pid>/maps`): the app maps Python Brotli 1.2.0, Pillow's libbrotlidec 1.2.0
|
||||||
|
and Debian's 1.1.0. It now has its own row (R-1337).
|
||||||
|
- **The file manager does not load Brotli on any of the three boxes.** Measured: `ldd` and the maps of every process.
|
||||||
|
|
||||||
|
## Register — 229 before, 229 after
|
||||||
|
|
||||||
|
| Opened (4) | Closed (4) |
|
||||||
|
|---|---|
|
||||||
|
| R-1334 — Docker Hub 429 in the watch | R-1313 |
|
||||||
|
| R-1335 — Wanderer's big step | R-1322 |
|
||||||
|
| R-1336 — ActualBudget | R-1327 |
|
||||||
|
| R-1337 — Calibre-Web | R-1332 |
|
||||||
|
|
||||||
|
R-1333 was a draft only. Its fix landed in the session (the watch's memory per group), so it was never filed.
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
| Observation | Result |
|
||||||
|
|---|---|
|
||||||
|
| Docker Hub's tag API answers 429; 19 DooPlex images NOT CHECKED in night 2446 | FILED: R-1334 |
|
||||||
|
| ghcr.io answered one blob with 429 during the dry run, while this session read images from the same address | NOT-A-FINDING: caused by this session's own parallel reads; R-1334 carries the rate-limit work |
|
||||||
|
| The security review flagged „version rollback" in the drill revert commits | NOT-A-FINDING: the drill is the test copy and must stand at FROM before a walk; the live catalog keeps the new versions |
|
||||||
|
| The evidence scan matched `Drill-1a37bb84e421` | NOT-A-FINDING: a seeded Ghost site title, not a secret |
|
||||||
|
| The evidence scan matched `"password": mpw` in `immich_repair.py` | NOT-A-FINDING: a variable name in code, not a value |
|
||||||
|
| A `git pull --autostash` in the shared felhom.eu clone emptied Dawarich's `walk.txt` while the walk was writing it | NOT-A-FINDING (fixed in the session): the log was rebuilt from the verdict's `log` field, and walk output now goes to the scratchpad |
|
||||||
|
| AdventureLog run 1's files were overwritten by run 2 (same folder) | NOT-A-FINDING: run 3 reproduced the failure on its own and is kept |
|
||||||
|
| A `pkill -f` matched its own shell (exit 144) | NOT-A-FINDING: the known trap; nothing had started; re-run in two commands |
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
- **Machine:** 9202 is back on the LIVE catalog (`repoint-9202-live.txt`, cache HEAD `0829994`). Every walk app was
|
||||||
|
removed through the product and verified gone: no container, no `app.yaml`, no volume.
|
||||||
|
- **Host:** nothing was provisioned. The bench (9401) was not started.
|
||||||
|
- **Hub:** no change.
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
# REPORT — security night 2026-11-02/03 (unattended): the watch, the new notices, six small fixes, unproven claims
|
||||||
|
|
||||||
|
Read first: `08-alarm-ladder.md` §11, `09-update-architecture.md` decision 255, `07-backup-architecture.md`, `03-host-agent.md`, `05-hub-architecture.md`, `00-capability-map.md`.
|
||||||
|
|
||||||
|
## Decisions taken by CC (operator may reverse)
|
||||||
|
|
||||||
|
- **Uptime Kuma's step crosses 2.5.8**, which the 10-27 batch refused: 2.5.8 turns a TCP-port monitor with TLS into a plain TCP one (upstream: „never worked as expected"). The fix (admin-only code execution; unauthenticated RSS cache poisoning) outweighs it, and no box runs Kuma.
|
||||||
|
- **R-1339's written fix was not built:** a sub-folder „not skipped" would write a `userdata/` inside the app's folder. The behaviour stays; only the false line goes.
|
||||||
|
- **A brief's expectation that turned out false, said here:** the watch was not blind to Docker Hub, and R-1334's fix had held. Traefik's line was no false alarm.
|
||||||
|
|
||||||
|
## Six lines
|
||||||
|
|
||||||
|
1. **The watch was not blind; it could not be heard.** A streak is per source, so seven pins that can never be read (six private images, `minio/minio`) held `registry:docker.io` and `runtime` open for a week: no real outage could have mailed. 0 Docker Hub 429 on five nights. Fixed: those seven are a stated GAP, timeouts are retried. Proven: run 2550 (runner), dry 2555 (runner-2), night 2565 (runner-2): `FAILED-SOURCES 0`, memory `failures {}`.
|
||||||
|
2. **The notices:** Vaultwarden, Uptime Kuma, n8n, PrivateBin and Gitea are real; **Traefik is real too, high, on every box** (GHSA-xv5x, hidden behind a word-only line). Live: gitea 28.3.0, uptime-kuma 2.5.9, grafana 13.2.5, actualbudget 26.11.0, n8n 2.46.4, vaultwarden 1.37.5-alpine. PrivateBin has no 2.1.1 image.
|
||||||
|
3. **The six waiters:** ActualBudget fixed (live). Paperless and SparkyFitness have a fixed release on a new main version (proven on the drill). cal.com, Calibre-Web, RomM: none.
|
||||||
|
4. **The small fixes:** 7 done with red first. Live-proven 3 (R-1347, R-1348, R-1339); R-1351 test only (the race fired on neither image in six boots); R-1323, R-1352, R-1302 test only, by the brief.
|
||||||
|
5. **RomM's night leg:** done, 2026-10-28 03:22, 88 s (closed; demo-hp's library is small).
|
||||||
|
6. **Claims:** 8 walked (31 → 23 not walked); 2 walked and failed in part: the hub down loses events (R-1361), the LAN name has gaps (R-1364).
|
||||||
|
|
||||||
|
## The morning release list, and the one decision
|
||||||
|
|
||||||
|
STATUS has both. First the golden (R-468). Controller v0.350.0: Traefik v3.7.17 first, R-1347, R-1348, R-1339, R-1351, R-1323. Agent v0.172.0 + vouch: R-1302. Hub v0.179.0: R-1352. Catalog, drill only: Paperless 3.4.1 and n8n 3.0.1 (bench, box, undo), SparkyFitness v1.11.0 (the same, behind the licence R-784), Mealie v3.30.0 and Termix 26.11.0 (bench, box). Decision D66 (R-1365): should a source that stays down mail again every 7 days? Pick: yes.
|
||||||
|
|
||||||
|
## Commits and CI (by head_sha, the jobs endpoint)
|
||||||
|
|
||||||
|
| Repo | Commit | CI job | What |
|
||||||
|
|---|---|---|---|
|
||||||
|
| felhom.eu | `4afa8c3c` | 2548 green | the watch: stated GAP, retries, R-1310's rules |
|
||||||
|
| felhom.eu | `2f7e836f` | 2553 green | hub R-1352, commit only |
|
||||||
|
| felhom.eu | `d02c0c2c`, `17820abb` | 2557, 2561 green | evidence |
|
||||||
|
| controller | `c4b6844`, `3764445`, `13e34d4` | 2551, 2560, 2562 green | Traefik pin, five fixes, commit only |
|
||||||
|
| agent | `50eeccc`, `ecbb0cc` | 2552, 2563 green | R-1302, commit only |
|
||||||
|
| catalog | **`c760fca`, the one live push** | 2559 green | six `security_fix` steps; negative control first (`walks/_negative-control/`: `failed`) |
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
|
||||||
|
Before 228, after 235. **Closed (5):** R-1303, R-1310, R-1324, R-1334, R-1336. **Opened (12):** R-1354 Traefik, R-1355 DooPlex's notices, R-1356 n8n 3, R-1357 PrivateBin, R-1358 cloudflared, R-1359 Mealie + Termix, R-1360 Meilisearch, R-1361 events lost, R-1362 Backups page, R-1363 raw JSON in a mail, R-1364 LAN DNS, R-1365 D66.
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- **FILED: R-1361, R-1362, R-1363, R-1364** (the walks); **R-1354–R-1360** (the triage).
|
||||||
|
- **Noted in R-468: the golden gate went red at midnight:** the Sunday bake of 11-01 was missed (in no brief), golden 0.344.0 is 8 days old. Baking is not this night's work; no waiver written. A code push to felhom.eu is refused until a bake and the vouch. First on the list.
|
||||||
|
- **NOT-A-FINDING: the watch's 03:20 run** is `gates`, not the watch.
|
||||||
|
- **NOT-A-FINDING: 9202's „Tárhely (…)" sources with no mount, and its refused removal of a per-app `HDD_PATH`:** legacy of its own old installs (R-839 refuses that shape); no other box holds one.
|
||||||
|
- **NOT-A-FINDING: `CT990000 stopped create` on Tester 1:** the restore test at that moment; it passed at 22:45.
|
||||||
|
- **NOT-A-FINDING: DooPlex restarted 2026-11-01 16:11 CET:** two boots, read only; its only effect here is that R-1353's log covers 31 h.
|
||||||
|
- **NOT-A-FINDING: the watch's „0 new, 63 reminders" mail of 11-03:** the 7-day reminder, as designed.
|
||||||
|
- **Fixed without a row (the size rule):** the Vaultwarden fixture (1.37's register route), the n8n fixture (`/healthz` before the routes), the box-admin guard (the drill mirror), the Actual fixture's log, the undo tool (`tools/undo_proof3.py`: a probe with a `container:` line, services read by label).
|
||||||
|
- **A mistake, said plainly:** one dead session token of a throwaway Actual reached the evidence in `d02c0c2c`; redacted in `03a71b68`; the app was already removed; it stays in git history. Dead, and the fixture no longer logs it.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
Machine: 9202 back on its image, its yaml byte-identical, on the live catalog; bench 9401 stopped; Tester 1's routes and firewall as found, address empty. Host: the drill reset to live; local test images removed; nothing else provisioned. Hub: untouched. **Left, on purpose:** a `restore_mixed` hold for `radicale` in 9202's settings (the fence forbids CC to clear one; the command is in `part3/README.md`); `email_cleared: true` in Tester 1's settings; 9202's Paperless is a fresh install, the throwaway of 09-22 went with the walks. No VM was installed, so no hub host to delete.
|
||||||
|
|
||||||
|
Poster: no fact on it changed. Instruction files: none edited. Evidence: `documentation/audits/security-2026-11-02/`.
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
# REPORT — security app updates, 2026-10-27 (day, attended)
|
||||||
|
|
||||||
|
> A sibling of `REPORT.md` (the release session of the same day owns that file).
|
||||||
|
|
||||||
|
## In plain words
|
||||||
|
|
||||||
|
- **Real notices:** the security watch flagged 38 pins. Read by hand: 33 real, in 32 apps plus 3 box programs; 5 apps
|
||||||
|
were false alarms (`audits/security-app-updates-2026-10-27/TRIAGE.md`).
|
||||||
|
- **Out tonight by the quick lane:** 14 apps, in three batches, each on your yes — Rallly, RomM, code-server, Outline,
|
||||||
|
Vikunja; n8n, PrivateBin, Docmost, Grafana; Homebox, Mealie, Audiobookshelf, BentoPDF, Uptime Kuma. Wishlist and
|
||||||
|
Bookstack had no notice id and went as normal steps. Immich (the lowest item) was not done.
|
||||||
|
- **Big steps:** Jellyfin 10.11 → 12.4 is done (bench, box, and the undo proven). Ten are left, a row each.
|
||||||
|
- **No-fix picks:** four questions for you (STATUS D56–D59); none changed a template.
|
||||||
|
- **Box programs released:** yes — controller 0.346.1 (Traefik v3.7.16, file manager 1.5.8-stable, cloudflared
|
||||||
|
2026.10.4), global floor 15:50:15Z on your yes.
|
||||||
|
|
||||||
|
## Catalog commits (app-catalog-felhom.eu) and CI by commit
|
||||||
|
|
||||||
|
| Commit | What | CI job |
|
||||||
|
|---|---|---|
|
||||||
|
| `06e603e` | batch 1 + wishlist; Rallly/Jellyfin fixtures; the privileges gate skips `steps/*.felhom.yml` | 2422 success |
|
||||||
|
| `7b0071e` | batch 2 + bookstack | 2425 success |
|
||||||
|
| `65a6b11` | batch 3; Homebox/Mealie fixtures | 2432 (see the final report for the read-back) |
|
||||||
|
| `540f185` | Jellyfin 10.11.11 → 12.4 | (see the final report) |
|
||||||
|
|
||||||
|
## Controller
|
||||||
|
|
||||||
|
`v0.346.1` = the release commit `b1b9be3` (v0.346.0) + one commit `5ba1b86` (tag `v0.346.1`; the same commit is on
|
||||||
|
`main`). CI 2427/2428 success. Image `gitea.dooplex.hu/admin/felhom-controller:0.346.1`
|
||||||
|
`sha256:b6b1e5488d31cb0b6596228fd6bf50900692ab0798d5da858159db67e8440190`, built from `ed3927f` (= `5ba1b86` + the
|
||||||
|
release session's REPORT.md only). MinAgent 0.158.1.
|
||||||
|
|
||||||
|
## Proofs
|
||||||
|
|
||||||
|
- The lane: a negative control first (busybox as TO → `failed`, undone). 14 walks `proven`, 12 with the seed read back.
|
||||||
|
- Part 5: 9202 (rc.1) and Tester 1, demo-hp, demo-felhom (per-box floors 15:41Z): all 29 public names redirect to
|
||||||
|
https; the file manager signs in (Tester 1, 9202: listing 200, 401 without a cookie); an app container cannot reach
|
||||||
|
it on any box (R-1281 probe with a control); the tunnel gap 4–5 s per box, the Traefik restart 7–20 s; a 30-minute
|
||||||
|
watch after the global floor was clean; the per-box floors cleared.
|
||||||
|
- Part 3: Jellyfin's undo — 12.4 migrated the database, failed on purpose, the box undid it, 10.11 signed the user in.
|
||||||
|
|
||||||
|
## Register
|
||||||
|
|
||||||
|
Rows at the task's start 221; now 224. This task opened 15 (R-1310, R-1311, R-1313–R-1325, R-1314 closed the same day)
|
||||||
|
and closed 3 (R-1304, R-1306, R-1314); R-1305 NARROWED. The release session closed ten and opened one in the same time.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
Machine: 9202 back on the live catalog, controller 0.346.1, its saved config copy removed (identical); the walks
|
||||||
|
removed every app they installed; the drill mirror in 9202's volume stays (harmless, as after B23). Bench 9401
|
||||||
|
stopped (as found). Host: nothing. Hub: per-box floors cleared; the global floor 0.346.1 stays (intended).
|
||||||
|
|
||||||
|
Evidence: `documentation/audits/security-app-updates-2026-10-27/`.
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
# REPORT — D39 = B: the security-fix lane (2026-10-23, attended)
|
||||||
|
|
||||||
|
**Plain words first.** An app with a public security hole can now get its fix at night on every box, after one test
|
||||||
|
on the test box. Controller 0.341.0 does it and is the floor since 13:24 (your yes). Older controllers skip such a fix
|
||||||
|
at night and take it when a person presses Update — both seen on the test box. Nothing needs you.
|
||||||
|
|
||||||
|
- Design: `09-update-architecture.md` §3 decision 255 (+ a dated note on decision 13). Capability map B23.
|
||||||
|
- Catalog `3898593` (+ a box-walk address fix after it); controller `63dcccf` = 0.341.0; drill commits `8c1d908` …
|
||||||
|
`17e0da0` (the drill only).
|
||||||
|
- Evidence and the full table: `documentation/audits/security-lane-2026-10-23/README.md`.
|
||||||
|
- Register 186 → 186: closed R-1239 (D39 = B); opened R-1240 (9202's 34 saved configs with the catalog token); R-1238
|
||||||
|
stays open, its text now says the next Plex fix can use the lane.
|
||||||
|
- `STATUS.md`: D39 out of the decide table, one „what works" line, the „Updated" paragraph replaced (the old one is in
|
||||||
|
`STATUS-HISTORY.md`). `CONTEXT.md`: the rulings. System poster: no fact changed (none of the three actions).
|
||||||
|
- Instruction / fact files edited: `app-catalog-felhom.eu/CLAUDE.md` (the lane, added); `operations/nodes.md` line 221
|
||||||
|
(9202 „same dashboard password" → its own password, measured) and line 46 (Tester 1 `.101` → DHCP, `.148` today).
|
||||||
|
- No `--no-verify` in this run.
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
# REPORT — the nightly security watch (R-1303), 2026-10-26 night
|
||||||
|
|
||||||
|
> A sibling of `REPORT.md`: the night session owns `REPORT.md` tonight. Readings taken alone (the brief's „safest
|
||||||
|
> reading", stated first):
|
||||||
|
> 1. **The key:** the operator said yes; the token did not exist by the end, so the watch runs in no-key mode (box
|
||||||
|
> programs nightly, apps' advisory lists every second night) and switches by itself when `GH_ADVISORY_TOKEN` exists.
|
||||||
|
> The name is `GH_…`, not `GITHUB_…`: Gitea reserves the `GITHUB_` and `GITEA_` prefixes for Actions secrets.
|
||||||
|
> 2. **The memory:** a file the CI job commits to `main` itself (`documentation/security-watch/state.json`), not the
|
||||||
|
> runner's disk (two runners, two disks), not the Actions cache (needs a node step), not DooPlex (a host change).
|
||||||
|
> 3. **A seed that could not read everything** (no key) leaves those pins pending; their first real read is recorded
|
||||||
|
> silently. Otherwise the first nights after the seed would have been the mail storm the brief forbids.
|
||||||
|
|
||||||
|
## In plain words
|
||||||
|
|
||||||
|
1. **Sources picked:** GitHub's per-repo security advisories (the main source), GitHub release notes, the Cloudflare
|
||||||
|
blog plus cloudflared's own `VULN-` notes, OSV for our Go builders, and the registries for „newer version". **They
|
||||||
|
cannot see:** a fix told only by mail, forum or a blog we do not read; a security fix nobody calls one; packages inside
|
||||||
|
our own images (Debian, Alpine, Samba); closed-source apps (Plex, Emby); floating tags like `redis:7-alpine`.
|
||||||
|
2. **Watched:** 108 pins — 11 box programs and build bases, 97 catalog images (21 of them with a stated „no source").
|
||||||
|
3. **First real run (the seed, CI job 2408):** ADVISORY on **30 pins (142 notices)**: 3 box programs (traefik, the file
|
||||||
|
manager, cloudflared) and 27 apps; BEHIND on 17; 0 NOT CHECKED. Nothing was mailed — the seed records them.
|
||||||
|
4. **The test mail arrived:** yes — „[Felhom security] TEST — not a real advisory …" in the admin@felhom.eu mailbox
|
||||||
|
(Resend id `01a17a5d-…`, CI job 2407).
|
||||||
|
5. **The operator does:** make the GitHub key and store it as `GH_ADVISORY_TOKEN` (CC's store and Gitea Actions). Then
|
||||||
|
decide the order of triage for the advisory list (R-1305, R-1306).
|
||||||
|
|
||||||
|
## The ADVISORY list today (box programs first; full list `audits/security-watch-2026-10-26/ADVISORIES-2026-10-26.md`)
|
||||||
|
|
||||||
|
| Program | Our version | Fixed in (notices) |
|
||||||
|
|---|---|---|
|
||||||
|
| traefik | v3.7.13 | v3.7.14–v3.7.16 (15; triage: none seems to apply to our config — R-1304) |
|
||||||
|
| file manager (FileBrowser Quantum) | 1.5.6-stable | 1.5.7-stable for 2 of 36 (GHSA-qx86, needs the password; GHSA-3846); the rest on 2.x (R-1305) |
|
||||||
|
| cloudflared | 2026.10.3 | 2026.10.4 (VULN-176088, metrics allowlist; our metrics are on the container's own loopback) + the eMudhra blog post (already fixed, R-1300) |
|
||||||
|
| 27 apps | — | e.g. Jellyfin 10.11.11 → 12.0; Gokapi 1.9.6 → 2.2.4; Seerr 2.7.3 → 3.5.0; Wanderer v0.20.0 → v0.21.2; Rallly 4.15.3 → 4.15.4; Immich v3.2.4 → v3.3.5 (notes); SparkyFitness: no fix yet (R-1306) |
|
||||||
|
|
||||||
|
## Commits and CI (by commit)
|
||||||
|
|
||||||
|
| Repo | Commit | What | CI |
|
||||||
|
|---|---|---|---|
|
||||||
|
| felhom.eu | `97ae7086` | spike + rows R-1303, R-1304 | job 2401 success |
|
||||||
|
| felhom.eu | `1f62d629` | the watch, the shared list + gate, tests, workflow | job 2402 success |
|
||||||
|
| felhom.eu | `6ee23a3b` | abandoned = GAP; seed-pending | (see below) |
|
||||||
|
| app catalog | `f61800a` | `upstream:` in 65 templates, gate + decoys, checklist 1.10 | job 2403 success |
|
||||||
|
| controller | `e01eb64b` | `upstream.txt`, `# upstream:` comments, gate + decoys | job 2404 success |
|
||||||
|
| felhom.eu | `ed843da7` | the memory, committed by the CI job itself | — |
|
||||||
|
| felhom.eu | (this commit) | docs, rows R-1305–R-1309, STATUS, this report | read at the end |
|
||||||
|
|
||||||
|
Live runs in CI: job 2405 (dry: every host reached; 30 NOT CHECKED because DooPlex's shared GitHub quota was at 0 —
|
||||||
|
printed and counted, the reason for the key), 2407 (test mail), 2408 (seed; memory pushed). Tests: 38 checks, no
|
||||||
|
network; red-proved five ways (dedupe, streak, the 30-day window, a swallowed source error, the seed-pending set); the
|
||||||
|
two gates' decoys red-proved with a reader that reads comments.
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
|
||||||
|
Before **215**, after **222**; opened **7** (R-1303 – R-1309), closed **0**.
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- traefik v3.7.13 inside 12 October advisories — **FILED: R-1304**.
|
||||||
|
- The file manager 1.5.6-stable inside 36 notices, 2 fixed in its own channel — **FILED: R-1305**.
|
||||||
|
- 27 app pins with published notices — **FILED: R-1306** (one row; the list is the file).
|
||||||
|
- Packages inside our own images not watched — **FILED: R-1307**. Floating tags — **FILED: R-1308**. Mail-only and
|
||||||
|
silent fixes, closed-source apps — **FILED: R-1309**.
|
||||||
|
- Three upstream repos renamed (cal.com → `calcom/cal.diy`, jellyseerr → `seerr-team/seerr`, wanderer →
|
||||||
|
`open-wanderer/wanderer`) — **NOT-A-FINDING: the mapping uses the new names; the images still pull.**
|
||||||
|
- plant-it's image name points nowhere — **NOT-A-FINDING: known and stated in its template (abandoned); the watch lists
|
||||||
|
it as GAP and never looks it up.**
|
||||||
|
- The hub's Dockerfile lines are not gated in felhom.eu's own runner (only the catalog's and the controller's are) —
|
||||||
|
**NOT-A-FINDING: the watch reads them nightly and reports a missing line as NOT CHECKED; a gate there would need its
|
||||||
|
own decoy suite for two lines.**
|
||||||
|
- nginx's channel test treats 1.31 (mainline) as the same channel as 1.30 (stable) — **NOT-A-FINDING: BEHIND is a weekly
|
||||||
|
hint; the notice match does not depend on it.**
|
||||||
|
- Without a key, the dry run spent DooPlex's whole GitHub quota (shared with every DooPlex client) — **NOT-A-FINDING:
|
||||||
|
that is R-1303's „blocked on"; the watch reported it as NOT CHECKED, as designed.**
|
||||||
|
|
||||||
|
## Documents changed
|
||||||
|
|
||||||
|
`runbooks/monthly-floating-retest.md` §4a (the BEHIND check is nightly; the monthly run keeps the same-name re-test);
|
||||||
|
`architecture/capability-map-detail/B.md` B23 (a watch mail starts the lane); `architecture/08-alarm-ladder.md` §11 (the
|
||||||
|
three signals, `[DESIGN]`); `scripts/CHANGELOG.md`; `REUSE.md`; the catalog's `NEW-APP-CHECKLIST.md` 1.10. System
|
||||||
|
poster: none of the three needed — no machine, path, backup tier or key changed (a CI job reads public sources).
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
Machine and host: nothing provisioned. Hub: nothing. CI: three manual runs, no lasting change but the memory file. Work
|
||||||
|
copies `felhom.eu-cfsec`, `app-catalog-cfsec`, `felhom-controller-cfsec` on DooPlex: removed at the end, all pushed.
|
||||||
@@ -0,0 +1,129 @@
|
|||||||
|
# REPORT — weekend night 2026-10-23: the hub's last plain tokens locked; the upload and CPU watch (spike, design, build)
|
||||||
|
|
||||||
|
**Safest readings taken (unattended), stated first:**
|
||||||
|
- **A test image is not a release.** The upload watch was proven on scratch 9202 with `felhom-controller:0.342.0-rc.1`,
|
||||||
|
set by hand in 9202's `/etc/felhom-controller-image` (the shape of today's security-lane rc.1). An `-rc` tag is never
|
||||||
|
offered to a household's update button. No floor was touched; only 9202 ran it, and 9202 is back on 0.341.0.
|
||||||
|
- **9202's line was seeded by hand** (one 40 Mbit/s record): its off-site backup is off by design, so it can never learn a
|
||||||
|
line itself. The learning path is pinned by unit tests and by tonight's one-second read of the real pushes on the
|
||||||
|
three test boxes (read only).
|
||||||
|
- **The hub 0.168.0 image was built** (the brief asked for it) **and not deployed.** The four new hub event types are in
|
||||||
|
hub 0.169.0, committed only.
|
||||||
|
- **Per-app trigger instead of a whole-box trigger.** The brief said „the box's upload above X % of the line". Inside its
|
||||||
|
container the controller cannot tell LAN from internet bytes, nor see the box's total; it can see each app's exact
|
||||||
|
bytes, and the per-app view is what the explanation compares. So the trigger is per app (§6.2 of the spike).
|
||||||
|
- **The first day:** apps the box can ask are judged at once on the 5 Mbit/s floor (a mismatch is evidence whatever the
|
||||||
|
line); apps it cannot ask are judged only after ~20 h of history. The brief's „first day: no alarm, only learning" holds
|
||||||
|
for those apps; for the others a mismatch on day one is still an alarm.
|
||||||
|
|
||||||
|
## Five lines
|
||||||
|
|
||||||
|
1. **Tokens locked:** the Cloudflare tunnel token, the Cloudflare API token and the git token are sealed in **hub 0.168.0**
|
||||||
|
— built, CI green, **not deployed** (you deploy it on Monday).
|
||||||
|
2. **Backup copies:** two options — encrypt the hub's Longhorn disk, or take it out of the Longhorn backup because the
|
||||||
|
hub's own copy to ep0 is already encrypted at rest and in transit (measured). **My pick: take it out** (D40).
|
||||||
|
3. **The line:** measured from the box's own nightly off-site upload (one-second samples, best 10 seconds) — a lower
|
||||||
|
bound; most nights the upload is too small to say anything, so a real test is your decision (R-1247).
|
||||||
|
4. **Apps that explain their traffic:** qBittorrent and Transmission (their own byte counters, within 1–7 %), Plex (its
|
||||||
|
viewers and transcodes). Jellyfin needs a key the household makes; Nextcloud and Immich give no number.
|
||||||
|
5. **The fake hacked app was caught: yes** — a flood from inside qBittorrent, hidden behind a real seed: one alarm after
|
||||||
|
15 minutes, one clear 15 minutes after it stopped. A CPU burner in Plex: the same. A real seed: no alarm.
|
||||||
|
|
||||||
|
## The spike's numbers
|
||||||
|
|
||||||
|
`documentation/audits/SPIKE-upload-watch-2026-10-23.md` (evidence beside it). Short form:
|
||||||
|
|
||||||
|
- **Normal levels** (three boxes, 19–31 days of one-minute samples): no app's own upload p95 reached 6 Mbit/s; the
|
||||||
|
longest run at ≥ 5 Mbit/s was **3 minutes**, at ≥ 1 Mbit/s 9 minutes. CPU: the longest run at ≥ 150 % of a core was
|
||||||
|
**7 minutes**, at ≥ 80 % 13 minutes. DooPlex's hacked Plex: ~44 Mbit/s for hours, ~10 cores.
|
||||||
|
- **Line speed:** at a one-minute sample the push exceeded 2 Mbit/s on 3 of 81 box-nights (the push moves a few MB);
|
||||||
|
the box keeps no byte count of it. Tonight's one-second read: §Night data below. No active test was run (ep0 changes
|
||||||
|
forbidden; a public test's terms not read).
|
||||||
|
- **Apps:** qBittorrent counted 131.1 Mbit/s while its container sent 134.6–140.5 (×1.03–1.07); Transmission ×1.001–1.02;
|
||||||
|
the limit API works and persists (KiB/s rounding); Plex answers its session lists (with the token it already has when
|
||||||
|
claimed); Jellyfin 401 without a household-made key.
|
||||||
|
- **A hard cap** (`tc` from a helper in the app's network namespace) works in an unprivileged guest: a flood held at
|
||||||
|
20 Mbit/s, the app kept answering. Not built (R-1248).
|
||||||
|
|
||||||
|
## The household's line (final text)
|
||||||
|
|
||||||
|
- HU: „A qBittorrent feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
|
||||||
|
- EN: "qBittorrent uploads at most N Mbit/s, so the family internet stays fast. You can change this in qBittorrent."
|
||||||
|
- The system settings page: „Feltöltési sebesség: legalább N Mbit/s (az éjszakai távoli mentésből mérve)" /
|
||||||
|
"Upload speed: at least N Mbit/s (measured from the nightly off-site backup)"; „még mérjük …" until known.
|
||||||
|
|
||||||
|
## Morning list (Monday)
|
||||||
|
|
||||||
|
1. Deploy hub 0.168.0 (the last plain tokens locked). Then read its start line `config tokens sealed at rest (8 …)`.
|
||||||
|
2. D40: the hub's plain Longhorn copies — stop that backup (my pick) or encrypt the disk; and whether to delete the two
|
||||||
|
plain copies already there.
|
||||||
|
3. D41: qBittorrent's default limit — 50 % of the measured line (my pick). D42: who is told — the family only about the
|
||||||
|
limit, the hacked-app alarms only you (my pick).
|
||||||
|
4. Release the upload watch when you say yes (hub 0.169.0 build + deploy, then controller 0.342.0 through the normal path).
|
||||||
|
5. For the homelab session (list only, no HM numbers): an upload-saturation alert on DooPlex's `eno1` (the line is
|
||||||
|
~100 Mbit/s; the flood peaked at 905 Mbit/s).
|
||||||
|
|
||||||
|
## Commits, images, CI (by commit)
|
||||||
|
|
||||||
|
| Repo | Commit | What | CI |
|
||||||
|
|---|---|---|---|
|
||||||
|
| felhom.eu | `5277d859` | hub 0.168.0: the tokens sealed; R-1243; STATUS D40 | run 2269 **success** |
|
||||||
|
| felhom.eu | `af5c005a` | hub 0.169.0: the four events (allowed, operator-only, mail line) | run 2271 **success** |
|
||||||
|
| felhom.eu | `a91d0b24` | spike, design (`08` §9), live proofs, rows R-1244..R-1248, STATUS | run 2272 **success** |
|
||||||
|
| felhom-controller | `a9c1997f` | v0.342.0 (unreleased): the upload and CPU watch | run 2270 **success** |
|
||||||
|
| felhom-controller | `60e487d` | CONTEXT + REPORT | run 2273 **success** |
|
||||||
|
|
||||||
|
Images: `felhom-hub:0.168.0` `sha256:3bfdecaae527d4762059600cdfaad83daf4e4b5b20661fc47acf128180d87cb3` (from `5277d859`,
|
||||||
|
not deployed); `felhom-controller:0.342.0-rc.1` (from `a9c1997f`, scratch 9202 only, removed from 9202 after).
|
||||||
|
|
||||||
|
**Teardown, three layers:** machine — 9202: the four throwaway apps removed through the product (containers and volumes
|
||||||
|
gone), my test files, probes and the `alpine:3.20` helper image removed, back on 0.341.0, the seeded record removed;
|
||||||
|
host — demo-hp: the receiver LXC 9471 destroyed with its disk; DooPlex: my private copy of the hub snapshot shredded,
|
||||||
|
the 9202 password copy shredded; hub — none. ep0: read only (the hub-copy snapshot list). No household box touched.
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
|
||||||
|
**Before 188, after 194; opened 6 (R-1243 … R-1248), closed 0.** R-975 stays open (WAITING-ON-OPERATOR: deploy 0.168.0).
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- The hub's Cloudflare/tunnel/git tokens unsealed in `config_json` — FIXED in hub 0.168.0 (R-975, not yet deployed).
|
||||||
|
- The hub's Longhorn copies are plain and sit in the open folder — FILED: R-1243.
|
||||||
|
- The upload watch is not released — FILED: R-1244; its limit size — FILED: R-1245; who is told — FILED: R-1246.
|
||||||
|
- The line is only a lower bound — FILED: R-1247. An app's API can lie — FILED: R-1248.
|
||||||
|
- `appliance_registrations.pairing_code` is plain — NOT-A-FINDING: it is shown on the box's screen and is one of two
|
||||||
|
factors; the other is hashed.
|
||||||
|
- R-879's sealing (0.138.0) ran no VACUUM, so old plaintext of those columns could sit in freed blocks of the live
|
||||||
|
`hub.db` — NOT-A-FINDING: 0.168.0's one-time VACUUM rewrites the file on its first start.
|
||||||
|
- The hub CHANGELOG's 0.167.0 header still said „NOT built" though it is deployed — fixed without a row (small).
|
||||||
|
- The shared memory index went over its size ceiling twice in one day (another session's lines) and blocked the push
|
||||||
|
gates — fixed without a row (shortened, no link lost); it sits 14 bytes under the ceiling now.
|
||||||
|
- The watch's very first minute after a start cannot reach apps by name (network joined seconds later) — NOT-A-FINDING:
|
||||||
|
that minute only sets baselines.
|
||||||
|
- The media wiring's generated override file stays after an app's removal — NOT-A-FINDING: rewritten before every start.
|
||||||
|
- `docker stats` text in the stored metrics has three significant digits — NOT-A-FINDING for the watch (it reads the
|
||||||
|
Engine API); the charts are display only.
|
||||||
|
- `felhom-samba` runs on the host network and has no counter — NOT-A-FINDING: infrastructure, not judged.
|
||||||
|
- Transmission's rate under its limit was not observed (no peer contact) — NOT-A-FINDING: the limit API was read back;
|
||||||
|
qBittorrent's was observed.
|
||||||
|
- A dated check came due on the push (R-1214, Tester 1's night copy): read read-only — the guest's memory „full" stall
|
||||||
|
was 444 µs in total over 11 h, 0 OOM kills; the copy was small (0.24 GB) and the capped container is not shown — the
|
||||||
|
row stays WATCHING, the due line removed.
|
||||||
|
- The night's line floor is low (10 Mbit/s on demo-hp, one second at 70) — FILED in R-1245's text (D41), no new row.
|
||||||
|
|
||||||
|
## Instruction files, poster
|
||||||
|
|
||||||
|
No instruction file edited. System poster: no fact on it changed (nothing deployed) — none of the three applied.
|
||||||
|
|
||||||
|
## Night data
|
||||||
|
|
||||||
|
Read 02:08–02:30Z, once a second, on the three test boxes (read only; spike §2.1):
|
||||||
|
|
||||||
|
- **demo-hp: counts.** Best 10 seconds 12.6 MB = **10.0 Mbit/s**. One second reached 70 Mbit/s.
|
||||||
|
- **demo-felhom: too small** (best 10 seconds 3.2 MB = 2.5 Mbit/s).
|
||||||
|
- **Tester 1: too small** (3.5 MB = 2.7 Mbit/s).
|
||||||
|
|
||||||
|
**What it means:** the nightly push is a burst of a few seconds. So the line it gives is a low floor. On demo-hp,
|
||||||
|
50 % of that floor is about 5 Mbit/s for qBittorrent — probably much less than the real line. The floor rises only when
|
||||||
|
a bigger night comes (the box keeps the best of 14 nights). Added to the D41 decision row (R-1245): if you want a
|
||||||
|
looser limit before a real line test exists (R-1247), pick a higher share or „none".
|
||||||
@@ -0,0 +1,55 @@
|
|||||||
|
# REPORT — 2026-10-27 evening: the clean-up window button (R-1299) and `sec rm`
|
||||||
|
|
||||||
|
## Plain words first
|
||||||
|
|
||||||
|
1. **The button is live** on the hub: customer page → **Backup & DR** tab → „Off-site clean-up window". You granted a
|
||||||
|
window on Tester 1 and cancelled it again; nothing was removed.
|
||||||
|
2. **`sec rm NAME` works** (cc-secrets 0.3.0): one name per call, saved in git, undo by `git revert`.
|
||||||
|
3. **46 store entries removed** on your yes (names in `documentation/audits/r1299-window-button-2026-10-27/sec-rm.txt`):
|
||||||
|
the two D55 token copies, 20 of the 10-20 rehearsal box, 14 of the night/fixes test boxes, 3 of the rig and the
|
||||||
|
release BYO box, 5 of the 10-24 sweep box, `R_C11_REWALK`, `R_PART4`. The store went from 92 to 46.
|
||||||
|
4. **You still do:** nothing from this run. (Today's three do-items are done: agent 0.167.0 vouched, the D55 token
|
||||||
|
revoked, `Tester4-b5dbbd` deleted.)
|
||||||
|
|
||||||
|
**A defect of my own, found by your click and fixed:** hub 0.174.0 logged each grant but stored NO event row (my code
|
||||||
|
called only the dispatcher, which writes no row, and my test checked the call, not the row). Hub 0.174.1 stores the row;
|
||||||
|
the test now reads the stored rows and fails on 0.174.0's shape. 0.174.1 also moved the card to Backup & DR, where you
|
||||||
|
looked for it first.
|
||||||
|
|
||||||
|
## Versions, digests, CI
|
||||||
|
|
||||||
|
| | Version | Digest | From | CI |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| hub | 0.174.0 | `sha256:4f9ee04a9ff7c108a37a411be3c3de521b6acccb431c9e550c3567f7211e2e94` | `d85198b8` | 2433 ✓ |
|
||||||
|
| hub | 0.174.1 | `sha256:bb43f88acd59b8b20914381932eb2d9f779d44c190c34d3349eed91d37ce71fb` | `de57c090` (manifest `714ee236`) | 2437 ✓ |
|
||||||
|
| cc-secrets | 0.3.0 | — | `dc82cf9` | no CI in that repo; local suite 72 OK, red-proofs 28/28 |
|
||||||
|
| agent pin | day0_agent 0.167.0 | — | `3055bc5` | (end-of-run check) |
|
||||||
|
|
||||||
|
Live: 0.174.0 at 19:25 CET, 0.174.1 at 20:26 CET; both Synced/Healthy, 0 restarts, hub-only ArgoCD sync, built from a
|
||||||
|
clean clone (the shared clone held another session's uncommitted installer edit). 30-minute watch after 0.174.1
|
||||||
|
(to 21:13 CET): 0 warning-or-worse events, 0 WARN/ERROR lines, 10 box reports in the last 20 minutes, 0 restarts, no
|
||||||
|
new alarm mail.
|
||||||
|
|
||||||
|
## Tests
|
||||||
|
|
||||||
|
Hub `TestR1299_*` (five): the page per branch (and in the Backup & DR tab), a grant through the form stores one grant with
|
||||||
|
its limit and one stored row, refusals store nothing (above the count, not a number, no confirm, no CSRF), the ceiling
|
||||||
|
when no count is known (260 passes, 501 refused), cancel. Red-proofs: seven mutations, all caught
|
||||||
|
(`redproof-0174.0.txt`, `redproof-0174.1.txt`). **The tests were written after the code in Part 1** — so each was then
|
||||||
|
seen failing by a mutation; Part 2's `sec rm` tests were written first and seen failing before the code existed.
|
||||||
|
|
||||||
|
## Rows
|
||||||
|
|
||||||
|
Before 224 open → after 224. Closed: R-1299. Opened: R-1326 (the hub's start-up, 31–36 s, eats half its liveness margin
|
||||||
|
of ~65 s). Updated: R-1237 (its dead names are gone from the store; the sweep no longer watches those values; the
|
||||||
|
transcripts are unchanged).
|
||||||
|
|
||||||
|
## Observations
|
||||||
|
|
||||||
|
- FILED R-1326 — hub start-up vs the liveness probe.
|
||||||
|
- FIXED in-session — the window event wrote no row (0.174.0 → 0.174.1).
|
||||||
|
- FIXED in-session — the card's place (Backup & DR).
|
||||||
|
- NOT-A-FINDING: CC cannot press the button — its token has no `offsite:write` and CC holds no hub password, by design
|
||||||
|
(`09` §3 decision 251); the live click was the operator's.
|
||||||
|
- NOT-A-FINDING: the `sec doctor` sweep line still lists the removed names — it reports the LAST night's sweep (03:40 UTC),
|
||||||
|
before the removal.
|
||||||
@@ -78,6 +78,7 @@
|
|||||||
| `offsitekeys.Service` (`RegisterKey`, `ConfirmKey`, `AuditAll`, `OpenWindowFor`, `CloseWindowFor`, `SweepExpiredWindows`) | hub/internal/offsitekeys/service.go | — | Binding the registrar to the store, descriptor and operator events | The box-facing API (`/api/v1/offsite/register-key…`) answers with NO credential — pinned by `TestOffsiteKeyEndpoints_AuthAndNoPasswordInAnyResponse`. |
|
| `offsitekeys.Service` (`RegisterKey`, `ConfirmKey`, `AuditAll`, `OpenWindowFor`, `CloseWindowFor`, `SweepExpiredWindows`) | hub/internal/offsitekeys/service.go | — | Binding the registrar to the store, descriptor and operator events | The box-facing API (`/api/v1/offsite/register-key…`) answers with NO credential — pinned by `TestOffsiteKeyEndpoints_AuthAndNoPasswordInAnyResponse`. |
|
||||||
| `offsitekeys` snapshot watch + `store.Get/SetOffsiteSnapshotListing` / `OffsiteRunTimesSince` (v0.158.0, R-895 slice 2) | hub/internal/offsitekeys/snapshot_watch.go, hub/internal/store/offsite_snapshots.go | — | Judging the 07:10 `snapshots/` listing against yesterday's (baseline in `hub_settings`) and the box's reported runs | Raises `offsite_snapshots_vanished` / `_unexplained` — operator-only and hub-origin (NOT in `allowedEventTypes`). The first listing is a baseline; a move-aside or RESET drops it. |
|
| `offsitekeys` snapshot watch + `store.Get/SetOffsiteSnapshotListing` / `OffsiteRunTimesSince` (v0.158.0, R-895 slice 2) | hub/internal/offsitekeys/snapshot_watch.go, hub/internal/store/offsite_snapshots.go | — | Judging the 07:10 `snapshots/` listing against yesterday's (baseline in `hub_settings`) and the box's reported runs | Raises `offsite_snapshots_vanished` / `_unexplained` — operator-only and hub-origin (NOT in `allowedEventTypes`). The first listing is a baseline; a move-aside or RESET drops it. |
|
||||||
| `accesslog.New` / `Middleware` / `Mark` (v0.158.0, R-1140) | hub/internal/accesslog/accesslog.go | `Mark(ctx, kind)` | The hub's own access log (`<data_dir>/access-log/`, 90 days) and marking the identity KIND an auth check established | Never log a value: the path only (no query; `/bind/<token>` and 32+-char segments redacted). A new auth path calls `Mark` with its kind. A logging failure never fails a request. |
|
| `accesslog.New` / `Middleware` / `Mark` (v0.158.0, R-1140) | hub/internal/accesslog/accesslog.go | `Mark(ctx, kind)` | The hub's own access log (`<data_dir>/access-log/`, 90 days) and marking the identity KIND an auth check established | Never log a value: the path only (no query; `/bind/<token>` and 32+-char segments redacted). A new auth path calls `Mark` with its kind. A logging failure never fails a request. |
|
||||||
|
| the timed SQLite driver `sqlite-timed` + `SetSlowStatementThreshold` (v0.178.0, R-1346) | hub/internal/store/timeddriver.go | `store.New` opens it — nothing to call | Names a statement over 1 s: `[WARN] slow statement: <ms> (over 1s): <SQL>` | The SQL text only, never a bound value. Put a heavy statement's SQL in a const and pin its plan (`r1346_speed.go`, `TestR1346_HeavyStatementsUseTheirIndex`): on the hub's network disk a whole-table read is seconds to minutes. |
|
||||||
| `(*Store).SaveOneTimeSecret` / `OffsitePassword` / `SealLegacyOffsiteSecrets` | hub/internal/store/offsite_seal.go | — | Storing / reading the sub-account password | **Sealed AES-256-GCM; no key → refused (fail-closed).** Under `go test` every store gets a fixed key (`testing.Testing()`); production needs `OFFSITE_SECRET_KEY`. Never serve the value to a box. |
|
| `(*Store).SaveOneTimeSecret` / `OffsitePassword` / `SealLegacyOffsiteSecrets` | hub/internal/store/offsite_seal.go | — | Storing / reading the sub-account password | **Sealed AES-256-GCM; no key → refused (fail-closed).** Under `go test` every store gets a fixed key (`testing.Testing()`); production needs `OFFSITE_SECRET_KEY`. Never serve the value to a box. |
|
||||||
| `(*Store).sealAtRest` / `openAtRest` / `sealAPIKey` / `apiKeyHash` / `SealLegacyBoxSecrets` | hub/internal/store/r879_box_seal.go | — | Any box-facing secret column (`hosts.api_key`, `customer_configs.api_key` / `retrieval_password`, `host_pbs_secrets.value`) | **R-879: same seal; a looked-up key is matched on `api_key_hash`, never opened** — box auth needs no sealing key. A sealed value that does not open sets `SecretsUnreadable` (field ""): serve/compare paths 500 on it and `SaveCustomerConfig` / `UpsertHost` refuse it (a load-modify-save would blank the secret). A new writer of an API key must write the hash in the same statement (`sealAPIKey`). Pinned by `store/r879_box_seal_test.go`, `api/r879_box_seal_test.go`. |
|
| `(*Store).sealAtRest` / `openAtRest` / `sealAPIKey` / `apiKeyHash` / `SealLegacyBoxSecrets` | hub/internal/store/r879_box_seal.go | — | Any box-facing secret column (`hosts.api_key`, `customer_configs.api_key` / `retrieval_password`, `host_pbs_secrets.value`) | **R-879: same seal; a looked-up key is matched on `api_key_hash`, never opened** — box auth needs no sealing key. A sealed value that does not open sets `SecretsUnreadable` (field ""): serve/compare paths 500 on it and `SaveCustomerConfig` / `UpsertHost` refuse it (a load-modify-save would blank the secret). A new writer of an API key must write the hash in the same statement (`sealAPIKey`). Pinned by `store/r879_box_seal_test.go`, `api/r879_box_seal_test.go`. |
|
||||||
|
|
||||||
@@ -96,7 +97,7 @@
|
|||||||
| `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
|
| `(*Server).configFormData` (v0.49.0) | hub/internal/web/configs.go (~L430) | `(r, isNew, cfg, overrides, errMsg) configFormView` | The ONE view-model builder for the customer config form (standalone chrome + the customer page Edit tab) | `overrides=nil` → parses the STORED cfg.ConfigJSON; pass the SUBMITTED map on the update validation-error re-render or typed values reset (red-proofed). |
|
||||||
| `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" <configFormView>}}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `</form>` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `<script>`. |
|
| `config_form_body` sub-template (v0.49.0) | hub/internal/web/templates/config_form_body.html | `{{template "config_form_body" <configFormView>}}` | Rendering the config form on ANY surface (config_form.html chrome + customer Edit tab) | The floor/geo/danger cards on the Edit tab are SIBLINGS after `</form>` — never nest a form inside it (breaks the offsite/PBS formaction sub-buttons). Includes the F5 in-flight `<script>`. |
|
||||||
| `handleHostEscrowGet` + `escrowSelfServiceRetrieval` | hub/internal/api/handler.go | `GET /hosts/{id}/escrow` | The box-authenticated MIRROR of the escrow PUT — self-scoped by the per-host key. **`escrowSelfServiceRetrieval` is THE single decision point** for whether a box may read its own blob without operator-armed recovery mode (§8.2 vs §8.3, R-199): flip it, do not scatter the condition. **Never merge this with `dr.go`'s `handleReEnroll`/`handleGetRestoreDirective`** — those rotate the host API key and serve the K-escrow and directive too, and keep their recovery-mode gate (pinned by `TestEscrowGet_OperatorDRPathUnchanged`). Every successful retrieval MUST raise `escrow_blob_served` before the bytes leave; that audit row is the mitigation the trade rests on. |
|
| `handleHostEscrowGet` + `escrowSelfServiceRetrieval` | hub/internal/api/handler.go | `GET /hosts/{id}/escrow` | The box-authenticated MIRROR of the escrow PUT — self-scoped by the per-host key. **`escrowSelfServiceRetrieval` is THE single decision point** for whether a box may read its own blob without operator-armed recovery mode (§8.2 vs §8.3, R-199): flip it, do not scatter the condition. **Never merge this with `dr.go`'s `handleReEnroll`/`handleGetRestoreDirective`** — those rotate the host API key and serve the K-escrow and directive too, and keep their recovery-mode gate (pinned by `TestEscrowGet_OperatorDRPathUnchanged`). Every successful retrieval MUST raise `escrow_blob_served` before the bytes leave; that audit row is the mitigation the trade rests on. |
|
||||||
| `demoteCurrentEscrowTx` (+ `(*Store).SaveHostEscrow`) | hub/internal/store/store.go (~L2597/~L2612) | `(tx, hostID) (int64, error)` / `(hostID, blob, fp, posture, createdAt, resticPwSHA) (superseded bool, prevResticPwSHA string, err error)` | **THE ONE escrow row-copy routine** — used by the re-escrow retention AND by `DeleteHost`'s custody demotion; never write a second one | **It must copy BOTH sealed artifacts** — `blob` (K-escrow / PBS key) and `identity_blob` (the age bundle carrying the offsite restic repo password). Omitting the second is R-198: two months of retaining the wrong key, with the ceremony as the destroying act. **Ordering it depends on:** `SaveHostDRBundle` writes `identity_blob` AFTER `SaveHostEscrow` returns, so the demote sees the PREVIOUS generation — invert that and the retained bytes are the new blob under the old hash. Pinned by `TestSaveHostEscrow_RetainsIdentityBlob` + `TestDeleteHost_DemotesIdentityBlob` (both callers). `prevResticPwSHA` feeds R-197's changed-key signal; it is a hash and never leaves the store. |
|
| `demoteCurrentEscrowTx` (+ `(*Store).SaveHostEscrow` / `SaveHostEscrowWithIdentity`) | hub/internal/store/store.go (~L2597/~L2612) | `(tx, hostID) (int64, error)` / `(hostID, blob, fp, posture, createdAt, resticPwSHA) (superseded bool, prevResticPwSHA string, err error)` | **THE ONE escrow row-copy routine** — used by the re-escrow retention AND by `DeleteHost`'s custody demotion; never write a second one | **It must copy BOTH sealed artifacts** — `blob` (K-escrow / PBS key) and `identity_blob` (the age bundle carrying the offsite restic repo password). Omitting the second is R-198: two months of retaining the wrong key, with the ceremony as the destroying act. **Ordering it depends on:** since R-1254 (unreleased, next hub v0.170.0) `SaveHostEscrowWithIdentity` writes the key, its hash and `identity_blob` in ONE transaction, with the demote FIRST inside it, so the demote sees the PREVIOUS generation — invert that and the retained bytes are the new blob under the old hash. `SaveHostEscrow` is that function with no identity blob; `SaveHostDRBundle` (a second statement) is no longer called by the escrow PUT. Pinned by `TestR1254_*`. Pinned by `TestSaveHostEscrow_RetainsIdentityBlob` + `TestDeleteHost_DemotesIdentityBlob` (both callers). `prevResticPwSHA` feeds R-197's changed-key signal; it is a hash and never leaves the store. |
|
||||||
| `(*Store).CountHostArtifacts` / `DeleteHost` | hub/internal/store/store.go (~L1640/~L1690) | `(hostID) (HostArtifacts, error)` / `(hostID, deleteEscrow bool) error` | Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (`ErrHostEscrowPresent`, tx never starts). log_bundles die by `scope_id == host_id` ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
|
| `(*Store).CountHostArtifacts` / `DeleteHost` | hub/internal/store/store.go (~L1640/~L1690) | `(hostID) (HostArtifacts, error)` / `(hostID, deleteEscrow bool) error` | Host-delete impact preview + the ONE-transaction cascade | ONLINE gate lives in the handler, escrow gate in the store (`ErrHostEscrowPresent`, tx never starts). log_bundles die by `scope_id == host_id` ONLY (customer-scoped bundles survive). The wg_peers delete is INSIDE the tx — never split it out. |
|
||||||
| `(*Server).commitCustomerReset` (v0.69.0) | hub/internal/web/customer_reset.go (~L165) | `(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError` | THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the `customer_resets` journal | Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). `purgeEscrow` governs ONLY whether `PurgeCustomerResetDBState` destroys retained custody: standalone RESET passes the operator's `escrow_ack`; the DELETE cascade passes **false** so custody dies exactly once, in its leg 3. Returns a `resetLegError` carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
|
| `(*Server).commitCustomerReset` (v0.69.0) | hub/internal/web/customer_reset.go (~L165) | `(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError` | THE committed RESET sequence — external teardown FIRST (Hetzner, PBS), then claim → descriptor → DB purge, each leg stamped into the `customer_resets` journal | Owns NO gate, NO audit event, NO journal open/close, NO redirect — those are the caller's (the two callers differ there). `purgeEscrow` governs ONLY whether `PurgeCustomerResetDBState` destroys retained custody: standalone RESET passes the operator's `escrow_ack`; the DELETE cascade passes **false** so custody dies exactly once, in its leg 3. Returns a `resetLegError` carrying the leg name + the exact status/message the standalone handler has always returned — do not re-word them. |
|
||||||
| `(*Store).CustomerResidue` / `PurgeCustomerResidue` (v0.70.0) | hub/internal/store/customer_delete.go | `(customerID) (*CustomerResidue, error)` / `(customerID) error` | Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind | **`GetCustomers()` is REPORT-derived** — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers’ work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared `residueQueries` list so a table can never be counted-but-not-purged. Includes `appliance_registrations` + `selfbind_tokens` (credential-bearing, not telemetry). NEVER touches `events`, `notification_log`, `host_deletions`, `customer_resets`. |
|
| `(*Store).CustomerResidue` / `PurgeCustomerResidue` (v0.70.0) | hub/internal/store/customer_delete.go | `(customerID) (*CustomerResidue, error)` / `(customerID) error` | Counting + purging the report-derived state and the credential-bearing bindings a deleted customer leaves behind | **`GetCustomers()` is REPORT-derived** — until the reports are gone the customer stays on the Customers list AND stays in the staleness/offsite checkers’ work list, so a deleted customer keeps emailing the operator. Both funcs walk ONE shared `residueQueries` list so a table can never be counted-but-not-purged. Includes `appliance_registrations` + `selfbind_tokens` (credential-bearing, not telemetry). NEVER touches `events`, `notification_log`, `host_deletions`, `customer_resets`. |
|
||||||
@@ -124,6 +125,7 @@
|
|||||||
| `offsite.DeliveryStateFor` (+ `DeliveryStatus`) | hub/internal/offsite/delivery.go | `(st, customerID) (DeliveryStatus, error)` | THE R-70 offsite last-mile detector — one implementation for every consumer (customer card `deliveryViewFor`, `monitor.OffsiteDeliveryChecker` event + R-71c heal) | Precedence: `applied` (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + `StaleStagedSince` flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. |
|
| `offsite.DeliveryStateFor` (+ `DeliveryStatus`) | hub/internal/offsite/delivery.go | `(st, customerID) (DeliveryStatus, error)` | THE R-70 offsite last-mile detector — one implementation for every consumer (customer card `deliveryViewFor`, `monitor.OffsiteDeliveryChecker` event + R-71c heal) | Precedence: `applied` (latest report has offsite) wins over every secret-row shape; applied+unconsumed-staged = applied + `StaleStagedSince` flag (demo-felhom's live specimen). Never add a sibling derivation — consumers read THIS. |
|
||||||
| `(*Store).GetOneTimeSecretInfo` / `LastEventAt` / `LatestReportOffsitePresence` / `CountReportsOffsiteSince` | hub/internal/store/store.go | `(customerID) (*OneTimeSecretInfo, error)` / `(customerID, eventType) (time.Time, error)` / … | Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | `GetOneTimeSecretInfo` never selects the value column — keep it that way. `SetOneTimeSecretTimesForTest` is the back-dating seam (PBSDR pattern). |
|
| `(*Store).GetOneTimeSecretInfo` / `LastEventAt` / `LatestReportOffsitePresence` / `CountReportsOffsiteSince` | hub/internal/store/store.go | `(customerID) (*OneTimeSecretInfo, error)` / `(customerID, eventType) (time.Time, error)` / … | Detector inputs + DURABLE event-cooldown source (events table survives restarts — prefer over in-memory maps for hub-emitted checker events) | `GetOneTimeSecretInfo` never selects the value column — keep it that way. `SetOneTimeSecretTimesForTest` is the back-dating seam (PBSDR pattern). |
|
||||||
| `monitor.OffsiteDeliveryChecker` + `OffsiteReissuer` | hub/internal/monitor/offsite_delivery.go | `NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger)` | R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in `maybeHeal`: re-reads the secret row at act time and refuses over an UNCONSUMED row — `SaveOneTimeSecret` clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. |
|
| `monitor.OffsiteDeliveryChecker` + `OffsiteReissuer` | hub/internal/monitor/offsite_delivery.go | `NewOffsiteDeliveryChecker(st, reissuer, onEvent, logger)` | R-70 stuck event + R-71c self-heal on the shared 60 s ticker | THE R-39(a) GUARD lives in `maybeHeal`: re-reads the secret row at act time and refuses over an UNCONSUMED row — `SaveOneTimeSecret` clobbers by design (Re-issue depends on supersede); never "fix" the store, never bypass the guard. reissuer nil = heal disabled (no provisioner) — required, else a heal-event fires for a silent no-op. |
|
||||||
|
| `monitor.HeldAppsChecker` + `store.UpsertHeldApp`/`ClearHeldApp`/`ListHeldApps` + `HeldAppGrace`/`HeldAppWhoClears` (D53, unreleased) | hub/internal/monitor/held_apps.go · hub/internal/store/held_apps.go · hub/internal/web/held_apps.go | `NewHeldAppsChecker(st, graceOverride, onEvent, logger)`; `.Check()` on the 60 s sweep | `08` §10: the boxes' held apps — the mirror (fed by `app_held`/`app_hold_cleared` in `handleEvent`), the „Held apps" list on the host page and the Hosts page, one operator reminder after the grace per reason, then weekly | **The hub never clears a hold** — a row goes only on the box's `app_hold_cleared`. The reminder is recorded BEFORE it is sent (a restart cannot mail twice); a row the box stopped repeating (> 50 h) is listed, never reminded. `app_held_reminder` is operator-only and per-app cooldown. `graceOverride` is a TEST configuration only (`alerting.held_app_grace_override`). Pinned by `TestD53_*` (monitor, api, web, notify) |
|
||||||
| `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. **R-86 (2026-08-03): the window is PER TIER, not one constant.** `restoreProvenWindow(tier, observed, ok)` = `clamp(4 × max(observed, declared), floor 7d, cap 12d)`, where `declared` is that tier's own backup-freshness threshold (`backupStaleAfter` 26 h / `offsiteBackupStaleAfter` 8 d — reuse those, never a second opinion) and `observed` comes from `observedArchiveIntervals` over the retained window. **Observation may only WIDEN**: a gap shorter than the declared rhythm is routine (a retry, a heal, a catch-up) and a live box proved it — demo-felhom's two PBS snapshots sit 8 h 54 m apart, which would read a WEEKLY tier as nine-hourly and re-create the false alarm. The cap keeps the window strictly inside offsite retention. `assessRestoreProven` takes the window as an argument and **every reason string names it** (R-100's corollary). |
|
| `monitor.RestoreTestChecker` + `assessRestoreProven` | hub/internal/monitor/restoretest.go | `NewRestoreTestChecker(st, onEvent, logger)`; `.Check()` | R-85: turns a restore-test result into a SIGNAL — it was a `[WARN]` log line and nothing else, even for the tier already being tested | **TWO event types, never merged**: `restore_test_failed` (broken now, error) vs `restore_test_stale` (unverified — *not* known-broken, warning). Merging collapses the second into the first, and the second is what quietly becomes the first. **Anchored on R-81** (`assessRestoreProven` reuses `backupAssessment`/`verdict*`): a never-proven tier on a newborn box is UNKNOWN, not FAILED. Per-tier proof comes from the hub's RETAINED WINDOW — the agent reports only its latest run, so the latest report alone cannot answer "when was the OTHER tier last proven?". Operator-tier only: **no `customerMessages` entry** — do not add one without copy review. **R-86 (2026-08-03): the window is PER TIER, not one constant.** `restoreProvenWindow(tier, observed, ok)` = `clamp(4 × max(observed, declared), floor 7d, cap 12d)`, where `declared` is that tier's own backup-freshness threshold (`backupStaleAfter` 26 h / `offsiteBackupStaleAfter` 8 d — reuse those, never a second opinion) and `observed` comes from `observedArchiveIntervals` over the retained window. **Observation may only WIDEN**: a gap shorter than the declared rhythm is routine (a retry, a heal, a catch-up) and a live box proved it — demo-felhom's two PBS snapshots sit 8 h 54 m apart, which would read a WEEKLY tier as nine-hourly and re-create the false alarm. The cap keeps the window strictly inside offsite retention. `assessRestoreProven` takes the window as an argument and **every reason string names it** (R-100's corollary). |
|
||||||
| `(*Server).applyPBSDR` + `mergePBSDR`/`readPBSDR` | hub/internal/web/pbsdr.go | `(ctx, r, cfg) error` | The config form's DR-tier section → HOST desired_json `pbs_dr` descriptor + generation bump | Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by `cfg.DRTier` (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. |
|
| `(*Server).applyPBSDR` + `mergePBSDR`/`readPBSDR` | hub/internal/web/pbsdr.go | `(ctx, r, cfg) error` | The config form's DR-tier section → HOST desired_json `pbs_dr` descriptor + generation bump | Descriptor lives in the host desired_json, NOT ConfigJSON (buildConfigJSON drops foreign keys on re-save). v0.51.0: driven by `cfg.DRTier` (set from the form BEFORE applyOffsite/applyPBSDR); UNMET preconditions are honest waiting stages (save succeeds), REAL failures stay fail-closed; already-provisioned = success-no-op (red-proofed); disable keeps the ep0 tenancy. |
|
||||||
| `(*Server).pbsdrProvisionAtom` + `PBSDRAutoProvision` | hub/internal/web/pbsdr.go | `(ctx, customerID, host, storageID) (blocked string, err error)` / `(ctx, customerID)` | The shared fresh-provision cascade atom; the WG-registration hook target (api `SetWGRegisteredHook`, wired in hub/cmd/hub/main.go when tenantsync is on) | `blocked != ""` = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. |
|
| `(*Server).pbsdrProvisionAtom` + `PBSDRAutoProvision` | hub/internal/web/pbsdr.go | `(ctx, customerID, host, storageID) (blocked string, err error)` / `(ctx, customerID)` | The shared fresh-provision cascade atom; the WG-registration hook target (api `SetWGRegisteredHook`, wired in hub/cmd/hub/main.go when tenantsync is on) | `blocked != ""` = waiting stage (never an error); the hook runs in a detached goroutine and must never fail registration. Scenario-A e2e test: TestPBSDR_AutoProvisionOnWGRegistration. |
|
||||||
@@ -158,6 +160,12 @@
|
|||||||
| `scheduleDaily` | hub/cmd/hub/main.go (~L449) | `(ctx, name, "HH:MM", fn, logger)` | Daily jobs in Europe/Budapest (prune etc.) | Blocking — run as goroutine. `parseHM` returns 0,0 (midnight) on bad input. |
|
| `scheduleDaily` | hub/cmd/hub/main.go (~L449) | `(ctx, name, "HH:MM", fn, logger)` | Daily jobs in Europe/Budapest (prune etc.) | Blocking — run as goroutine. `parseHM` returns 0,0 (midnight) on bad input. |
|
||||||
| `hu_grep.py` | scripts/hu_grep.py | `PATTERN --anchor ASCII [--negative TEXT] PATH…` | ANY search for Hungarian (accented) text in files — prints file:line hits, and refuses to report a zero unless an ASCII anchor hits and a negative control misses (R-364) | Reads bytes in Python, never via a shell; refuses a pattern that arrived transformed (octal escapes, U+FFFD). Exit 0 found · 1 tested zero · 2 refused |
|
| `hu_grep.py` | scripts/hu_grep.py | `PATTERN --anchor ASCII [--negative TEXT] PATH…` | ANY search for Hungarian (accented) text in files — prints file:line hits, and refuses to report a zero unless an ASCII anchor hits and a negative control misses (R-364) | Reads bytes in Python, never via a shell; refuses a pattern that arrived transformed (octal escapes, U+FFFD). Exit 0 found · 1 tested zero · 2 refused |
|
||||||
|
|
||||||
|
### The list of everything we ship, with its upstream (R-1303, scripts/)
|
||||||
|
- `scripts/security_watch_inventory.py` — THE reader of every pin (catalog `image:` + `.felhom.yml` `upstream:`,
|
||||||
|
controller `infra.go` + `upstream.txt`, every Dockerfile `FROM` with its `# upstream:` comment) and of the source
|
||||||
|
grammar. The nightly `scripts/security_watch.py` and the catalog's and controller's `upstream` gates all import or call
|
||||||
|
it — never copy its parser.
|
||||||
|
|
||||||
## 2. Canonical patterns (copy structure from THE named file)
|
## 2. Canonical patterns (copy structure from THE named file)
|
||||||
|
|
||||||
| Pattern | Canonical file | Key traits |
|
| Pattern | Canonical file | Key traits |
|
||||||
|
|||||||
@@ -1,27 +1,31 @@
|
|||||||
# STATUS — what works, what's broken, what's next
|
# STATUS — what works, what's broken, what's next
|
||||||
|
|
||||||
**Ready for Tester 2? Yes — tell the friend go (hand-over list: `documentation/pilot/TESTER2-handover.md`).**
|
**Ready for Tester 2? Yes — tell the friend go (`documentation/pilot/TESTER2-handover.md`).**
|
||||||
|
|
||||||
**Updated 2026-10-23 (morning).** Your three picks are built (the tunnel repairs itself with a new key, the hub skips a keyless deleted install, the controller repository stays public), and the night copy can no longer outlive the controller. Four fresh reviews found 34 points, every real one fixed or a row; the test night PASSED (no failed check at night, every night job passed, nothing lost). Nothing was released or deployed; the morning list has the order, and the DooPlex survey asks four questions (D32–D35).
|
**Updated 2026-11-03.** Every box runs controller 0.350.0 with Traefik v3.7.17; the hole is closed, proven on the scratch box. Golden 0.350.0 and agent 0.172.0 are vouched (10:18 CET), hub 0.179.0 is live, and Paperless 3, n8n 3, Mealie and Termix are offered.
|
||||||
|
|
||||||
## What works
|
## What works
|
||||||
|
- **The Plex attack did not reach Felhom's secrets** (R-1241).
|
||||||
- A box installs from nothing with the public installer, links to the hub, and moves to the floor by itself (map A1, A3, A4).
|
- A box installs from nothing with the public installer, links to the hub, and moves to the floor by itself (map A1, A3, A4).
|
||||||
- 65 catalog apps install, update one tested step at a time, and a failed update undoes itself (map B1, B3, B4, B6).
|
- 65 catalog apps install, update one tested step at a time, and a failed update undoes itself (map B1, B3, B4, B6).
|
||||||
- Backups: database dumps, a second-drive copy, an off-site copy and whole-box copies; restore tests run by themselves (map C1–C4, A17, A18).
|
- Backups: database dumps, a second-drive copy, an off-site copy and whole-box copies; restore tests run by themselves (map C1–C4, A17, A18).
|
||||||
- A restore brings an app back with its data and its database password; a failed or cut restore puts the app back as it was — off-site live, local on the test image (map C5, D6, D16).
|
- A restore brings an app back with its data; a failed or cut restore puts it back as it was; one that does not start says so (map C5, D6, D7, D16).
|
||||||
- After a power cut the box starts by itself, cleans up a half-written backup, and Immich repairs broken photos (map C10, F12).
|
- After a power cut the box starts by itself and cleans up a half-written backup (map C10, F12).
|
||||||
- A second drive is offered as the backup target; media apps see every drive; a one-disk box gets its own data area (map D1, D13, D15).
|
- A second drive is offered as the backup target; media apps see every drive (map D1, D13, D15).
|
||||||
- The three test boxes answer only over https; the hub's key opens nothing on a box; password guesses are counted (map E12, E13, E17).
|
- The three test boxes answer only over https; the hub's key opens nothing on a box (map E12, E13, E17).
|
||||||
- The hub mails alarms and recoveries, and keeps one whole-box copy per week on ep0; CC uses its own limited token, not your password (map F1, F11, G15).
|
- A hacked app that uploads or computes more than it explains alarms only you (map F13).
|
||||||
|
- An urgent security fix reaches the boxes at night after one test on the test box; the box backs up first and undoes a failed update (map B23).
|
||||||
|
- The hub mails alarms and recoveries and keeps a weekly whole-box copy on ep0 (map F1, F11, G15).
|
||||||
|
|
||||||
## What is broken
|
## What is broken
|
||||||
- **P1:** about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours).
|
- **P1:** any hacked app on DooPlex can read the hub's nightly backups, and maybe the signing keys (R-1241, homelab session); about 12 homelab logins still use a password that was public, and the public Gitea repo is not decided (R-925, yours).
|
||||||
- Tester 2 is off since 2026-10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862).
|
- Tester 2 off since 10-04: old emergency password, no off-site check, no bootstrap (R-1139, R-1104, R-862). Scratch box: no agent features (R-1083).
|
||||||
- DooPlex: the failed disk is still in the box, its alarms silenced (R-934); something sets the shared checkout to „bare" (R-935).
|
- DooPlex: something sets the shared checkout to „bare" (R-935).
|
||||||
- The scratch box cannot use agent features (R-1083).
|
- A box's event is lost if the hub is away for 9 seconds; a DooPlex restart can swallow a mail (R-1361).
|
||||||
|
|
||||||
## What we work on
|
## What we work on
|
||||||
- Dated checks, run by CC. 2026-10-23: no app stop when the host is busy (R-1173). 2026-10-26: each hub daily job ran once on the clock-change day (R-1183). 2026-10-27: the weekly clean-up window (R-435; the logs of the last one were lost to restarts).
|
- HTTP2Shell waiters: cal.com, Calibre-Web, RomM have no fix; SparkyFitness has one on a new main version (R-785, behind R-784).
|
||||||
|
- Dated checks, run by CC. 2026-11-06: ask you to retire DooPlex's old copy of the website (R-1228).
|
||||||
|
|
||||||
## Waiting on you — decide
|
## Waiting on you — decide
|
||||||
|
|
||||||
@@ -29,36 +33,30 @@ Newest first. CC's pick is in the „Pick" column. Each row also has its full te
|
|||||||
|
|
||||||
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|
| # | Row | Question | Pick (why) | The other option costs | If you do nothing | Blocked |
|
||||||
|---|---|---|---|---|---|---|
|
|---|---|---|---|---|---|---|
|
||||||
| D36 | R-1225, R-1226 | Two website sentences before the publish: „8 GB runs the basics; the photo app needs 16 GB” and „what you open from outside the home passes Cloudflare's network, readable there; the backups leave only encrypted”? | **Yes, both** (the patch's own aim is the whole truth) | — | the patch publishes with both promises a little too strong | — |
|
| D66 | R-1365 | Should a security-watch source that stays down mail again every 7 days, as an open notice does? Today it mails once, then only the Monday summary lists it. | **Yes** (a real outage then cannot be forgotten; the seven pins that can never be read no longer count, so it stays quiet when all is well) | no: one line in a long Monday mail stays the only reminder | a source that stays down mails once; the Monday list still shows it | — |
|
||||||
| D35 | R-1228 | After the release week, serve the website and the installer from Cloudflare, like the ISO? | **Yes, later** (no new installs stop when DooPlex is down) | keep them home: after a DooPlex loss, no install until it is back | the same | — |
|
| D38 | R-1235 | Put the newest golden (637 MB) into the nightly release copy on ep0 too? | **Yes** (a new install after a DooPlex loss needs it; it fits under the 3 GB cap) | leave it out: after a loss, no new install until a golden is rebuilt and vouched | the same | — |
|
||||||
| D34 | R-232 | Copy the hub database to ep0 every 6 hours instead of every night? | **Yes** (0 €, about 30 minutes, one DooPlex change) | keep nightly: up to a day of hub changes lost, e.g. a new box's recovery package | the same | — |
|
| D37 | R-1236 | Raise Gitea's memory limit from 1 GiB to 2 GiB? It was killed for memory 4 times in 17 days, once during today's release. | **Yes** (DooPlex has the RAM; a DooPlex change, so yours) | keep 1 GiB: a restart in release hours fails box updates and CI for a minute | the same | — |
|
||||||
| D33 | R-1227 | Copy the released images and agent packages to ep0 every night? | **Yes** (0 €, about 2 hours of CC, a few GB on ep0) | a timed rebuild drill instead: about 3 hours once, and rebuilt images are not the same | after a DooPlex loss no box updates, reinstalls or installs until everything is rebuilt | — |
|
|
||||||
| D32 | R-923, R-924 | Print the break-glass sheet now (eight keys and logins + ep0's fingerprint) and keep it outside the home? | **Yes** (30 minutes; nothing else opens the off-site copies after a loss) | an encrypted USB stick outside the home: ~10 €, 1 hour, one more passphrase | after a DooPlex loss no off-site copy opens | — |
|
|
||||||
| D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — |
|
| D9 | R-1062 | Split the N100's USB drive in two, for multi-drive tests? | **No — keep one drive** (the lived-in box keeps its nightly whole-box backup) | Split: the box loses its local whole-box backup, or you buy a second drive first | it stays one drive; multi-drive tests stay on Tester 1 | — |
|
||||||
| D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — |
|
| D7 | R-990, R-989 | Rule on the forum ideas and the selfh.st ideas now, or park them? | **Park them** (P4; nothing waits on them) | Rule now: about one hour of reading | nothing changes; the research gets old | — |
|
||||||
|
|
||||||
## Waiting on you — do
|
## Waiting on you — do
|
||||||
|
|
||||||
**Still needed? (yes/no)** — CC could not check these. Answer yes or no per line.
|
**Still needed? (yes/no)** — answer per line.
|
||||||
1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). Since 2026-10-16.
|
1. Look at `roms/roms/` on the N100's drive in the file manager (R-1099). (since 10-16)
|
||||||
2. The N100 sheet, about 30 minutes: Plex, the admins in Prowlarr, Sonarr and Radarr, your download sources (`documentation/audits/n100-lived-in-2026-10-15/SHEET.md`). Since 2026-10-15.
|
2. The N100 sheet, about 30 minutes: Plex, the admins in Prowlarr, Sonarr and Radarr, your download sources (`documentation/audits/n100-lived-in-2026-10-15/SHEET.md`). (since 10-15)
|
||||||
3. Look at these pages on demo-hp when you have time: Calibre-Web („Adatok és mentés"), the file manager, AdventureLog, Plex, the app page, the Plex and Sonarr guides („Útmutató"). Since 2026-10-16.
|
3. Look at these demo-hp pages when you have time: Calibre-Web, the file manager, AdventureLog, Plex, the app page, the Plex and Sonarr guides. (since 10-16)
|
||||||
4. Remove the test server „plex-k6" from your Plex account (plex.tv → Settings → Authorized Devices). Since 2026-10-15.
|
4. Remove the test server „plex-k6" from your Plex account (plex.tv → Settings → Authorized Devices). (since 10-15)
|
||||||
5. Ask Google to index `https://felhom.eu/sajat-felho-tarhely` in Search Console (R-1035). Since 2026-10-14.
|
5. Ask Google to index `https://felhom.eu/sajat-felho-tarhely` in Search Console (R-1035). (since 10-14)
|
||||||
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. Since 2026-10-14.
|
6. BIOS: on the N100, read „After Power Loss"; on the HP, set „Bypass F1 Prompt on Configuration Changes" to Enable. (since 10-14)
|
||||||
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). Since 2026-10-12.
|
7. Pin the first Facebook post (R-917), and look at the new cover in the Facebook phone app (R-919). (since 10-12)
|
||||||
|
**2026-11-03.** 1. From about 11:00 CET: delete host `Tester4-e286d3` on the Hosts page (CC's proof VM, gone). If not: stale mails. 2. To the homelab session: R-1355.
|
||||||
**Morning, 2026-10-23 — in this order.**
|
**2026-10-30.** On the hub's Hosts page: Discard the appliance of 10:44:59 (uuid d3529d2d…) and appliance 49 (CC's proof VMs). If not: dead entries alarm. **2026-10-29 (release run).** 1. (since 10-28) Give the homelab session `audits/http2shell-2026-10-27/HOMELAB-LIST.md` (19 images, R-1328). If not: DooPlex keeps the bad runtimes.
|
||||||
1. Hold or go: CC says go — the test night passed on a fresh 8 GB box (no failed check at night, every night job passed, nothing lost; one gap: no whole-box copy to ep0, R-1232). One question from the review: D31 as built also covers an install that got the whole-box password but no recovery package — its password is rotated without your press (R-1205). CC: go. If not: revert one hub commit before item 4.
|
**2026-10-23 do list.**
|
||||||
2. Release controller 0.340.0 (main after tonight; tested as rc.8 on the scratch box, rc.6 on the test box): the safe night copy, the catch-up fix, R-49, R-1187, R-1189, R-1214. If not: these wait.
|
1. Print the break-glass sheet (`documentation/runbooks/break-glass-sheet.md`, the part from „The sheet”, one A4), fill it by hand, keep it outside the home (D32; R-923, R-924). If not: after a DooPlex loss no off-site copy opens.
|
||||||
3. Release the agent (the tunnel repair, its tunnel field, the root-SSH bundle) and the installer tag with the root-SSH removal (R-1204, R-982); then vouch. If not: a clock-ahead box stays cut off; root keeps its password on the LAN.
|
2. Give the homelab session R-1241, R-1242, the lists at the end of `documentation/audits/plex2shell-felhom-impact-2026-10-23/README.md` and `documentation/audits/dooplex-alarms-2026-10-24/README.md`, and an upload alarm on DooPlex's `eno1` (~100 Mbit/s line; the flood peaked at 905). If not: the next hacked app on DooPlex can read the hub's backups.
|
||||||
4. Build and deploy hub 0.167.0 (D31, the tunnel alarm and its clearing). If not: a dead tunnel stays silent.
|
|
||||||
5. Publish the website patch (D3, D5), with D36's two sentences if yes. If not: the old wording stays.
|
|
||||||
6. On the hub's Hosts page: Discard appliance 49, and delete host Tester4-16fdef (tonight's test box). If not: two dead entries alarm.
|
|
||||||
7. The DooPlex hour: D32–D35 in the table. If not: a DooPlex loss stays as the survey describes.
|
|
||||||
**The rest**
|
**The rest**
|
||||||
- Pull the failed disk Z6A07P2G out of DooPlex (R-934). If not: its alarms return on 11 November. Since 2026-10-12.
|
- System poster needs a refresh: the new 01:40 „release copy” arrow from DooPlex to ep0 (D33); the website now on Cloudflare, only its mail form on DooPlex (D35). If not: the drawing is out of date.
|
||||||
- Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). Since 2026-10-09.
|
- Rotate the reused homelab logins and decide the Gitea repo's visibility (R-925, P1). (since 10-09)
|
||||||
- Two leftovers on DooPlex: the Facebook `first-post/` folder in `leftovers/2026-10-14/` (keep, commit or delete); the `deny`-list edit in homelab-manifests' `.claude/settings.json` (commit or drop). If not: they stay. Since 2026-10-14.
|
- Two DooPlex leftovers: `leftovers/2026-10-14/first-post/` (keep or delete); the `deny`-list edit in homelab-manifests' `.claude/settings.json` (commit or drop). (since 10-14)
|
||||||
- Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784); print the break-glass sheet and the signing keys (R-923, R-924).
|
- Before the first paying customer: Tandoor's permission (R-789); a lawyer for the licence list and the terms (R-802, R-813); SparkyFitness by e-mail (R-784).
|
||||||
- Later (yours, not urgent): R-1229 (ArgoCD's sync settings into git), R-862 (Tester 2's bootstrap, when it is back), R-504, R-1071, R-1073, R-1077, R-1042, R-132, R-908, R-779, R-904, R-232, R-433, R-933, R-213, R-832, R-388, R-882, R-883, R-814, R-1056, R-1164, R-930, R-920, R-902.
|
- Later (yours, not urgent): R-1248 (a hard upload cap per app), R-1229 (ArgoCD's sync settings into git), R-862 (Tester 2's bootstrap, when it is back), R-504, R-1071, R-1073, R-1077, R-1042, R-132, R-908, R-779, R-904, R-232, R-433, R-933, R-213, R-832, R-388, R-882, R-883, R-814, R-1056, R-1164, R-930, R-920, R-902.
|
||||||
@@ -80,18 +80,18 @@
|
|||||||
| <a id="a4"></a>[A4](capability-map-detail/A.md#a4) | Bare-metal Felhom ISO (blank hardware → zero-touch auto-install → first-boot `host-install`); selectable UEFI loader; universal secret-free / operator-bind mode | scripts v1.19.0 (`scripts/iso/`) + hub v0.62.0 + assistant container | **PROVEN-LIVE** | N100 (2026-07-18) and HP t740 (2026-07-21) | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18); `audits/SPIKE-baremetal-iso-2026-07-16.md` (2026-07-16) |
|
| <a id="a4"></a>[A4](capability-map-detail/A.md#a4) | Bare-metal Felhom ISO (blank hardware → zero-touch auto-install → first-boot `host-install`); selectable UEFI loader; universal secret-free / operator-bind mode | scripts v1.19.0 (`scripts/iso/`) + hub v0.62.0 + assistant container | **PROVEN-LIVE** | N100 (2026-07-18) and HP t740 (2026-07-21) | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18); `audits/SPIKE-baremetal-iso-2026-07-16.md` (2026-07-16) |
|
||||||
| <a id="a5"></a>[A5](capability-map-detail/A.md#a5) | Box survives a wrong-NIC install: hub-unreachable first boot → legible Hungarian console screen (NIC table + remedy) + NIC sweep self-heal (bounded DHCP + hub probe per NIC, success-only persist), and the baked root password is operator-knowable (`<iso>.rootpw.txt`) | scripts v1.24.0 (`felhom-bootstrap.sh`, `build-felhom-iso.sh`) | **PROVEN-LIVE** | nested drill; metal proof pending | `audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` (2026-07-22) |
|
| <a id="a5"></a>[A5](capability-map-detail/A.md#a5) | Box survives a wrong-NIC install: hub-unreachable first boot → legible Hungarian console screen (NIC table + remedy) + NIC sweep self-heal (bounded DHCP + hub probe per NIC, success-only persist), and the baked root password is operator-knowable (`<iso>.rootpw.txt`) | scripts v1.24.0 (`felhom-bootstrap.sh`, `build-felhom-iso.sh`) | **PROVEN-LIVE** | nested drill; metal proof pending | `audits/SPIKE-firstboot-nic-sweep-2026-07-22.md` (2026-07-22) |
|
||||||
| <a id="a6"></a>[A6](capability-map-detail/A.md#a6) | Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** | drill VM; no non-operator human yet (R-3) | `audits/DRILL-day0-vm-2026-07-12.md` (2026-07-12) |
|
| <a id="a6"></a>[A6](capability-map-detail/A.md#a6) | Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** | drill VM; no non-operator human yet (R-3) | `audits/DRILL-day0-vm-2026-07-12.md` (2026-07-12) |
|
||||||
| <a id="a7"></a>[A7](capability-map-detail/A.md#a7) | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** | customer-zero, on metal | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
|
| <a id="a7"></a>[A7](capability-map-detail/A.md#a7) | Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** | customer-zero, on metal; **re-check (2026-10-24):** mechanism changed after its evidence — f90e2229 (bind mail W3–W5) | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
|
||||||
| <a id="a8"></a>[A8](capability-map-detail/A.md#a8) | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** | drill VM, endpoint-exact; browser wizard first fired 2026-07-18 | `audits/SPIKE-controller-escrow-2026-07-13.md` (2026-07-13); `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
|
| <a id="a8"></a>[A8](capability-map-detail/A.md#a8) | Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** | drill VM, endpoint-exact; browser wizard first fired 2026-07-18; **re-check (2026-10-24):** mechanism changed after its evidence — agent c11e3e2 + controller ba4a6f0 (typed-back words, R-1151) | `audits/SPIKE-controller-escrow-2026-07-13.md` (2026-07-13); `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18) |
|
||||||
| <a id="a9"></a>[A9](capability-map-detail/A.md#a9) | One name per secret on every surface — „Beállító kód" (the setup code) and „Tulajdonosi jelmondat" (the owner passphrase) — and the re-enrol mail points at a page a rebuilt box shows | controller v0.211.0, hub v0.104.0 + v0.105.0 (R-295, R-323) | **IMPLEMENTED** | written BUILT: source-read; no customer has typed the passphrase | R-295; R-323 |
|
| <a id="a9"></a>[A9](capability-map-detail/A.md#a9) | One name per secret on every surface — „Beállító kód" (the setup code) and „Tulajdonosi jelmondat" (the owner passphrase) — and the re-enrol mail points at a page a rebuilt box shows | controller v0.211.0, hub v0.104.0 + v0.105.0 (R-295, R-323) | **IMPLEMENTED** | written BUILT: source-read; no customer has typed the passphrase | R-295; R-323 |
|
||||||
| <a id="a10"></a>[A10](capability-map-detail/A.md#a10) | The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE** | demo-felhom, real endpoints | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04); `audits/RECON-offsite-dr-chain-2026-08-04.md` (2026-08-04) |
|
| <a id="a10"></a>[A10](capability-map-detail/A.md#a10) | The offsite repository password can be RECOVERED from the sealed escrow with the customer's recovery code | hub v0.94.0, agent v0.125.0, controller v0.195.0 | **PROVEN-LIVE** | demo-felhom, real endpoints; **re-check (2026-10-24):** mechanism changed after its evidence — hub 9688e76e (sealed package back to a rebuilt box, R-1176) | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04); `audits/RECON-offsite-dr-chain-2026-08-04.md` (2026-08-04) |
|
||||||
| <a id="a11"></a>[A11](capability-map-detail/A.md#a11) | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** | demo/drill fleet only | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
|
| <a id="a11"></a>[A11](capability-map-detail/A.md#a11) | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **PROVEN-LIVE** | demo/drill fleet only; **re-check (2026-10-24):** mechanism changed after its evidence — agent ae69ad4 (D30 WireGuard re-key) | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
|
||||||
| <a id="a12"></a>[A12](capability-map-detail/A.md#a12) | A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES — the rebuilt Tester4 box (public ISO) got the household's sealed package from the hub, asked for the recovery code, opened the history; Nextcloud came back with 3 031 of 3 033 manifest files byte-identical (the 2 never reached the old box) and the next off-site runs appended (63 → 64). Earlier the same day it FAILED (R-1176, now closed); the failure paths are not claimed (R-379, R-380) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `audits/DRILL-r201-night-run-2026-08-04.md` (2026-08-04); `audits/DRILL-r356-hot-only-restore-2026-08-22/` (2026-08-22); `audits/DRILL-r356b-driveless-db-restore-2026-08-22/` (2026-08-22) |
|
| <a id="a12"></a>[A12](capability-map-detail/A.md#a12) | A customer's file survives a machine rebuild and comes back — the whole off-site story, end to end | hub v0.94.0, agent v0.125.0, controller v0.197.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES — the rebuilt Tester4 box (public ISO) got the household's sealed package from the hub, asked for the recovery code, opened the history; Nextcloud came back with 3 031 of 3 033 manifest files byte-identical (the 2 never reached the old box) and the next off-site runs appended (63 → 64). Earlier the same day it FAILED (R-1176, now closed); the failure paths are not claimed (R-379, R-380) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `audits/DRILL-r201-night-run-2026-08-04.md` (2026-08-04); `audits/DRILL-r356-hot-only-restore-2026-08-22/` (2026-08-22); `audits/DRILL-r356b-driveless-db-restore-2026-08-22/` (2026-08-22) |
|
||||||
| <a id="a13"></a>[A13](capability-map-detail/A.md#a13) | The customer's own UNAIDED recovery journey, end to end | controller v0.206.0, hub v0.98.0, agent v0.127.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES with ONE operator act by design (D19: delete the dead host) — install, bind, claim, the recovery offer, a wrong code refused, the right code, „Teljes visszaállítás", all through the household's own pages; the new-link page now tells a household whose old box is still linked to write to Felhom (R-1203). Earlier the same day it stopped at the rebuilt dashboard (R-1176) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `tests/walk5-r201-2026-08-07/journal.md` (2026-08-07); `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md` (2026-08-05) |
|
| <a id="a13"></a>[A13](capability-map-detail/A.md#a13) | The customer's own UNAIDED recovery journey, end to end | controller v0.206.0, hub v0.98.0, agent v0.127.0 | **PROVEN-LIVE** | 2026-10-21 (fix brief, hub 0.163.0): PASSES with ONE operator act by design (D19: delete the dead host) — install, bind, claim, the recovery offer, a wrong code refused, the right code, „Teljes visszaállítás", all through the household's own pages; the new-link page now tells a household whose old box is still linked to write to Felhom (R-1203). Earlier the same day it stopped at the rebuilt dashboard (R-1176) | `audits/fixes-2026-10-21/P1-live/` (2026-10-21); `audits/rehearsal-2026-10-20/B7-died/README.md` (2026-10-21); `tests/walk5-r201-2026-08-07/journal.md` (2026-08-07); `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md` (2026-08-05) |
|
||||||
| <a id="a14"></a>[A14](capability-map-detail/A.md#a14) | A random night of household actions while random things go wrong — twelve rounds, unattended | controller v0.245.0, agent v0.131.0, hub v0.116.0, golden 0.245.0, ISO 1.28.0 | **PROVEN-LIVE** | nested box, twelve chaos rounds | `audits/DRILL-chaos-night-2026-09-17.md` (2026-09-17) |
|
| <a id="a14"></a>[A14](capability-map-detail/A.md#a14) | A random night of household actions while random things go wrong — twelve rounds, unattended | controller v0.245.0, agent v0.131.0, hub v0.116.0, golden 0.245.0, ISO 1.28.0 | **PROVEN-LIVE** | nested box, twelve chaos rounds | `audits/DRILL-chaos-night-2026-09-17.md` (2026-09-17) |
|
||||||
| <a id="a15"></a>[A15](capability-map-detail/A.md#a15) | A stranger's FIRST HOUR on a fresh install — download → install → claim → two apps → use → backup → remove → restore, plus a power cut and a mistyped code | controller v0.336.0, agent v0.161.0, hub v0.162.0, golden 0.331.0, installer 1.35.0, ISO 1.29.0 | **PARTIAL** | 2026-10-20: download → install → bind → claim → 8 apps → use → backup → remove → every restore path → power cut → mistyped codes all worked; the 8 GB box FROZE twice in the first afternoon and only a plug pull brought it back (R-1192), and the owner passphrase needed the operator | `audits/rehearsal-2026-10-20/journal.md` (2026-10-20); `audits/DRILL-fresh-install-0242-2026-09-14.md` (2026-09-14); `audits/DRILL-new-household-2026-09-30.md` (2026-09-30) |
|
| <a id="a15"></a>[A15](capability-map-detail/A.md#a15) | A stranger's FIRST HOUR on a fresh install — download → install → claim → two apps → use → backup → remove → restore, plus a power cut and a mistyped code | controller v0.336.0, agent v0.161.0, hub v0.162.0, golden 0.331.0, installer 1.35.0, ISO 1.29.0 | **PARTIAL** | 2026-10-20: download → install → bind → claim → 8 apps → use → backup → remove → every restore path → power cut → mistyped codes all worked; the 8 GB box FROZE twice in the first afternoon and only a plug pull brought it back (R-1192), and the owner passphrase needed the operator | `audits/rehearsal-2026-10-20/journal.md` (2026-10-20); `audits/DRILL-fresh-install-0242-2026-09-14.md` (2026-09-14); `audits/DRILL-new-household-2026-09-30.md` (2026-09-30) |
|
||||||
| <a id="a16"></a>[A16](capability-map-detail/A.md#a16) | Off-site app-data capture covers the paths an app declares MANDATORY — on BOTH drive layouts | controller v0.197.0 | **PROVEN-LIVE** | both drive layouts | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04) |
|
| <a id="a16"></a>[A16](capability-map-detail/A.md#a16) | Off-site app-data capture covers the paths an app declares MANDATORY — on BOTH drive layouts | controller v0.197.0 | **PROVEN-LIVE** | both drive layouts | `audits/DRILL-r201-offsite-recovery-2026-08-04.md` (2026-08-04) |
|
||||||
| <a id="a17"></a>[A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) |
|
| <a id="a17"></a>[A17](capability-map-detail/A.md#a17) | Recurring offsite (PBS) whole-guest backups actually LAND, and RESTORE — local daily + offsite weekly as scheduled work | agent v0.97–0.103, controller v0.174/0.175, hub v0.76.0, host-install 1.20.0 | **PROVEN-LIVE** | demo-hp; **re-check (2026-10-24):** mechanism changed after its evidence — agent 1b6e316 (local keep-daily, R-1149) | `audits/SPIKE-r82-phase0-2026-07-26.md` (2026-07-26) |
|
||||||
| <a id="a18"></a>[A18](capability-map-detail/A.md#a18) | Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard | agent v0.104.0 → v0.121.0, hub v0.77.0 → v0.91.0 | **PROVEN-LIVE** | — | *(evidence not on file — not re-verifiable)*: the row cites per-repo CHANGELOGs and the spec `backlog/SPEC-r85-phase4-5-2026-07-26.md` — a spec, not a run record |
|
| <a id="a18"></a>[A18](capability-map-detail/A.md#a18) | Restore-proof is UNATTENDED — the scheduler covers EVERY tier, follows the BACKUP rather than the clock, and a failure is heard | agent v0.104.0 → v0.121.0, hub v0.77.0 → v0.91.0 | **PROVEN-LIVE** | — | `audits/security-2026-11-02/part4/restore-test-demo-hp.txt`, `restore-test-tester1.txt`, `restore-test-n100.txt` (2026-11-02: the journals of all three boxes, every tier passed unattended in 10 days); earlier: the row cited per-repo CHANGELOGs and a spec |
|
||||||
| <a id="a19"></a>[A19](capability-map-detail/A.md#a19) | A restore-test can never fill the box's disk: it sizes the restore first (uncompressed), keeps off the tested guest's pool when it can, refuses what does not fit, and retries a failed clean-up on a timer | agent v0.133.0 (RELEASED, NOT DELIVERED), hub v0.124.0 | **IMPLEMENTED** | written BUILT: the refusal proven on demo-hp 2026-09-24; the full-test path not live | `audits/r672-2026-09-24/` (2026-09-24); R-672 |
|
| <a id="a19"></a>[A19](capability-map-detail/A.md#a19) | A restore-test can never fill the box's disk: it sizes the restore first (uncompressed), keeps off the tested guest's pool when it can, refuses what does not fit, and retries a failed clean-up on a timer | agent v0.133.0 (RELEASED, NOT DELIVERED), hub v0.124.0 | **IMPLEMENTED** | written BUILT: the refusal proven on demo-hp 2026-09-24; the full-test path not live | `audits/r672-2026-09-24/` (2026-09-24); R-672 |
|
||||||
| <a id="a20"></a>[A20](capability-map-detail/A.md#a20) | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0; purge-first hub v0.142.0 | **PROVEN-LIVE** | two real firings (2026-07-18); third 2026-10-22 on a real customer (Tester-2) with the R-32 purge-first leg — all legs ok, others untouched | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18); `audits/tester2-reset-2026-10-22/` |
|
| <a id="a20"></a>[A20](capability-map-detail/A.md#a20) | Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0; purge-first hub v0.142.0 | **PROVEN-LIVE** | two real firings (2026-07-18); third 2026-10-22 on a real customer (Tester-2) with the R-32 purge-first leg — all legs ok, others untouched | `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (2026-07-18); `audits/tester2-reset-2026-10-22/` |
|
||||||
| <a id="a21"></a>[A21](capability-map-detail/A.md#a21) | Customer DELETE cascade (top lifecycle tier): one guided operator action → `hosts → RESET → residue → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **IMPLEMENTED** | written UNIT-PROVEN: live leg pending | hub unit tests `hub/internal/web/customer_delete_test.go` (outside `documentation/`) |
|
| <a id="a21"></a>[A21](capability-map-detail/A.md#a21) | Customer DELETE cascade (top lifecycle tier): one guided operator action → `hosts → RESET → residue → purge`; host rows deleted (custody DEMOTED), full external teardown (Hetzner repo destroyed, PBS namespace/token revoked, tunnel + zone removed), then the customer record and ALL escrow ciphertext purged | hub v0.69.0 | **IMPLEMENTED** | written UNIT-PROVEN: live leg pending | hub unit tests `hub/internal/web/customer_delete_test.go` (outside `documentation/`) |
|
||||||
@@ -102,17 +102,17 @@
|
|||||||
|
|
||||||
| ID | Capability | Component | Status | Scope | Evidence |
|
| ID | Capability | Component | Status | Scope | Evidence |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
| <a id="b1"></a>[B1](capability-map-detail/B.md#b1) | Deploy an app from the catalog (env config, memory guard, health-aware progress) | controller, catalog (~52 apps, images pinned) | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/RERUN-p1p3-2026-07-06.md` (2026-07-06) |
|
| <a id="b1"></a>[B1](capability-map-detail/B.md#b1) | Deploy an app from the catalog (env config, memory guard, health-aware progress) | controller, catalog (65 apps, images pinned) | **PROVEN-LIVE** | **re-checked 2026-10-24** on controller 0.342.0 (after 6d90374 + 2fe6341): all 65 apps through the deploy request on a fresh box — 62 installed and answered, Wanderer installed but cannot run without DNS (R-739), 2 not offered by lifecycle; the memory guard did not fire (one app at a time) | `audits/catalog-sweep-2026-10-24/README.md` (2026-10-24); `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/RERUN-p1p3-2026-07-06.md` (2026-07-06) |
|
||||||
| <a id="b2"></a>[B2](capability-map-detail/B.md#b2) | App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE** | the actions; remove-with-data re-proven 2026-09-13 | `audits/R442-2026-09-13/` (2026-09-13); `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
|
| <a id="b2"></a>[B2](capability-map-detail/B.md#b2) | App lifecycle: start/stop/restart/update/logs/remove/redeploy | controller | **PROVEN-LIVE** | the actions; remove-with-data re-proven 2026-09-13; **re-checked 2026-10-28 night on 9202** after 8ae9178 (removal deletes labelled volumes, R-1117): install, stop, start, restart, logs, the protected stacks refuse a stop (403), remove with data → its labelled volume and container gone (`audits/night-2026-10-28/part5-claims/lifecycle-walk.*`) | `audits/R442-2026-09-13/` (2026-09-13); `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
|
||||||
| <a id="b3"></a>[B3](capability-map-detail/B.md#b3) | Update is GUARDED: it refuses without a restorable backup, backs up first when the copy is stale, and HOLDS an app that does not come up — on ANY backup tier, and the release itself arrives by the managed floor | controller v0.237.0 + v0.238.0 + v0.238.1 + v0.239.0, hub v0.112.0 | **PROVEN-LIVE** | — | `audits/slice4-2026-09-13/` (2026-09-13); `audits/rulings-r472-r475-2026-09-13/` (2026-09-13) |
|
| <a id="b3"></a>[B3](capability-map-detail/B.md#b3) | Update is GUARDED: it refuses without a restorable backup, backs up first when the copy is stale, and HOLDS an app that does not come up — on ANY backup tier, and the release itself arrives by the managed floor | controller v0.237.0 + v0.238.0 + v0.238.1 + v0.239.0, hub v0.112.0 | **PROVEN-LIVE** | — | `audits/slice4-2026-09-13/` (2026-09-13); `audits/rulings-r472-r475-2026-09-13/` (2026-09-13) |
|
||||||
| <a id="b4"></a>[B4](capability-map-detail/B.md#b4) | A failed update is UNDONE by the box itself — the previous version back with its data from seconds before the update; the app is held only if that undo fails too | controller v0.263.2 (the undo), v0.264.0 + hub v0.120.0 (the mail) | **PROVEN-LIVE** | scratch 9202 | `audits/undo-live-2026-09-23/README.md` (2026-09-23) |
|
| <a id="b4"></a>[B4](capability-map-detail/B.md#b4) | A failed update is UNDONE by the box itself — the previous version back with its data from seconds before the update; the app is held only if that undo fails too | controller v0.263.2 (the undo), v0.264.0 + hub v0.120.0 (the mail) | **PROVEN-LIVE** | scratch 9202 | `audits/undo-live-2026-09-23/README.md` (2026-09-23) |
|
||||||
| <a id="b5"></a>[B5](capability-map-detail/B.md#b5) | The catalog holds only TESTED steps — an image move without a proven test record (bench + box, digests the registry still serves) is refused at push time | catalog `6db08a5` (test-record gate, `ladder.py`) | **PROVEN-LIVE** | written PROVEN: the catalog gate against the live registry | `audits/DRILL-night-2026-09-23.md` (2026-09-23) |
|
| <a id="b5"></a>[B5](capability-map-detail/B.md#b5) | The catalog holds only TESTED steps — an image move without a proven test record (bench + box, digests the registry still serves) is refused at push time | catalog `6db08a5` (test-record gate, `ladder.py`) | **PROVEN-LIVE** | written PROVEN: the catalog gate against the live registry | `audits/DRILL-night-2026-09-23.md` (2026-09-23) |
|
||||||
| <a id="b6"></a>[B6](capability-map-detail/B.md#b6) | A box behind climbs ONE tested step per press, each with its own definition (`09` §3 decision 14) | controller v0.268.0 + catalog `5ed599c` | **PROVEN-LIVE** | — | `audits/ladder-2026-09-24/partD/10-romm-two-steps.json` (2026-09-24) |
|
| <a id="b6"></a>[B6](capability-map-detail/B.md#b6) | A box behind climbs ONE tested step per press, each with its own definition (`09` §3 decision 14) | controller v0.268.0 + catalog `5ed599c` | **PROVEN-LIVE** | — | `audits/ladder-2026-09-24/partD/10-romm-two-steps.json` (2026-09-24) |
|
||||||
| <a id="b7"></a>[B7](capability-map-detail/B.md#b7) | A file app comes back WHOLE from the second drive — its files by four rules (never delete, never overwrite a newer file, bring back what is missing, keep an older replaced file beside it), then its settings and database (`09` §3 decision 26) | controller v0.269.0 (`backup/tier2_whole.go`) | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A1/21-readback.json` (2026-09-24) |
|
| <a id="b7"></a>[B7](capability-map-detail/B.md#b7) | A file app comes back WHOLE from the second drive — its files by four rules (never delete, never overwrite a newer file, bring back what is missing, keep an older replaced file beside it), then its settings and database (`09` §3 decision 26) | controller v0.269.0 (`backup/tier2_whole.go`) | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d582feb + 5929b8e (D22, D25) | `audits/night-2026-09-24/A1/21-readback.json` (2026-09-24) |
|
||||||
| <a id="b8"></a>[B8](capability-map-detail/B.md#b8) | A crash-looping or out-of-memory app is STOPPED by the box, the household and operator are told, and Start gives one more try (decision 28) | controller v0.269.0, hub v0.123.0 | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A3/21-gokapi-page-event.txt` (2026-09-24) |
|
| <a id="b8"></a>[B8](capability-map-detail/B.md#b8) | A crash-looping or out-of-memory app is STOPPED by the box, the household and operator are told, and Start gives one more try (decision 28) | controller v0.269.0, hub v0.123.0 | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A3/21-gokapi-page-event.txt` (2026-09-24) |
|
||||||
| <a id="b9"></a>[B9](capability-map-detail/B.md#b9) | A held app whose page says support is informed can only be removed KEEPING its data (decision 27) | controller v0.269.0 | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A2/10-held-no-copy-keep-data.json` (2026-09-24) |
|
| <a id="b9"></a>[B9](capability-map-detail/B.md#b9) | A held app whose page says support is informed can only be removed KEEPING its data (decision 27) | controller v0.269.0 | **PROVEN-LIVE** | — | `audits/night-2026-09-24/A2/10-held-no-copy-keep-data.json` (2026-09-24) |
|
||||||
| <a id="b10"></a>[B10](capability-map-detail/B.md#b10) | The box runs the exact TESTED image of a floating tag, and an installed app keeps its image until a guarded Update moves it (`09` §6.4 part 6, box half) | controller v0.269.1 (`stacks/digest.go`) | **PROVEN-LIVE** | — | `audits/night-2026-09-24/B/21-floating-tag-0.269.1.json` (2026-09-24) |
|
| <a id="b10"></a>[B10](capability-map-detail/B.md#b10) | The box runs the exact TESTED image of a floating tag, and an installed app keeps its image until a guarded Update moves it (`09` §6.4 part 6, box half) | controller v0.269.1 (`stacks/digest.go`) | **PROVEN-LIVE** | — | `audits/night-2026-09-24/B/21-floating-tag-0.269.1.json` (2026-09-24) |
|
||||||
| <a id="b11"></a>[B11](capability-map-detail/B.md#b11) | Automatic updates at night — the update leg (`09` §6.4 part 7) | controller v0.271.0 | **PROVEN-LIVE** | scratch 9202, six simulated nights; demo boxes in Part D | `audits/DRILL-night-2026-09-25.md` (2026-09-25) |
|
| <a id="b11"></a>[B11](capability-map-detail/B.md#b11) | Automatic updates at night — the update leg (`09` §6.4 part 7) | controller v0.271.0 | **PROVEN-LIVE** | scratch 9202, six simulated nights; demo boxes in Part D; **re-check (2026-10-24):** mechanism changed after its evidence — 63dcccf (security_fix steps, 0.341.0) | `audits/DRILL-night-2026-09-25.md` (2026-09-25) |
|
||||||
| <a id="b12"></a>[B12](capability-map-detail/B.md#b12) | What `restart` and `update` do to a deployed app whose compose file the catalog already moved | controller v0.235.0 | **PROVEN-LIVE** | HISTORY: the behaviour up to v0.234.0; changed 2026-09-06 — current behaviour is B17 | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
|
| <a id="b12"></a>[B12](capability-map-detail/B.md#b12) | What `restart` and `update` do to a deployed app whose compose file the catalog already moved | controller v0.235.0 | **PROVEN-LIVE** | HISTORY: the behaviour up to v0.234.0; changed 2026-09-06 — current behaviour is B17 | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
|
||||||
| <a id="b13"></a>[B13](capability-map-detail/B.md#b13) | Whether the box UPGRADES an app by itself, with nobody pressing anything | controller | **PROVEN-LIVE** | answer: yes, only when an app fails to come back (measured 2026-09-01) | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
|
| <a id="b13"></a>[B13](capability-map-detail/B.md#b13) | Whether the box UPGRADES an app by itself, with nobody pressing anything | controller | **PROVEN-LIVE** | answer: yes, only when an app fails to come back (measured 2026-09-01) | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
|
||||||
| <a id="b14"></a>[B14](capability-map-detail/B.md#b14) | Whether an app UPGRADE can be undone | controller + catalog | **PROVEN-LIVE** | answer: no, once a migration has run | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
|
| <a id="b14"></a>[B14](capability-map-detail/B.md#b14) | Whether an app UPGRADE can be undone | controller + catalog | **PROVEN-LIVE** | answer: no, once a migration has run | `audits/SPIKE-app-update-2026-09-01.md` (2026-09-01) |
|
||||||
@@ -121,9 +121,10 @@
|
|||||||
| <a id="b17"></a>[B17](capability-map-detail/B.md#b17) | An app's VERSION is frozen to what the customer has; only a deliberate Update moves it, while template CORRECTIONS still arrive | controller v0.235.0 | **PROVEN-LIVE** | two real catalog pushes, real 15-minute cycle | `tests/VALIDATION-update-slice3-2026-09-06.md` (2026-09-06) |
|
| <a id="b17"></a>[B17](capability-map-detail/B.md#b17) | An app's VERSION is frozen to what the customer has; only a deliberate Update moves it, while template CORRECTIONS still arrive | controller v0.235.0 | **PROVEN-LIVE** | two real catalog pushes, real 15-minute cycle | `tests/VALIDATION-update-slice3-2026-09-06.md` (2026-09-06) |
|
||||||
| <a id="b18"></a>[B18](capability-map-detail/B.md#b18) | Catalog sync (git, 15 min) + orphan lifecycle + validation choke point (bad `backup:` block degrades to legacy, loudly) | controller v0.132, catalog | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
|
| <a id="b18"></a>[B18](capability-map-detail/B.md#b18) | Catalog sync (git, 15 min) + orphan lifecycle + validation choke point (bad `backup:` block degrades to legacy, loudly) | controller v0.132, catalog | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
|
||||||
| <a id="b19"></a>[B19](capability-map-detail/B.md#b19) | Lemez-egészség felügyelet: per-disk SMART kártya („Lemezek állapota") + degradáció-riasztás (Rendben/Figyelmeztetés/Hiba/Nincs adat) | agent v0.94.0→v0.95.0, controller v0.169.0→v0.171.0→v0.215.0, hub v0.73.1 | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the Hiba-from-counters path is unit-only (R-332) | `audits/SPIKE-smart-coverage-2026-07-25.md` (2026-07-25); `audits/DIAG-smart-passed-trap-2026-08-14.md` (2026-08-14) |
|
| <a id="b19"></a>[B19](capability-map-detail/B.md#b19) | Lemez-egészség felügyelet: per-disk SMART kártya („Lemezek állapota") + degradáció-riasztás (Rendben/Figyelmeztetés/Hiba/Nincs adat) | agent v0.94.0→v0.95.0, controller v0.169.0→v0.171.0→v0.215.0, hub v0.73.1 | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the Hiba-from-counters path is unit-only (R-332) | `audits/SPIKE-smart-coverage-2026-07-25.md` (2026-07-25); `audits/DIAG-smart-passed-trap-2026-08-14.md` (2026-08-14) |
|
||||||
| <a id="b20"></a>[B20](capability-map-detail/B.md#b20) | App crashes → customer notified (one event per transition, no flapping spam) | controller v0.120, hub v0.48 | **IMPLEMENTED** | crash→customer mail never live-confirmed | `audits/CAMPAIGN-3-2026-07-11.md` (2026-07-11) |
|
| <a id="b20"></a>[B20](capability-map-detail/B.md#b20) | App crashes → customer notified (one event per transition, no flapping spam) | controller v0.120, hub v0.48 | **PROVEN-LIVE** | — (the household's leg needs an address on its page; Tester 1 had none) | `audits/security-2026-11-02/part5/crash-walk.log`, `crash-walk-customer.log`, `mail-crash-operator.txt`, `mail-crash-household.txt` (2026-11-02, Tester 1: a crash loop stopped the app, the operator's mail in 1 s, the household's Hungarian mail in 1 s, read in the mailbox); `audits/CAMPAIGN-3-2026-07-11.md` |
|
||||||
| <a id="b21"></a>[B21](capability-map-detail/B.md#b21) | Post-deploy optional config (API keys etc.) with restart | controller, catalog `.felhom.yml` | **IMPLEMENTED** | config-save + restart in no campaign | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
|
| <a id="b21"></a>[B21](capability-map-detail/B.md#b21) | Post-deploy optional config (API keys etc.) with restart | controller, catalog `.felhom.yml` | **IMPLEMENTED** | config-save + restart in no campaign | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
|
||||||
| <a id="b22"></a>[B22](capability-map-detail/B.md#b22) | Backup classification: 13 bind-bearing apps carry mandatory/optional/excluded classes | catalog, controller v0.132–133 | **PROVEN-LIVE** | — | `audits/SPIKE-backup-classification-2026-07-14.md` (2026-07-14); `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15) |
|
| <a id="b22"></a>[B22](capability-map-detail/B.md#b22) | Backup classification: 13 bind-bearing apps carry mandatory/optional/excluded classes | catalog, controller v0.132–133 | **PROVEN-LIVE** | — | `audits/SPIKE-backup-classification-2026-07-14.md` (2026-07-14); `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15) |
|
||||||
|
| <a id="b23"></a>[B23](capability-map-detail/B.md#b23) | An urgent security fix reaches the boxes at night — a `security_fix` ladder step (box walk required, bench optional, a CVE/GHSA notice), taken by the night update (`09` §3 decision 255, D39 = B) | catalog `3898593`, controller v0.341.0 | **PROVEN-LIVE** | scratch 9202 (drill catalog), Plex 1.43.4 → 1.43.5; an app with no box fixture is checked by health, version and a soak only (`seed: none`) | `audits/security-lane-2026-10-23/` (2026-10-23); batch use: `audits/security-app-updates-2026-10-27/` (2026-10-27, 14 steps live) |
|
||||||
|
|
||||||
## C. Protection & recovery (the product promise)
|
## C. Protection & recovery (the product promise)
|
||||||
|
|
||||||
@@ -146,18 +147,18 @@
|
|||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
| <a id="c1"></a>[C1](capability-map-detail/C.md#c1) | Whole-guest backup lands OFF the guest's own physical device; a single-drive box is recorded DEGRADED rather than silently normal (installer Case A/B) | agent v0.113, host-install v1.22.0 | **PROVEN-LIVE** | — | `audits/E2D-fresh-vm-2026-07-29.md` (2026-07-29) |
|
| <a id="c1"></a>[C1](capability-map-detail/C.md#c1) | Whole-guest backup lands OFF the guest's own physical device; a single-drive box is recorded DEGRADED rather than silently normal (installer Case A/B) | agent v0.113, host-install v1.22.0 | **PROVEN-LIVE** | — | `audits/E2D-fresh-vm-2026-07-29.md` (2026-07-29) |
|
||||||
| <a id="c2"></a>[C2](capability-map-detail/C.md#c2) | Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15) |
|
| <a id="c2"></a>[C2](capability-map-detail/C.md#c2) | Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15) |
|
||||||
| <a id="c3"></a>[C3](capability-map-detail/C.md#c3) | Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14) |
|
| <a id="c3"></a>[C3](capability-map-detail/C.md#c3) | Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 03c0a69 + 66a69d4 + 5929b8e (capped container R-1214, kept copies D25) | `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14) |
|
||||||
| <a id="c4"></a>[C4](capability-map-detail/C.md#c4) | Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/VALIDATION-offbox-storagebox-2026-07-09.md` (2026-07-09) |
|
| <a id="c4"></a>[C4](capability-map-detail/C.md#c4) | Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 8f5bea0 + d5f88a2 (abandon R-1182, off-site proof R-1189) | `audits/security-2026-11-02/part5/offsite-demo-hp.txt`, `offsite-tester1.txt` (2026-11-02: snapshots per day read from the repositories, unbroken 10-21 → 11-02 on demo-hp, 10-27 → 11-02 on Tester 1); `audits/CAMPAIGN-6D-2026-07-15.md`; `audits/VALIDATION-offbox-storagebox-2026-07-09.md` |
|
||||||
| <a id="c5"></a>[C5](capability-map-detail/C.md#c5) | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/DIAG-immich-restore-round2-2026-07-19.md` (2026-07-19) |
|
| <a id="c5"></a>[C5](capability-map-detail/C.md#c5) | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — 93fb251 + d582feb (D20, D22) **2026-10-25:** changed again by controller 0.343.0 (D48 pull-first, R-1270 fix A: the verdict needs a stable start, the undo copy goes after 3 more minutes); on 0.343.0 Tester 1's local unit restore and set-aside restore passed and released the copy (`audits/release-2026-10-26/t1-restores-0.343.0/`); this path itself not re-walked. | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/DIAG-immich-restore-round2-2026-07-19.md` (2026-07-19) |
|
||||||
| <a id="c6"></a>[C6](capability-map-detail/C.md#c6) | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6B-2026-07-14.md` (2026-07-14) |
|
| <a id="c6"></a>[C6](capability-map-detail/C.md#c6) | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d5f88a2 + 5929b8e (R-1187 c, R-1191) | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6B-2026-07-14.md` (2026-07-14) |
|
||||||
| <a id="c7"></a>[C7](capability-map-detail/C.md#c7) | Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `pilot/DRILL-GL6-2026-07-08.md` (2026-07-08) |
|
| <a id="c7"></a>[C7](capability-map-detail/C.md#c7) | Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07); `pilot/DRILL-GL6-2026-07-08.md` (2026-07-08) |
|
||||||
| <a id="c8"></a>[C8](capability-map-detail/C.md#c8) | PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | scoped to one host | `audits/SPIKE-pbsdr-selfheal-2026-07-15.md` (2026-07-15) |
|
| <a id="c8"></a>[C8](capability-map-detail/C.md#c8) | PBS-DR secret self-heal on reused-peer re-provision | hub v0.56 | **IMPLEMENTED** | scoped to one host | `audits/SPIKE-pbsdr-selfheal-2026-07-15.md` (2026-07-15) |
|
||||||
| <a id="c9"></a>[C9](capability-map-detail/C.md#c9) | Box survives an unattended app or guest-network failure (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0→v0.190.0, agent v0.92.1 | **PROVEN-LIVE** | demo box; boot-orphan leg 6 of 6 hard resets | `audits/DIAG-bootrecon-drive-absent-2026-08-02.md` (2026-08-02) |
|
| <a id="c9"></a>[C9](capability-map-detail/C.md#c9) | Box survives an unattended app or guest-network failure (a dead app member, a boot-orphaned app, a dead DHCP client) — it is noticed, and where safe it is repaired | controller v0.156.0→v0.190.0, agent v0.92.1 | **PROVEN-LIVE** | demo box; boot-orphan leg 6 of 6 hard resets | `audits/DIAG-bootrecon-drive-absent-2026-08-02.md` (2026-08-02) |
|
||||||
| <a id="c10"></a>[C10](capability-map-detail/C.md#c10) | Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15) |
|
| <a id="c10"></a>[C10](capability-map-detail/C.md#c10) | Crash/power-loss mid-backup/mid-migration → self-heal on next run | controller, agent | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15) |
|
||||||
| <a id="c11"></a>[C11](capability-map-detail/C.md#c11) | An app can be withdrawn from the catalog without orphaning the customers running it (available / hidden / abandoned) | controller v0.158.1, catalog metadata | **PROVEN-LIVE** | guest 9201, real endpoints | *(evidence not on file — not re-verifiable)*: the row cites TASK-F Part 1 and `felhom-controller/REPORT.md` (2026-07-21), both outside `documentation/` |
|
| <a id="c11"></a>[C11](capability-map-detail/C.md#c11) | An app can be withdrawn from the catalog without orphaning the customers running it (available / hidden / abandoned) | controller v0.158.1, catalog metadata | **PROVEN-LIVE** | guest 9201, real endpoints | *(evidence not on file — not re-verifiable)*: the row cites TASK-F Part 1 and `felhom-controller/REPORT.md` (2026-07-21), both outside `documentation/` |
|
||||||
| <a id="c12"></a>[C12](capability-map-detail/C.md#c12) | Box survives a site/network change (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE** | — | `audits/SPIKE-island-bridge-2026-07-25.md` (2026-07-25) |
|
| <a id="c12"></a>[C12](capability-map-detail/C.md#c12) | Box survives a site/network change (relocation, different subnet, DHCP re-lease) with the control plane intact | agent v0.96.0 (island NIC), host-install v1.19.0, controller (unchanged), bootstrap | **PROVEN-LIVE** | — | `audits/SPIKE-island-bridge-2026-07-25.md` (2026-07-25) |
|
||||||
| <a id="c13"></a>[C13](capability-map-detail/C.md#c13) | The customer is warned BEFORE a filesystem fills — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered | controller v0.191.0/.1/.2, hub v0.89.0 (R-167, decision D-c) | **PROVEN-LIVE** | guest 9201, a real filesystem | *(evidence not on file — not re-verifiable)*: the row cites `felhom-controller/REPORT.md` (2026-08-02); `audits/SPIKE-r165-mp1-merge-2026-08-02.md` is cited as context only |
|
| <a id="c13"></a>[C13](capability-map-detail/C.md#c13) | The customer is warned BEFORE a filesystem fills — per filesystem, in Hungarian, naming the drive and the free space, edge-triggered | controller v0.191.0/.1/.2, hub v0.89.0 (R-167, decision D-c) | **PROVEN-LIVE** | guest 9201, a real filesystem | `audits/security-2026-11-02/part5/fill-walk.log`, `mail-fill-household.txt` (2026-11-03, Tester 1: a drive taken to 89 %, the watcher's next 10-minute tick, the household's Hungarian mail naming the drive and 5,2 GB free, the operator's copy, once) |
|
||||||
| <a id="c14"></a>[C14](capability-map-detail/C.md#c14) | A failed per-app Tier-1 backup reaches the OPERATOR — EVERY failing app, in ONE mail per run, and every failure recorded whether or not it is mailed | controller v0.194.0, hub v0.90.1 (R-158 → R-167 → R-182) | **PROVEN-LIVE** | guest 9201 | *(evidence not on file — not re-verifiable)*: the row cites `felhom-controller/REPORT.md` (2026-08-03), outside `documentation/` |
|
| <a id="c14"></a>[C14](capability-map-detail/C.md#c14) | A failed per-app Tier-1 backup reaches the OPERATOR — EVERY failing app, in ONE mail per run, and every failure recorded whether or not it is mailed | controller v0.194.0, hub v0.90.1 (R-158 → R-167 → R-182) | **PROVEN-LIVE** | guest 9201 | `audits/security-2026-11-02/part5/mail-operator-digest.txt` (read in the mailbox 2026-11-03: „Tester4: 3 of 8 apps failed to back up (manual run)", each app named, one mail) |
|
||||||
| <a id="c15"></a>[C15](capability-map-detail/C.md#c15) | A local backup is bounded by the box's FREE SPACE, not by a partition set at build time — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime | golden `build-golden.sh` v3.0.0, agent v0.120.0, controller v0.193.1 | **PROVEN-LIVE** | both halves | `audits/SPIKE-r165-phase0-2026-08-03.md` (2026-08-03) |
|
| <a id="c15"></a>[C15](capability-map-detail/C.md#c15) | A local backup is bounded by the box's FREE SPACE, not by a partition set at build time — the appliance ships ONE data volume, and a capture that would exhaust it is refused per app rather than allowed to stop the container runtime | golden `build-golden.sh` v3.0.0, agent v0.120.0, controller v0.193.1 | **PROVEN-LIVE** | both halves | `audits/SPIKE-r165-phase0-2026-08-03.md` (2026-08-03) |
|
||||||
| <a id="c16"></a>[C16](capability-map-detail/C.md#c16) | Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15) |
|
| <a id="c16"></a>[C16](capability-map-detail/C.md#c16) | Soft-quota: usage bar, pre-push enlargement block, customer notification | controller v0.109/134, hub v0.41/55 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15); `audits/CAMPAIGN-6E-2026-07-15.md` (2026-07-15) |
|
||||||
| <a id="c17"></a>[C17](capability-map-detail/C.md#c17) | A customer (not the operator) performs a restore via UI alone | all | **MISSING** | as evidence | `audits/DIAG-immich-restore-2026-07-19.md` (2026-07-19); R-3 |
|
| <a id="c17"></a>[C17](capability-map-detail/C.md#c17) | A customer (not the operator) performs a restore via UI alone | all | **MISSING** | as evidence | `audits/DIAG-immich-restore-2026-07-19.md` (2026-07-19); R-3 |
|
||||||
@@ -171,31 +172,31 @@
|
|||||||
| <a id="d3"></a>[D3](capability-map-detail/D.md#d3) | Data migration between drives (all / per-app), crash-safe | controller | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14); `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
|
| <a id="d3"></a>[D3](capability-map-detail/D.md#d3) | Data migration between drives (all / per-app), crash-safe | controller | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14); `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
|
||||||
| <a id="d4"></a>[D4](capability-map-detail/D.md#d4) | NAS (NFS/SMB-client) verify-before-commit, uid-1000 probe, categorized Hungarian errors, DSM-validated | controller v0.113–117, agent v0.81/84/85 | **PROVEN-LIVE** | — | `audits/SPIKE-nas-verify-2026-07-11.md` (2026-07-11); `audits/SPIKE-nas-dsm-2026-07-11.md` (2026-07-11) |
|
| <a id="d4"></a>[D4](capability-map-detail/D.md#d4) | NAS (NFS/SMB-client) verify-before-commit, uid-1000 probe, categorized Hungarian errors, DSM-validated | controller v0.113–117, agent v0.81/84/85 | **PROVEN-LIVE** | — | `audits/SPIKE-nas-verify-2026-07-11.md` (2026-07-11); `audits/SPIKE-nas-dsm-2026-07-11.md` (2026-07-11) |
|
||||||
| <a id="d5"></a>[D5](capability-map-detail/D.md#d5) | Network storage (NAS) is browse + bulk-media only — it may NOT host an app's data namespace | controller v0.187.0 | **PROVEN-LIVE** | — | `audits/R108-network-app-namespace-2026-07-30.md` (2026-07-30) |
|
| <a id="d5"></a>[D5](capability-map-detail/D.md#d5) | Network storage (NAS) is browse + bulk-media only — it may NOT host an app's data namespace | controller v0.187.0 | **PROVEN-LIVE** | — | `audits/R108-network-app-namespace-2026-07-30.md` (2026-07-30) |
|
||||||
| <a id="d6"></a>[D6](capability-map-detail/D.md#d6) | A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5) | controller v0.188.0 | **PROVEN-LIVE** | — | `audits/D5-drive-alone-restore-2026-07-30.md` (2026-07-30) |
|
| <a id="d6"></a>[D6](capability-map-detail/D.md#d6) | A Tier-1/Tier-2 app restore works from the DRIVE ALONE — the recovery unit carries the app's secrets (D5) | controller v0.188.0 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — d582feb (D22, R-1178) **2026-10-25:** changed again by controller 0.343.0 (D48 pull-first, R-1270 fix A: the verdict needs a stable start, the undo copy goes after 3 more minutes); on 0.343.0 Tester 1's local unit restore and set-aside restore passed and released the copy (`audits/release-2026-10-26/t1-restores-0.343.0/`); this path itself not re-walked. | `audits/D5-drive-alone-restore-2026-07-30.md` (2026-07-30) |
|
||||||
| <a id="d7"></a>[D7](capability-map-detail/D.md#d7) | A restore SAYS what it returned, and refuses what it cannot do — the four restore-surface truth defects from the 2026-08-21 drill | controller v0.226.0 (R-353, R-357, R-358, R-360, R-396) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: three of four live; R-357 implemented only | `audits/evidence-r353-r360-live-2026-08-30/live-validation.txt` (2026-08-30) |
|
| <a id="d7"></a>[D7](capability-map-detail/D.md#d7) | A restore SAYS what it returned, and refuses what it cannot do — the four restore-surface truth defects from the 2026-08-21 drill | controller v0.226.0 (R-353, R-357, R-358, R-360, R-396) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: three of four live; R-357 implemented only **2026-10-25:** controller 0.343.0 adds two truths here — a restore whose app does not start says so and keeps its copy (R-1270, D49 = A; live on 9202 rc.6), and a `.fab` import that cannot download its version refuses before it touches the app (R-1279, live). | `audits/evidence-r353-r360-live-2026-08-30/live-validation.txt` (2026-08-30) |
|
||||||
| <a id="d8"></a>[D8](capability-map-detail/D.md#d8) | The box PROVES its own off-site copy still HOLDS something — a backup that is intact and EMPTY is caught without a person | controller v0.231.0 (R-87) + hub v0.110.0 | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the scheduled firing is implemented only | `tests/r87-offsite-proof-2026-08-31/` (2026-08-31) |
|
| <a id="d8"></a>[D8](capability-map-detail/D.md#d8) | The box PROVES its own off-site copy still HOLDS something — a backup that is intact and EMPTY is caught without a person | controller v0.231.0 (R-87) + hub v0.110.0 | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the scheduled firing is implemented only | `tests/r87-offsite-proof-2026-08-31/` (2026-08-31) |
|
||||||
| <a id="d9"></a>[D9](capability-map-detail/D.md#d9) | The off-site store is VERIFIED on a cadence — something checks that the customer's backups are still readable | controller v0.228.0 (R-359, R-397, R-399) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the scheduled firing is implemented only | `tests/r359-integrity-2026-08-30/` (2026-08-30) |
|
| <a id="d9"></a>[D9](capability-map-detail/D.md#d9) | The off-site store is VERIFIED on a cadence — something checks that the customer's backups are still readable | controller v0.228.0 (R-359, R-397, R-399) | **PARTIAL** | LOWERED from PROVEN-LIVE by its own words: the scheduled firing is implemented only | `tests/r359-integrity-2026-08-30/` (2026-08-30) |
|
||||||
| <a id="d10"></a>[D10](capability-map-detail/D.md#d10) | USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | — | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15); `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13) |
|
| <a id="d10"></a>[D10](capability-map-detail/D.md#d10) | USB drive enrollment + unplug detection + recommission | controller, agent | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 (R-1124) | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15); `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13) |
|
||||||
| <a id="d11"></a>[D11](capability-map-detail/D.md#d11) | Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | — | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
|
| <a id="d11"></a>[D11](capability-map-detail/D.md#d11) | Decommission (migrate-first and anyway-paths), eject | agent, controller | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 + 5c3b218, controller 6c6bd87 (R-1124, R-1144) | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15) |
|
||||||
| <a id="d12"></a>[D12](capability-map-detail/D.md#d12) | Boot ordering: automount + networking survive reboot; appliance self-heal watchdog | agent v0.85 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13); `audits/CAMPAIGN-6A-2026-07-14.md` (2026-07-14) |
|
| <a id="d12"></a>[D12](capability-map-detail/D.md#d12) | Boot ordering: automount + networking survive reboot; appliance self-heal watchdog | agent v0.85 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-4-2026-07-13.md` (2026-07-13); `audits/CAMPAIGN-6A-2026-07-14.md` (2026-07-14) |
|
||||||
| <a id="d13"></a>[D13](capability-map-detail/D.md#d13) | Media apps see EVERY drive; a drive added, pulled, back or safely removed is handled per app (notice, real restart, wiring, torrents paused/resumed) | controller v0.313.0, catalog `0188430`, hub v0.151.0 | **PROVEN-LIVE** | Tier-0 boxes (Tester 1, demo-hp 9201); the fleet with golden 0.313.0 + the floor | `audits/media-build-2026-10-15/G/` (2026-10-15); `audits/media-drive-events-2026-10-14/` (2026-10-14) |
|
| <a id="d13"></a>[D13](capability-map-detail/D.md#d13) | Media apps see EVERY drive; a drive added, pulled, back or safely removed is handled per app (notice, real restart, wiring, torrents paused/resumed) | controller v0.313.0, catalog `0188430`, hub v0.151.0 | **PROVEN-LIVE** | Tier-0 boxes (Tester 1, demo-hp 9201); the fleet with golden 0.313.0 + the floor | `audits/media-build-2026-10-15/G/` (2026-10-15); `audits/media-drive-events-2026-10-14/` (2026-10-14) |
|
||||||
| <a id="d14"></a>[D14](capability-map-detail/D.md#d14) | A second local copy COUNTS only on a proven different physical disk: the agent knows each place's disks; a VM box makes the copy but says it cannot check; a same-disk copy is not made; a NAS never hides the one-copy notice; a box with no drive refuses a data-folder app in words | agent v0.157.0, controller v0.317.0, hub v0.154.0 | **PARTIAL** | live: VM `unknown` (Tester 1) and two real-disk boxes `proven` (demo-hp, N100); unit only: the same-disk skip, the NAS notice, the no-drive sentence; `usb_bridge` never seen on a device | `audits/storage-identity-buildA-2026-10-15/README.md` (2026-10-15); `architecture/16-storage-physical-identity.md` §13 |
|
| <a id="d14"></a>[D14](capability-map-detail/D.md#d14) | A second local copy COUNTS only on a proven different physical disk: the agent knows each place's disks; a VM box makes the copy but says it cannot check; a same-disk copy is not made; a NAS never hides the one-copy notice; a box with no drive refuses a data-folder app in words | agent v0.157.0, controller v0.317.0, hub v0.154.0 | **PARTIAL** | live: VM `unknown` (Tester 1) and two real-disk boxes `proven` (demo-hp, N100); unit only: the same-disk skip, the NAS notice, the no-drive sentence; `usb_bridge` never seen on a device | `audits/storage-identity-buildA-2026-10-15/README.md` (2026-10-15); `architecture/16-storage-physical-identity.md` §13 |
|
||||||
| <a id="d15"></a>[D15](capability-map-detail/D.md#d15) | A one-disk box gets its own data area („Belső tárhely") outside every whole-box copy, registered as the default drive with no press, so media and data-folder apps install with no extra step; a free disk's partitions can be formatted or used one by one | installer 1.33.0, agent v0.158.1, controller v0.318.0, golden 0.318.0 | **PROVEN-LIVE** | fresh 512 GB one-disk VM on demo-hp; partitions on a scratch disk; the old-box signed job and the D4 sentence unit-proven only; an empty part asks for confirmation (R-1093) | `audits/storage-identity-buildB-2026-10-15/README.md` (2026-10-16) |
|
| <a id="d15"></a>[D15](capability-map-detail/D.md#d15) | A one-disk box gets its own data area („Belső tárhely") outside every whole-box copy, registered as the default drive with no press, so media and data-folder apps install with no extra step; a free disk's partitions can be formatted or used one by one | installer 1.33.0, agent v0.158.1, controller v0.318.0, golden 0.318.0 | **PROVEN-LIVE** | fresh 512 GB one-disk VM on demo-hp; partitions on a scratch disk; the old-box signed job and the D4 sentence unit-proven only; an empty part asks for confirmation (R-1093) | `audits/storage-identity-buildB-2026-10-15/README.md` (2026-10-16) |
|
||||||
| <a id="d16"></a>[D16](capability-map-detail/D.md#d16) | A failed or cut restore puts the household's data back EXACTLY as it was and starts the app — local unit, second drive, kept load and off-site, and after a power cut or a killed controller (D22) | controller `main` (0.338.0-rc.1), unreleased | **IMPLEMENTED** | proven live on 9202 (rc.1): a damaged unit and a controller killed between the volume step and the replay; the second-drive pre-check (R-1178), the mirror skip (R-1180) and the off-site power cut by test only; a database outside the replaced volumes is not covered | `audits/night-2026-10-21/P3-restore/02-run.log` (2026-10-21) |
|
| <a id="d16"></a>[D16](capability-map-detail/D.md#d16) | A failed or cut restore puts the household's data back EXACTLY as it was and starts the app — local unit, second drive, kept load and off-site, and after a power cut or a killed controller (D22) | controller 0.338.0 (D22) → **0.343.0** (released 2026-10-25: R-1251 a copy on an unbound drive waits; fix A) | **IMPLEMENTED** | proven live on 9202 (rc.1): a damaged unit and a controller killed between the volume step and the replay; the second-drive pre-check (R-1178), the mirror skip (R-1180) and the off-site power cut by test only; a database outside the replaced volumes is not covered **2026-10-25 (re-check):** released in 0.343.0; the release run proved the success path's copy release on Tester 1, not a failure put-back on the release build — the status stays. | `audits/night-2026-10-21/P3-restore/02-run.log` (2026-10-21) |
|
||||||
|
|
||||||
## E. Access, networking & household use
|
## E. Access, networking & household use
|
||||||
|
|
||||||
| ID | Capability | Component | Status | Scope | Evidence |
|
| ID | Capability | Component | Status | Scope | Evidence |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
| <a id="e1"></a>[E1](capability-map-detail/E.md#e1) | Remote access via Cloudflare Tunnel + Traefik (per-app subdomains) | cloudflared, traefik | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
|
| <a id="e1"></a>[E1](capability-map-detail/E.md#e1) | Remote access via Cloudflare Tunnel + Traefik (per-app subdomains) | cloudflared, traefik | **PROVEN-LIVE** | — | `tests/CAMPAIGN-2-2026-07-07.md` (2026-07-07) |
|
||||||
| <a id="e2"></a>[E2](capability-map-detail/E.md#e2) | LAN access when internet is down (lan_resolver) | agent | **IMPLEMENTED** | never drilled as a customer experience | R-19 |
|
| <a id="e2"></a>[E2](capability-map-detail/E.md#e2) | LAN access when internet is down (lan_resolver) | agent | **PROVEN-LIVE** | the NAME works only if the home network's DNS points at the box, and on a host whose own FQDN is the dashboard's name it answers the host (R-1364) | `audits/security-2026-11-02/part5/outage-wan.log`, `lan-resolver.txt` (2026-11-03, Tester 1, internet and hub dropped 13 min: the dashboard answered at the box's address, 10 apps ran, the public name 530, back in 60 s after) |
|
||||||
| <a id="e3"></a>[E3](capability-map-detail/E.md#e3) | Phone photo backup | immich (classified) | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15) |
|
| <a id="e3"></a>[E3](capability-map-detail/E.md#e3) | Phone photo backup | immich (classified) | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6D-2026-07-15.md` (2026-07-15) |
|
||||||
| <a id="e4"></a>[E4](capability-map-detail/E.md#e4) | Documents/OCR | paperless-ngx (classified) | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14) |
|
| <a id="e4"></a>[E4](capability-map-detail/E.md#e4) | Documents/OCR | paperless-ngx (classified) | **PROVEN-LIVE** | — | `audits/CAMPAIGN-6C-2026-07-14.md` (2026-07-14) |
|
||||||
| <a id="e5"></a>[E5](capability-map-detail/E.md#e5) | Files from Windows Explorer / Mac Finder (SMB server) | controller v0.145.0 + `felhom-samba:1.0.0` | **PROVEN-LIVE** | — | *(evidence not on file — not re-verifiable)*: the row cites `felhom-controller` REPORT.md (v0.144.0, outside `documentation/`), the design `controller/sharing.md` and the transport verdict `audits/SPIKE-lan-discovery-2026-07-18.md` — none is a run of the shipped server |
|
| <a id="e5"></a>[E5](capability-map-detail/E.md#e5) | Files from Windows Explorer / Mac Finder (SMB server) | controller v0.145.0 + `felhom-samba:1.0.0` | **PROVEN-LIVE** | — | *(evidence not on file — not re-verifiable)*: the row cites `felhom-controller` REPORT.md (v0.144.0, outside `documentation/`), the design `controller/sharing.md` and the transport verdict `audits/SPIKE-lan-discovery-2026-07-18.md` — none is a run of the shipped server |
|
||||||
| <a id="e6"></a>[E6](capability-map-detail/E.md#e6) | Media to TV via DLNA | — | **MISSING** | — | R-6; R-8 |
|
| <a id="e6"></a>[E6](capability-map-detail/E.md#e6) | Media to TV via DLNA | — | **MISSING** | — | R-6; R-8 |
|
||||||
| <a id="e7"></a>[E7](capability-map-detail/E.md#e7) | File access via browser | FileBrowser (infra app, auto-mount sync) | **IMPLEMENTED** | browse/download through FileBrowser in no doc | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15); R-75 |
|
| <a id="e7"></a>[E7](capability-map-detail/E.md#e7) | File access via browser | FileBrowser (infra app, auto-mount sync) | **IMPLEMENTED** | browse/download through FileBrowser in no doc | `audits/storage-lifecycle-acceptance-2026-06-15.md` (2026-06-15); R-75 |
|
||||||
| <a id="e8"></a>[E8](capability-map-detail/E.md#e8) | Indítópult (app launcher) — one-tap grid of the household's openable apps | controller v0.163.0 | **IMPLEMENTED** | live click-through pending | R-72 |
|
| <a id="e8"></a>[E8](capability-map-detail/E.md#e8) | Indítópult (app launcher) — one-tap grid of the household's openable apps | controller v0.163.0 | **PROVEN-LIVE** (2026-10-28 night, 9202, endpoint-level — the household's page lists a tile per openable app, each linking to its public address) | `audits/night-2026-10-28/part5-claims/launcher-walk.*` | R-72 |
|
||||||
| <a id="e9"></a>[E9](capability-map-detail/E.md#e9) | Indítópult megosztás (vendég link) — capability URL `/s/<token>` serves a standalone read-only guest launcher (no account, no admin session); optional per-share password; QR | controller v0.165.0 | **IMPLEMENTED** | — | R-15 |
|
| <a id="e9"></a>[E9](capability-map-detail/E.md#e9) | Indítópult megosztás (vendég link) — capability URL `/s/<token>` serves a standalone read-only guest launcher (no account, no admin session); optional per-share password; QR | controller v0.165.0 | **PROVEN-LIVE** (2026-10-28 night, 9202, endpoint-level: switched on, opened with NO session — the apps, no admin link; a wrong token 404; rotate → the old link 404, the new 200; switched off → 404; the box left as found) | `audits/night-2026-10-28/part5-claims/launcher-walk.*` | R-15 |
|
||||||
| <a id="e10"></a>[E10](capability-map-detail/E.md#e10) | Forgot dashboard password → instant reset code | controller v0.123, hub | **PROVEN-LIVE** | — | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
|
| <a id="e10"></a>[E10](capability-map-detail/E.md#e10) | Forgot dashboard password → instant reset code | controller v0.123, hub | **PROVEN-LIVE** | — | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
|
||||||
| <a id="e11"></a>[E11](capability-map-detail/E.md#e11) | The dashboard in English — the household picks its language; Hungarian unchanged | controller v0.247.0 (spike) → v0.250.0 (slice 1, R-556 CLOSED) | **PARTIAL** | every dashboard template live; other surfaces open | `audits/i18n-slice1-2026-09-17/C/live/` (2026-09-17); `audits/i18n-2026-09-17/live/` (2026-09-17) |
|
| <a id="e11"></a>[E11](capability-map-detail/E.md#e11) | The dashboard in English — the household picks its language; Hungarian unchanged | controller v0.247.0 (spike) → v0.250.0 (slice 1, R-556 CLOSED) | **PARTIAL** | every dashboard template live; other surfaces open | `audits/i18n-slice1-2026-09-17/C/live/` (2026-09-17); `audits/i18n-2026-09-17/live/` (2026-09-17) |
|
||||||
| <a id="e12"></a>[E12](capability-map-detail/E.md#e12) | The box tells visitors apart — a stranger's wrong passwords lock only the stranger (dashboard, setup gate, apps that read the visitor from the right) | controller v0.286.1, catalog router resets | **PARTIAL** | — | `audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt` (2026-10-01); R-779 |
|
| <a id="e12"></a>[E12](capability-map-detail/E.md#e12) | The box tells visitors apart — a stranger's wrong passwords lock only the stranger (dashboard, setup gate, apps that read the visitor from the right) | controller v0.286.1, catalog router resets | **PARTIAL** | — | `audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt` (2026-10-01); R-779 |
|
||||||
@@ -203,7 +204,7 @@
|
|||||||
| <a id="e14"></a>[E14](capability-map-detail/E.md#e14) | Multiple household users / per-person accounts | — | **MISSING** | — | R-15 |
|
| <a id="e14"></a>[E14](capability-map-detail/E.md#e14) | Multiple household users / per-person accounts | — | **MISSING** | — | R-15 |
|
||||||
| <a id="e15"></a>[E15](capability-map-detail/E.md#e15) | A second login step for the dashboard (a TOTP code or a passkey) | — | **MISSING** | — | R-811 |
|
| <a id="e15"></a>[E15](capability-map-detail/E.md#e15) | A second login step for the dashboard (a TOTP code or a passkey) | — | **MISSING** | — | R-811 |
|
||||||
| <a id="e16"></a>[E16](capability-map-detail/E.md#e16) | The household can leave Felhom, or outlive it — the box runs without the hub, the household owns its domain, tunnel and off-site account, and can export everything | — | **MISSING** | as a written answer | R-810 |
|
| <a id="e16"></a>[E16](capability-map-detail/E.md#e16) | The household can leave Felhom, or outlive it — the box runs without the hub, the household owns its domain, tunnel and off-site account, and can export everything | — | **MISSING** | as a written answer | R-810 |
|
||||||
| <a id="e17"></a>[E17](capability-map-detail/E.md#e17) | The host agent cannot reach root without the operator key: exact sudo patterns, no agent-written file installed where root reads it without a content check, the agent binary only by an operator-signed update checked as root | agent v0.146.1 (R-861; delivered by a step bundle, R-880) | **PROVEN-LIVE** | both demo boxes; three named residuals | `audits/hub-safety-2026-10-05/partF/` (2026-10-05) |
|
| <a id="e17"></a>[E17](capability-map-detail/E.md#e17) | The host agent cannot reach root without the operator key: exact sudo patterns, no agent-written file installed where root reads it without a content check, the agent binary only by an operator-signed update checked as root | agent v0.146.1 (R-861; delivered by a step bundle, R-880) | **PROVEN-LIVE** | both demo boxes; three named residuals; **re-check (2026-10-24):** mechanism changed after its evidence — agent e15d1d3 + a0c88e5 (sudoers + priv-apply, D4 drop-in) | `audits/hub-safety-2026-10-05/partF/` (2026-10-05) |
|
||||||
| <a id="e18"></a>[E18](capability-map-detail/E.md#e18) | WireGuard base infra always-on; OOB operator access (felhom-sshd, /32 peer) | agent v0.72, hub v0.35 | **IMPLEMENTED** | — | `audits/SPIKE-oob-wg-operator-peer-2026-07-05.md` (2026-07-05); `audits/SPIKE-felhom-sshd-2026-07-05.md` (2026-07-05) |
|
| <a id="e18"></a>[E18](capability-map-detail/E.md#e18) | WireGuard base infra always-on; OOB operator access (felhom-sshd, /32 peer) | agent v0.72, hub v0.35 | **IMPLEMENTED** | — | `audits/SPIKE-oob-wg-operator-peer-2026-07-05.md` (2026-07-05); `audits/SPIKE-felhom-sshd-2026-07-05.md` (2026-07-05) |
|
||||||
| <a id="e19"></a>[E19](capability-map-detail/E.md#e19) | The operator can see WHERE a managed host is — its LAN address and its WireGuard address, on the host page | agent v0.119.0, hub v0.85.0 | **PROVEN-LIVE** | — | `audits/host-addresses-visible-2026-07-31.md` (2026-07-31) |
|
| <a id="e19"></a>[E19](capability-map-detail/E.md#e19) | The operator can see WHERE a managed host is — its LAN address and its WireGuard address, on the host page | agent v0.119.0, hub v0.85.0 | **PROVEN-LIVE** | — | `audits/host-addresses-visible-2026-07-31.md` (2026-07-31) |
|
||||||
| <a id="e20"></a>[E20](capability-map-detail/E.md#e20) | The operator can see whether a managed host's guests still have working networking — and how often the watchdog had to repair them | agent v0.92.0 (emitter, 2026-07-21), hub v0.104.0 (reader, 2026-08-13) | **IMPLEMENTED** | not proven live | R-319 |
|
| <a id="e20"></a>[E20](capability-map-detail/E.md#e20) | The operator can see whether a managed host's guests still have working networking — and how often the watchdog had to repair them | agent v0.92.0 (emitter, 2026-07-21), hub v0.104.0 (reader, 2026-08-13) | **IMPLEMENTED** | not proven live | R-319 |
|
||||||
@@ -222,6 +223,7 @@
|
|||||||
| <a id="f7"></a>[F7](capability-map-detail/F.md#f7) | Always-on debug rings + on-demand log-bundle pulls with TTL/custody | controller v0.116, agent v0.83, hub v0.46 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-3-2026-07-11.md` (2026-07-11) |
|
| <a id="f7"></a>[F7](capability-map-detail/F.md#f7) | Always-on debug rings + on-demand log-bundle pulls with TTL/custody | controller v0.116, agent v0.83, hub v0.46 | **PROVEN-LIVE** | — | `audits/CAMPAIGN-3-2026-07-11.md` (2026-07-11) |
|
||||||
| <a id="f8"></a>[F8](capability-map-detail/F.md#f8) | Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | no corpus validation doc | operator infra; no evidence document |
|
| <a id="f8"></a>[F8](capability-map-detail/F.md#f8) | Operator alerting (Healthchecks → monitoring@felhom.eu) | k3s, Resend | **IMPLEMENTED** | no corpus validation doc | operator infra; no evidence document |
|
||||||
| <a id="f9"></a>[F9](capability-map-detail/F.md#f9) | Backup-deadline alerting (`expected_backup_missed`) is ANCHORED — absence of signal is UNKNOWN, not failure | hub v0.75.0 | **IMPLEMENTED** | — | `audits/DIAG-backup-missed-2026-07-26.md` (2026-07-26); R-82 |
|
| <a id="f9"></a>[F9](capability-map-detail/F.md#f9) | Backup-deadline alerting (`expected_backup_missed`) is ANCHORED — absence of signal is UNKNOWN, not failure | hub v0.75.0 | **IMPLEMENTED** | — | `audits/DIAG-backup-missed-2026-07-26.md` (2026-07-26); R-82 |
|
||||||
|
| <a id="f13"></a>[F13](capability-map-detail/F.md#f13) | A hacked app is noticed: an app that uploads or computes far more than its own numbers explain alarms the operator only; each box tests its upload line weekly and qBittorrent gets half of it (`08` §9; D41, D42; R-1244, R-1247) | controller v0.342.0, agent v0.164.0, hub v0.169.0, ep0 `felhom-iperf3` | **PROVEN-LIVE** | the alarm path on scratch 9202 (a hidden flood and a CPU burner, 2026-10-23); the line test on the three test boxes; the qBittorrent limit on demo-felhom. **Gaps:** a seeding qBittorrent is never tested (R-1250); an app's API can lie (R-1248) | `audits/SPIKE-upload-watch-2026-10-23.md` §7, `audits/line-test-2026-10-24/` (2026-10-24) |
|
||||||
| <a id="f12"></a>[F12](capability-map-detail/F.md#f12) | After a power cut, an app rebuilds the files the cut left empty, and the household sees any lost photo on the app page | controller v0.311.0, hub v0.150.0, catalog Immich block | **PROVEN-LIVE** | demo-hp 2026-10-14: a real cut mid-upload, 41 of 41 broken photos repaired, 120 controls untouched; the lost-photo note not exercised live (0 lost; unit- and 9202-tested). **Fleet delivery done 2026-10-14:** golden 0.311.0 + global floor; all three demo boxes on 0.311.0; catalog gate `unclean-boot-images` | `audits/immich-repair-2026-10-14/README.md` (2026-10-14); `09` §3 decision 236; R-1007 |
|
| <a id="f12"></a>[F12](capability-map-detail/F.md#f12) | After a power cut, an app rebuilds the files the cut left empty, and the household sees any lost photo on the app page | controller v0.311.0, hub v0.150.0, catalog Immich block | **PROVEN-LIVE** | demo-hp 2026-10-14: a real cut mid-upload, 41 of 41 broken photos repaired, 120 controls untouched; the lost-photo note not exercised live (0 lost; unit- and 9202-tested). **Fleet delivery done 2026-10-14:** golden 0.311.0 + global floor; all three demo boxes on 0.311.0; catalog gate `unclean-boot-images` | `audits/immich-repair-2026-10-14/README.md` (2026-10-14); `09` §3 decision 236; R-1007 |
|
||||||
| <a id="f11"></a>[F11](capability-map-detail/F.md#f11) | The household turns the whole machine off („A gép kikapcsolása"); the hub holds every silence alarm for 14 days and shows KIKAPCSOLVA | agent v0.155.0, hub v0.149.0, controller v0.310.0 | **PROVEN-LIVE** | proven on the N100 2026-10-13; fleet delivery waits on the floor | `audits/poweroff-2026-10-13/README.md` (2026-10-13); `architecture/14-host-power-off.md`; R-1000, R-1002 |
|
| <a id="f11"></a>[F11](capability-map-detail/F.md#f11) | The household turns the whole machine off („A gép kikapcsolása"); the hub holds every silence alarm for 14 days and shows KIKAPCSOLVA | agent v0.155.0, hub v0.149.0, controller v0.310.0 | **PROVEN-LIVE** | proven on the N100 2026-10-13; fleet delivery waits on the floor | `audits/poweroff-2026-10-13/README.md` (2026-10-13); `architecture/14-host-power-off.md`; R-1000, R-1002 |
|
||||||
| <a id="f10"></a>[F10](capability-map-detail/F.md#f10) | A box that is not always on (off at its backup time) makes the missed night up once when it comes back; the household sees a banner; the operator gets the missed-backup alarm, down or not | controller v0.295.0, hub v0.134.0, agent v0.145.0 | **PARTIAL** | catch-up and banner live; R-872 alarm by tests; suspend not measured | `audits/catchup-2026-10-05/partA/` (2026-10-05); R-872 |
|
| <a id="f10"></a>[F10](capability-map-detail/F.md#f10) | A box that is not always on (off at its backup time) makes the missed night up once when it comes back; the household sees a banner; the operator gets the missed-backup alarm, down or not | controller v0.295.0, hub v0.134.0, agent v0.145.0 | **PARTIAL** | catch-up and banner live; R-872 alarm by tests; suspend not measured | `audits/catchup-2026-10-05/partA/` (2026-10-05); R-872 |
|
||||||
@@ -236,7 +238,7 @@
|
|||||||
| <a id="g4"></a>[G4](capability-map-detail/G.md#g4) | Publish train: MinAgent floors, gated auto-Reissue, version channels, floor-field-LAST rules | hub v0.45/0.53, agent | **PARTIAL** | — | `runbooks/publish-train-rules.md` (2026-09-13); R-23 |
|
| <a id="g4"></a>[G4](capability-map-detail/G.md#g4) | Publish train: MinAgent floors, gated auto-Reissue, version channels, floor-field-LAST rules | hub v0.45/0.53, agent | **PARTIAL** | — | `runbooks/publish-train-rules.md` (2026-09-13); R-23 |
|
||||||
| <a id="g5"></a>[G5](capability-map-detail/G.md#g5) | Agent self-update: A/B slots, crash-loop auto-rollback, operator-signed | agent v0.70+ | **PROVEN-LIVE** | demo | `audits/SPIKE-agent-selfupdate-2026-07-05.md` (2026-07-05) |
|
| <a id="g5"></a>[G5](capability-map-detail/G.md#g5) | Agent self-update: A/B slots, crash-loop auto-rollback, operator-signed | agent v0.70+ | **PROVEN-LIVE** | demo | `audits/SPIKE-agent-selfupdate-2026-07-05.md` (2026-07-05) |
|
||||||
| <a id="g6"></a>[G6](capability-map-detail/G.md#g6) | Controller self-update: anonymous registry, no credentials in guest | controller v0.112 | **PROVEN-LIVE** | demo; the file found is the floor-driven self-update leg | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
|
| <a id="g6"></a>[G6](capability-map-detail/G.md#g6) | Controller self-update: anonymous registry, no credentials in guest | controller v0.112 | **PROVEN-LIVE** | demo; the file found is the floor-driven self-update leg | `audits/DRILL-day0-take2-2026-07-12.md` (2026-07-12) |
|
||||||
| <a id="g7"></a>[G7](capability-map-detail/G.md#g7) | Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue | hub v0.37–0.39 | **PROVEN-LIVE** | — | `audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md` (2026-07-09); `audits/SPIKE-hetzner-api-provisioning-2026-07-09.md` (2026-07-09) |
|
| <a id="g7"></a>[G7](capability-map-detail/G.md#g7) | Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue | hub v0.37–0.39 | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — hub 9688e76e (scoped abandonment, R-1182) | `audits/VALIDATION-offsite-provisioning-e2e-2026-07-09.md` (2026-07-09); `audits/SPIKE-hetzner-api-provisioning-2026-07-09.md` (2026-07-09) |
|
||||||
| <a id="g8"></a>[G8](capability-map-detail/G.md#g8) | Per-customer offsite fill + staleness + freeze lever | hub v0.41 | **IMPLEMENTED** | no live-fired leg | `audits/CAMPAIGN-offsite-overnight-2026-07-10.md` (2026-07-10) |
|
| <a id="g8"></a>[G8](capability-map-detail/G.md#g8) | Per-customer offsite fill + staleness + freeze lever | hub v0.41 | **IMPLEMENTED** | no live-fired leg | `audits/CAMPAIGN-offsite-overnight-2026-07-10.md` (2026-07-10) |
|
||||||
| <a id="g9"></a>[G9](capability-map-detail/G.md#g9) | Box-level Storage Box aggregate (total fill, Σ quotas, oversubscription alert) | hub v0.64.0 (R-5) | **IMPLEMENTED** | data pipeline live; UI render and alert mail not | hub v0.64.0 code and unit tests (outside `documentation/`) |
|
| <a id="g9"></a>[G9](capability-map-detail/G.md#g9) | Box-level Storage Box aggregate (total fill, Σ quotas, oversubscription alert) | hub v0.64.0 (R-5) | **IMPLEMENTED** | data pipeline live; UI render and alert mail not | hub v0.64.0 code and unit tests (outside `documentation/`) |
|
||||||
| <a id="g10"></a>[G10](capability-map-detail/G.md#g10) | Operator sees PBS DR datastore fill at a glance (Offsite "PBS DR" tab + dashboard gauge) | hub v0.65.0 + tenantsync v1.2.0 (R-5) | **IMPLEMENTED** | data pipeline live; UI render and alert mail not | hub v0.65.0 code and unit tests (outside `documentation/`) |
|
| <a id="g10"></a>[G10](capability-map-detail/G.md#g10) | Operator sees PBS DR datastore fill at a glance (Offsite "PBS DR" tab + dashboard gauge) | hub v0.65.0 + tenantsync v1.2.0 (R-5) | **IMPLEMENTED** | data pipeline live; UI render and alert mail not | hub v0.65.0 code and unit tests (outside `documentation/`) |
|
||||||
@@ -244,10 +246,10 @@
|
|||||||
| <a id="g12"></a>[G12](capability-map-detail/G.md#g12) | Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | — | the 2026-07-13 hub closing bundle (no evidence document) |
|
| <a id="g12"></a>[G12](capability-map-detail/G.md#g12) | Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | — | the 2026-07-13 hub closing bundle (no evidence document) |
|
||||||
| <a id="g13"></a>[G13](capability-map-detail/G.md#g13) | Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | — | hub v0.54 (2026-07-13; no evidence document) |
|
| <a id="g13"></a>[G13](capability-map-detail/G.md#g13) | Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | — | hub v0.54 (2026-07-13; no evidence document) |
|
||||||
| <a id="g14"></a>[G14](capability-map-detail/G.md#g14) | The operator sees boxes left behind: per-customer controller floors with their age (and which a global raise cannot move), each box's agent against the vouched one, a 7-day "agent behind" alarm and a "floor raise skipped boxes" mail | hub v0.135.0 (R-604, R-530) | **IMPLEMENTED** | the page live; the alarm and the mail unit-proven | `audits/hub-safety-2026-10-05/partD/` (2026-10-05) |
|
| <a id="g14"></a>[G14](capability-map-detail/G.md#g14) | The operator sees boxes left behind: per-customer controller floors with their age (and which a global raise cannot move), each box's agent against the vouched one, a 7-day "agent behind" alarm and a "floor raise skipped boxes" mail | hub v0.135.0 (R-604, R-530) | **IMPLEMENTED** | the page live; the alarm and the mail unit-proven | `audits/hub-safety-2026-10-05/partD/` (2026-10-05) |
|
||||||
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a nightly consistent copy, encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | — | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
|
| <a id="g15"></a>[G15](capability-map-detail/G.md#g15) | The hub database survives the loss of DooPlex: a consistent copy every 6 hours since D34 (nightly before 2026-10-23; corrected 2026-10-24), encrypted, on ep0; restore-tested weekly; an alarm when either stops | hub v0.136.0 (R-173), `scripts/hub-db-backup/`, homelab-manifests rules | **PROVEN-LIVE** | **re-check (2026-10-24):** mechanism changed after its evidence — be7e97dc + 52af38df (D34 6-hourly) | `audits/hub-db-offsite-2026-10-05/` (2026-10-05); `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
|
||||||
| <a id="g16"></a>[G16](capability-map-detail/G.md#g16) | Gitea (all code), the operator's password manager (Vaultwarden, since 2026-10-09 afternoon, R-923) and DooPlex's secrets survive the loss of DooPlex: a nightly encrypted copy on ep0; restore-tested weekly; an alarm when either stops; a failure mail | `scripts/dooplex-offsite/` (R-232), homelab-manifests rules | **PROVEN-LIVE** | — | `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
|
| <a id="g16"></a>[G16](capability-map-detail/G.md#g16) | Gitea (all code), the operator's password manager (Vaultwarden, since 2026-10-09 afternoon, R-923) and DooPlex's secrets survive the loss of DooPlex: a nightly encrypted copy on ep0; restore-tested weekly; an alarm when either stops; a failure mail | `scripts/dooplex-offsite/` (R-232), homelab-manifests rules | **PROVEN-LIVE** | — | `audits/dooplex-survival-2026-10-09/` (2026-10-09) |
|
||||||
| <a id="g17"></a>[G17](capability-map-detail/G.md#g17) | The operator surface refuses a cross-site form post on BOTH login paths (session token; Basic auth + `X-Felhom-Operator`) | hub v0.135.0 (R-135) | **PROVEN-LIVE** | — | `audits/hub-safety-2026-10-05/partA/live.txt` (2026-10-05) |
|
| <a id="g17"></a>[G17](capability-map-detail/G.md#g17) | The operator surface refuses a cross-site form post on BOTH login paths (session token; Basic auth + `X-Felhom-Operator`) | hub v0.135.0 (R-135) | **PROVEN-LIVE** | — | `audits/hub-safety-2026-10-05/partA/live.txt` (2026-10-05) |
|
||||||
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
|
| <a id="g18"></a>[G18](capability-map-detail/G.md#g18) | Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.143.0, hub v0.133.0; kernel lane agent v0.152.0+ | **PROVEN-LIVE** | all four lanes; the kernel lane on both demo boxes (2026-10-08, 2026-10-10); no automatic undo; **re-check (2026-10-24):** mechanism changed after its evidence — agent a0c88e5 (os-apply sshd check, D4) | `audits/kernel-night-2026-10-09/readback/RESULT.md` (2026-10-10); `audits/kernel-night-2026-10-08/readback/` (2026-10-08); `audits/os-guest-lane-2026-10-04/` (2026-10-04); `audits/os-host-lane-2026-10-04/` (2026-10-04); `audits/os-docker-crash-2026-10-04/` (2026-10-04) |
|
||||||
| <a id="g19"></a>[G19](capability-map-detail/G.md#g19) | An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps | controller v0.259.0 + hub v0.119.0 + ISO 1.29.0 + the whole catalog | **PROVEN-LIVE** | walked on 0.258.0 with one blocker; fixed, the walk not repeated | `audits/DRILL-first-hour-en-0258-2026-09-20.md` (2026-09-20); `audits/i18n-closing-2026-09-21/live/` (2026-09-21) |
|
| <a id="g19"></a>[G19](capability-map-detail/G.md#g19) | An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps | controller v0.259.0 + hub v0.119.0 + ISO 1.29.0 + the whole catalog | **PROVEN-LIVE** | walked on 0.258.0 with one blocker; fixed, the walk not repeated | `audits/DRILL-first-hour-en-0258-2026-09-20.md` (2026-09-20); `audits/i18n-closing-2026-09-21/live/` (2026-09-21) |
|
||||||
| <a id="g20"></a>[G20](capability-map-detail/G.md#g20) | A deletion of a customer's off-site history is NOTICED within a day | hub v0.111.0 (R-431) | **IMPLEMENTED** | not yet proven live | R-431 |
|
| <a id="g20"></a>[G20](capability-map-detail/G.md#g20) | A deletion of a customer's off-site history is NOTICED within a day | hub v0.111.0 (R-431) | **IMPLEMENTED** | not yet proven live | R-431 |
|
||||||
|
|
||||||
|
|||||||
@@ -16,15 +16,17 @@
|
|||||||
|
|
||||||
|
|
||||||
**Status:** draft (decisions from the topology/trust design sessions).
|
**Status:** draft (decisions from the topology/trust design sessions).
|
||||||
**Platform facts** referenced here live in `docs/proxmox-platform.md`; this document
|
**Platform facts** referenced here live in `documentation/proxmox-platform.md` (path corrected 2026-10-24); this document
|
||||||
records *Felhom's decisions*, not Proxmox behaviour.
|
records *Felhom's decisions*, not Proxmox behaviour.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 1. Model at a glance
|
## 1. Model at a glance
|
||||||
|
|
||||||
Three components. **Control is always box-initiated** — the hub never connects *into* a
|
Three components. **Control is always box-initiated** — the hub never ~~connects~~ sends a command *into* a
|
||||||
customer box.
|
customer box. **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** The hub may POKE a box: through ep0, one empty UDP datagram to the box's WireGuard
|
||||||
|
address (port 51822; `hub/internal/poke/client.go`, agent `internal/poke/poke.go`). A poke carries no command; it only
|
||||||
|
makes the box pull its desired state at once. The box still pulls.
|
||||||
|
|
||||||
```
|
```
|
||||||
operator side customer box (per Proxmox host)
|
operator side customer box (per Proxmox host)
|
||||||
@@ -98,14 +100,18 @@ credentials.
|
|||||||
|
|
||||||
- CGNAT does **not** force this: the Cloudflare Tunnel already makes a box reachable through
|
- CGNAT does **not** force this: the Cloudflare Tunnel already makes a box reachable through
|
||||||
Cloudflare's edge. We *choose* box-initiated control for the smallest attack surface — the
|
Cloudflare's edge. We *choose* box-initiated control for the smallest attack surface — the
|
||||||
box exposes no control endpoint at all.
|
box exposes no control endpoint ~~at all~~: no route on the box takes a command from the hub. **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]**
|
||||||
- The agent and the controller **poll** the hub; the hub never initiates inbound.
|
Two listeners sit behind the WireGuard tunnel: the agent's poke listener (a poke carries no command; the box still
|
||||||
|
pulls) and the operator door, `felhom-sshd` on the host, reachable only from the operator's `/32` peer — the
|
||||||
|
documented break-glass (`03` §5, `06` §4.5).
|
||||||
|
- The agent and the controller **poll** the hub; the hub never initiates ~~inbound~~ a command inbound — a poke only
|
||||||
|
makes them poll at once (corrected 2026-10-25, D45, R-1266).
|
||||||
- Operator actions are delivered as **signed jobs**: the agent verifies an operator signature
|
- Operator actions are delivered as **signed jobs**: the agent verifies an operator signature
|
||||||
before executing, so a compromised hub database alone cannot forge commands.
|
before executing, so a compromised hub database alone cannot forge commands.
|
||||||
- **[FACT, 2026-10-16, R-970]** The last contradiction is gone: until controller v0.322.0 the box's own hub API key,
|
- **[FACT, 2026-10-16, R-970]** The last contradiction is gone: until controller v0.322.0 the box's own hub API key,
|
||||||
sent as a Bearer token, opened `/api/config/` (read `controller.yaml` with every secret in it, and REPLACE it,
|
sent as a Bearer token, opened `/api/config/` (read `controller.yaml` with every secret in it, and REPLACE it,
|
||||||
unsigned) and the self-update and geo routes. Nothing called them (the hub never connects to a box). The config
|
unsigned) and the self-update and geo routes. Nothing called them (the hub never connects to a box). The config
|
||||||
routes are removed and no route accepts the hub key; every `/api/` route needs a dashboard session (pinned by
|
routes are removed and no route accepts the hub key; every `/api/` route but the read-only `/api/health` needs a dashboard session (pinned by
|
||||||
`TestR970_HubKeyOpensNoDoor` on the real route table). The box exposes no control endpoint, as stated above.
|
`TestR970_HubKeyOpensNoDoor` on the real route table). The box exposes no control endpoint, as stated above.
|
||||||
- All operator-initiated actions are recorded in a **customer-visible audit log**.
|
- All operator-initiated actions are recorded in a **customer-visible audit log**.
|
||||||
|
|
||||||
@@ -123,6 +129,7 @@ credentials.
|
|||||||
| box ↔ PBS | encrypted backups | outbound; per-customer namespace; client-side encryption | ciphertext only (operator can't read) |
|
| box ↔ PBS | encrypted backups | outbound; per-customer namespace; client-side encryption | ciphertext only (operator can't read) |
|
||||||
| guest ↔ Proxmox host | **(none direct)** | the guest holds no Proxmox creds; all via the agent | — |
|
| guest ↔ Proxmox host | **(none direct)** | the guest holds no Proxmox creds; all via the agent | — |
|
||||||
| hub ↔ Cloudflare API | geo-restriction WAF (enforcement) | the **hub** holds the CF API token; reconciles geo desired-state → WAF | the customer's zone/WAF |
|
| hub ↔ Cloudflare API | geo-restriction WAF (enforcement) | the **hub** holds the CF API token; reconciles geo desired-state → WAF | the customer's zone/WAF |
|
||||||
|
| new box ↔ public site | the installer script (`felhom.eu/scripts/felhom-host-install.sh`) + the website | **[DESIGN] 2026-10-30 (D35 / R-1228): Cloudflare Pages** (project `felhom-site`), uploaded by `scripts/publish-site-cloudflare.sh` from `main` + the installer tag; the contact form posts to `api.felhom.eu` (DooPlex's mailer). Before: DooPlex nginx + git-sync, kept 7 days as the way back | a changed installer runs as root on every new box — the script refuses an upload whose installer is not the tag's byte for byte |
|
||||||
|
|
||||||
**Every app is on the internet from its first minute** (`*.domain` through the tunnel), so an app's FIRST admin login is
|
**Every app is on the internet from its first minute** (`*.domain` through the tunnel), so an app's FIRST admin login is
|
||||||
a trust boundary too: a default password, or a "first visitor creates the admin" screen, is open to a stranger until the
|
a trust boundary too: a default password, or a "first visitor creates the admin" screen, is open to a stranger until the
|
||||||
@@ -132,7 +139,7 @@ of all 53 apps, 2026-09-28).
|
|||||||
**[DESIGN, 2026-10-17, R-1126, operator decision D11 = A] The tunnel runs before the claim, and FileBrowser never
|
**[DESIGN, 2026-10-17, R-1126, operator decision D11 = A] The tunnel runs before the claim, and FileBrowser never
|
||||||
answers on a default password.** The box starts cloudflared as soon as it has a tunnel token, before it is claimed,
|
answers on a default password.** The box starts cloudflared as soon as it has a tunnel token, before it is claimed,
|
||||||
because the claim itself is reached through it: the claim mail links `https://felhom.<domain>`
|
because the claim itself is reached through it: the claim mail links `https://felhom.<domain>`
|
||||||
(`hub/internal/notify/templates.go:27`). Before the claim a stranger reaches only the claim page (a code: 72 h life, 5
|
(`hub/internal/notify/templates.go` `FormatClaimEmail`; line corrected 2026-10-24). Before the claim a stranger reaches only the claim page (a code: 72 h life, 5
|
||||||
tries, 15-minute lockout — controller `web/claim.go`) and `files.<domain>`. FileBrowser starts every new database on
|
tries, 15-minute lockout — controller `web/claim.go`) and `files.<domain>`. FileBrowser starts every new database on
|
||||||
admin/admin, and its own admin-password setting cannot replace that (it re-applies at every start and would overwrite
|
admin/admin, and its own admin-password setting cannot replace that (it re-applies at every start and would overwrite
|
||||||
the household's own password — FileBrowser 1.5.6 `backend/cmd/user.go:50-57`, measured). So FileBrowser starts OFF
|
the household's own password — FileBrowser 1.5.6 `backend/cmd/user.go:50-57`, measured). So FileBrowser starts OFF
|
||||||
@@ -219,7 +226,10 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
|||||||
This removes any zero-touch remote-enrollment problem.
|
This removes any zero-touch remote-enrollment problem.
|
||||||
- A **one-time retrieval code** mints durable identity. Single-use (burned on the successful
|
- A **one-time retrieval code** mints durable identity. Single-use (burned on the successful
|
||||||
config fetch) plus a short *pre-use* TTL; one-click regenerate for the only real failure
|
config fetch) plus a short *pre-use* TTL; one-click regenerate for the only real failure
|
||||||
case (fetch fails before anything is persisted). After the fetch, the code is irrelevant —
|
case (fetch fails before anything is persisted). *(As built — corrected 2026-10-24: the passphrase is not burned
|
||||||
|
and has no TTL; it gives keys until the box has a host record AND a controller ≥ 0.334.0 has reported with its own
|
||||||
|
key, then `/config`, `/recovery` and `/host-enroll` answer 409 — D19, R-1159, `hub/internal/api/passphrase_scope.go:19-30`;
|
||||||
|
`05` §8 has the full rule.)* After the fetch, the code is irrelevant —
|
||||||
everything downstream runs on durable credentials, so retries don't need it.
|
everything downstream runs on durable credentials, so retries don't need it.
|
||||||
- **Order:** the agent enrolls first (and, running as root at setup, mints its own scoped
|
- **Order:** the agent enrolls first (and, running as root at setup, mints its own scoped
|
||||||
operator-tier Proxmox token), then provisions the customer LXC from the golden template and
|
operator-tier Proxmox token), then provisions the customer LXC from the golden template and
|
||||||
@@ -242,7 +252,8 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
|||||||
2026-10-08, R-900). Replacing it with our own relay is deferred until after the first customers (R-904, `09` §3
|
2026-10-08, R-900). Replacing it with our own relay is deferred until after the first customers (R-904, `09` §3
|
||||||
decision 184).
|
decision 184).
|
||||||
- **Outbound only** for control/report/backup (poll to hub, push to PBS). No inbound control
|
- **Outbound only** for control/report/backup (poll to hub, push to PBS). No inbound control
|
||||||
endpoint exists in the chosen model.
|
endpoint exists in the chosen model. **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** A poke carries no command; the box still pulls. The operator
|
||||||
|
door (`felhom-sshd` over the WireGuard tunnel) is the documented break-glass (§4, `03` §5).
|
||||||
- **Every customer has their OWN domain — never a name under `felhom.eu`** (operator ruling
|
- **Every customer has their OWN domain — never a name under `felhom.eu`** (operator ruling
|
||||||
2026-09-14). The free Cloudflare tier's certificate covers one level below a zone, so nested names
|
2026-09-14). The free Cloudflare tier's certificate covers one level below a zone, so nested names
|
||||||
such as `felhom.<customer>.felhom.eu` are not covered; the customer's own domain is a few thousand
|
such as `felhom.<customer>.felhom.eu` are not covered; the customer's own domain is a few thousand
|
||||||
@@ -253,7 +264,7 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
|||||||
2026-09-14 first-hour drill used a `*.felhom.eu` customer domain with no tunnel, and the dashboard link
|
2026-09-14 first-hour drill used a `*.felhom.eu` customer domain with no tunnel, and the dashboard link
|
||||||
in the setup-code mail did not resolve.
|
in the setup-code mail did not resolve.
|
||||||
- **A pasted Cloudflare API token must reach exactly the customer's own zone** (R-138 option C, `09` §3 decision 190;
|
- **A pasted Cloudflare API token must reach exactly the customer's own zone** (R-138 option C, `09` §3 decision 190;
|
||||||
hub main 2026-10-08, unreleased). On save the hub asks Cloudflare `GET /zones` with the token and keeps it only when
|
released in hub v0.144.0 — corrected 2026-10-24). On save the hub asks Cloudflare `GET /zones` with the token and keeps it only when
|
||||||
the token sees exactly ONE zone and that zone IS the customer's domain (a zone above it is refused: a sibling
|
the token sees exactly ONE zone and that zone IS the customer's domain (a zone above it is refused: a sibling
|
||||||
customer added later would be reachable). Cloudflare unreachable → the save fails and the old token stays. The token
|
customer added later would be reachable). Cloudflare unreachable → the save fails and the old token stays. The token
|
||||||
is never logged. **Limit:** `/zones` lists what the token can READ; a hand-built token with Zone:Read on one zone and
|
is never logged. **Limit:** `/zones` lists what the token can READ; a hand-built token with Zone:Read on one zone and
|
||||||
@@ -271,6 +282,17 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
|||||||
tunnel. Since controller v0.286.0 it sits ALONE on the `felhom-tunnel` network at the fixed address `172.16.253.2`
|
tunnel. Since controller v0.286.0 it sits ALONE on the `felhom-tunnel` network at the fixed address `172.16.253.2`
|
||||||
(traefik at `.3`), so traefik can believe forwarded headers from it and from nothing else (§5). Geo-restriction WAF
|
(traefik at `.3`), so traefik can believe forwarded headers from it and from nothing else (§5). Geo-restriction WAF
|
||||||
is **hub-enforced** (the hub holds the CF API token; the controller only reports geo desired-state).
|
is **hub-enforced** (the hub holds the CF API token; the controller only reports geo desired-state).
|
||||||
|
**Version and rule (2026-10-26, R-1300):** the image is `cloudflare/cloudflared:2026.10.3` since controller v0.345.1
|
||||||
|
(`infra.CloudflaredImage`). **cloudflared follows Cloudflare's security notices** — a notice is a patch release of
|
||||||
|
the controller, and a controller update recreates the container on the new image by itself
|
||||||
|
(`TestEnsureCloudflared_ANewImageRecreatesTheTunnel`). **Never turn off a certificate check to make an update
|
||||||
|
work.** The one check that is off today is the route's own: `*.<domain>` → `https://traefik` with `noTLSVerify`
|
||||||
|
(R-510; traefik shows its self-signed default certificate on a leg that never leaves the guest) — a recorded
|
||||||
|
exception, not a pattern to copy.
|
||||||
|
- **[DESIGN] 2026-10-30 — the public site and the installer are on Cloudflare Pages (D35 / R-1228).** `felhom.eu`
|
||||||
|
and `www` point to the Pages project `felhom-site` (uploaded by `scripts/publish-site-cloudflare.sh`); the contact
|
||||||
|
form posts to `api.felhom.eu` (DooPlex's mailer). The paragraph below describes DooPlex's copy, kept running 7 days
|
||||||
|
from the switch as the way back (`audits/release-2026-10-30/p3-cloudflare/PLAN.md` §5).
|
||||||
- **[DESIGN] The website's content comes from Gitea IN-CLUSTER** (2026-10-19). The `felhom-webpage` pod pulls
|
- **[DESIGN] The website's content comes from Gitea IN-CLUSTER** (2026-10-19). The `felhom-webpage` pod pulls
|
||||||
`website/` (from `main`) and `scripts/` (from the `installer-v…` tag, R-110) with four git-syncs that read
|
`website/` (from `main`) and `scripts/` (from the `installer-v…` tag, R-110) with four git-syncs that read
|
||||||
`http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git` — plain HTTP, safe only because it never
|
`http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git` — plain HTTP, safe only because it never
|
||||||
@@ -313,7 +335,8 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
|||||||
>
|
>
|
||||||
> **What being unmarked cost.** The consequence of this bullet — that 40 of 53 catalogue templates
|
> **What being unmarked cost.** The consequence of this bullet — that 40 of 53 catalogue templates
|
||||||
> declare no configurable path because they are all-hot — is stated as **[FACT]** at
|
> declare no configurable path because they are all-hot — is stated as **[FACT]** at
|
||||||
> `07-backup-architecture.md:296-299`. A reader met a marked observation beside an unmarked choice
|
> `07-backup-architecture.md:296-299`. *(Dated note 2026-10-24: that pointer is dead — those lines are about the local
|
||||||
|
> plaintext posture now, and no architecture file but this one states „40 of 53 … all-hot".)* A reader met a marked observation beside an unmarked choice
|
||||||
> and reasonably asked whether it *should* be so. **Between 19 and 22 August that reader called this
|
> and reasonably asked whether it *should* be so. **Between 19 and 22 August that reader called this
|
||||||
> decision a defect in four places** (R-370), and one session's work went into correcting the record
|
> decision a defect in four places** (R-370), and one session's work went into correcting the record
|
||||||
> rather than into the product.
|
> rather than into the product.
|
||||||
@@ -326,7 +349,12 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
|||||||
- **Key custody (zero-knowledge default):** three tiers the customer chooses —
|
- **Key custody (zero-knowledge default):** three tiers the customer chooses —
|
||||||
*customer-only* / *zero-knowledge escrow (default)* / *operator-managed*. Default escrows
|
*customer-only* / *zero-knowledge escrow (default)* / *operator-managed*. Default escrows
|
||||||
the **PBS passphrase-protected keyfile** in the hub, wrapped under a **customer recovery
|
the **PBS passphrase-protected keyfile** in the hub, wrapped under a **customer recovery
|
||||||
code** the operator can't open; DR needs the customer's code. Access-notification is an
|
code**; DR needs the customer's code. ~~the operator can't open~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** The hub keeps only the
|
||||||
|
locked key package, and every older one it retained; without the customer's code the operator cannot open any of
|
||||||
|
them, and from the off-site storage and the hub nobody can read the backup. Two exact limits, as the website says
|
||||||
|
since R-985: while the box runs, the operator has root access for support and with it could reach the data and the
|
||||||
|
backup key on the box; and the operator can open an OLDER off-site store from a retained package, by hand, when the
|
||||||
|
household types its old code on its own box (`runbooks/RUNBOOK-open-retained-package.md`, R-304). Access-notification is an
|
||||||
audit signal, never the primary guard. (Don't build bespoke crypto — use PBS's native
|
audit signal, never the primary guard. (Don't build bespoke crypto — use PBS's native
|
||||||
keyfile passphrase.)
|
keyfile passphrase.)
|
||||||
|
|
||||||
@@ -343,14 +371,19 @@ app. Controller down → the gated app answers an error, never the app. Measured
|
|||||||
restores from PBS; the controller returns on the same domain (tunnel reused from the hub
|
restores from PBS; the controller returns on the same domain (tunnel reused from the hub
|
||||||
record). DR = provisioning + a restore mode, not a separate mechanism.
|
record). DR = provisioning + a restore mode, not a separate mechanism.
|
||||||
- **Snapshot-before-deploy:** controller asks the agent to snapshot, deploys, runs its
|
- **Snapshot-before-deploy:** controller asks the agent to snapshot, deploys, runs its
|
||||||
post-deploy health check, asks the agent to roll back on failure. (Transient snapshot, §8.)
|
post-deploy health check, asks the agent to roll back on failure. (Transient snapshot, §8.) **[DESIGN — not built,
|
||||||
|
noted 2026-10-24]:** the agent serves `/snapshot` and `/rollback`, but no controller code calls them
|
||||||
|
(`felhom-controller/controller/internal/agentapi`); the update's guard is the per-app backup + undo of `09` §3 decision 15.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 10. How this embodies the product values
|
## 10. How this embodies the product values
|
||||||
|
|
||||||
- **Zero-knowledge offsite** — the operator holds the offsite backup but cannot read it.
|
- **Zero-knowledge offsite** — the operator holds the offsite backup but cannot read it without the customer's code
|
||||||
- **Box-initiated control + signed jobs** — no standing operator backdoor; a hub compromise
|
(the two exact limits are in §8 — corrected 2026-10-25, D45, R-1266).
|
||||||
|
- **Box-initiated control + signed jobs** — ~~no standing operator backdoor~~ no hidden operator door: the one
|
||||||
|
operator door is `felhom-sshd` over the WireGuard tunnel, the documented break-glass (`03` §5), and the website tells
|
||||||
|
the household about the operator's root access (R-985) — corrected 2026-10-25, D45, R-1266; a hub compromise
|
||||||
alone can't forge commands.
|
alone can't forge commands.
|
||||||
- **Customer-visible audit log** — every operator action is visible to the customer.
|
- **Customer-visible audit log** — every operator action is visible to the customer.
|
||||||
- **Never hold data hostage** — subscriptions cover ongoing labour (monitoring, offsite,
|
- **Never hold data hostage** — subscriptions cover ongoing labour (monitoring, offsite,
|
||||||
|
|||||||
@@ -105,7 +105,9 @@ Tri-state: `""` (unknown) · `"running"` · `"stopped"`.
|
|||||||
|
|
||||||
### In-flight — a marker file, one per owner
|
### In-flight — a marker file, one per owner
|
||||||
|
|
||||||
Two exist and they are deliberately **separate files**: `quiesce-state.json` (the whole-guest backup
|
Two exist and they are deliberately **separate files** *(four in-flight records by 2026-10-24: also `update-journal.json`
|
||||||
|
— the guarded update's crash journal read by `RecoverUpdates`, `internal/stacks/update.go:49,1306` — and
|
||||||
|
`.felhom-install-pending` — an interrupted install, R-681, `install_interrupted.go:28`; noted 2026-10-24)*: `quiesce-state.json` (the whole-guest backup
|
||||||
window, `internal/quiesce`) and `appstop-state.json` (app-data operations that stop an app —
|
window, `internal/quiesce`) and `appstop-state.json` (app-data operations that stop an app —
|
||||||
`backup.AppStopGuard`, covering the volume dump, offbox reconstitution and `.fab` export). **One
|
`backup.AppStopGuard`, covering the volume dump, offbox reconstitution and `.fab` export). **One
|
||||||
file, one writer**: sharing would give one record two lifetimes, and one owner clearing the other's
|
file, one writer**: sharing would give one record two lifetimes, and one owner clearing the other's
|
||||||
@@ -169,7 +171,8 @@ A recovery completing after the grace emits a `LATE RECOVERY` warning naming the
|
|||||||
|
|
||||||
**Nothing is started without asking whether it may be.** `bootrecon.StartGate` is the one question
|
**Nothing is started without asking whether it may be.** `bootrecon.StartGate` is the one question
|
||||||
the sweep asks per candidate, and it is **fail-safe: cannot determine ⇒ do not start.** Three holders
|
the sweep asks per candidate, and it is **fail-safe: cannot determine ⇒ do not start.** Three holders
|
||||||
answer it, and the last two only became reachable once the window widened past T+5 s:
|
answer it *(four since slice 4: check „1b", a guarded update moving the app — `IsUpdating`, `cmd/controller/main.go:2459-2477`;
|
||||||
|
noted 2026-10-24)*, and the last two only became reachable once the window widened past T+5 s:
|
||||||
|
|
||||||
| Holder | Why starting would be wrong |
|
| Holder | Why starting would be wrong |
|
||||||
|---|---|
|
|---|---|
|
||||||
@@ -218,6 +221,13 @@ own; every caller that is not the customer must decide for itself whether the ap
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
> *Dated note 2026-10-24 (review round 2): this inventory is v0.33's. On `main` (`321df1f`) 22 packages under
|
||||||
|
> `controller/internal/` have no row here: `aftercrash`, `agentapi`, `appbackup`, `backupwindow`, `bootrecon`,
|
||||||
|
> `bootstrap`, `channelhealth`, `crashboot`, `dockerexec`, `family`, `fillwatch`, `i18n`, `infra`, `logx`, `mailrelay`,
|
||||||
|
> `mediawire`, `nightchain`, `offsiteapply`, `privpolicy`, `quiesce`, `sockheal`, `upwatch` (an `ls` of the tree). Each
|
||||||
|
> package's own doc comment is its description; `sockheal` (re-heal after the guest's Docker socket is recreated,
|
||||||
|
> R-860) and `channelhealth` (a periodic proof of the controller→agent channel) are in no architecture document.*
|
||||||
|
|
||||||
## 2. Classification table (per package/file)
|
## 2. Classification table (per package/file)
|
||||||
|
|
||||||
### `cmd/`
|
### `cmd/`
|
||||||
@@ -266,6 +276,11 @@ own; every caller that is not the customer must decide for itself whether the ap
|
|||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| `client.go`,`zone.go`,`waf.go`,`geosync.go`,`countries.go` | **DELETE (→hub)** | The **hub** holds the CF API token and reconciles geo desired-state → WAF (doc 01 §5, doc 03 §2). The controller no longer calls the Cloudflare API — it reports geo desired-state up. The customer-facing geo *preference UI/data* stays (see `api/geo.go`). | needs-rework |
|
| `client.go`,`zone.go`,`waf.go`,`geosync.go`,`countries.go` | **DELETE (→hub)** | The **hub** holds the CF API token and reconciles geo desired-state → WAF (doc 01 §5, doc 03 §2). The controller no longer calls the Cloudflare API — it reports geo desired-state up. The customer-facing geo *preference UI/data* stays (see `api/geo.go`). | needs-rework |
|
||||||
|
|
||||||
|
*As built (noted 2026-10-24, review round 2): the `cloudflare/` package is still wired — with `cf_api_token` set the
|
||||||
|
controller builds a `GeoSyncManager` and calls the Cloudflare API (`cmd/controller/main.go:1992-2000`; the field is still in
|
||||||
|
`config.go:160`). The DELETE classification above is the plan, not the state; whether any box sets the token is not
|
||||||
|
readable from source.*
|
||||||
|
|
||||||
### `config/`, `crypto/`, `util/`
|
### `config/`, `crypto/`, `util/`
|
||||||
| File | Class | Reason | Risk |
|
| File | Class | Reason | Risk |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
@@ -325,6 +340,7 @@ household's file names. The household's lost list is shown on the app page (`los
|
|||||||
| File | Class | Reason | Risk |
|
| File | Class | Reason | Risk |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| `version.go` | **KEEP** | Semver parse / version string (still used for reporting). | clean |
|
| `version.go` | **KEEP** | Semver parse / version string (still used for reporting). | clean |
|
||||||
|
| *(as built 2026-10-24: both files below are LIVE — the updater decides the version and the floor and hands the container swap to the agent, `AgentSwapper`, `internal/selfupdate/updater.go:38-47`; `state.go` keeps `update-state.json`)* | | | |
|
||||||
| `state.go` | **DELETE (obsolete)** | Self-update audit state — the agent owns controller updates now (doc 03 §11). | clean |
|
| `state.go` | **DELETE (obsolete)** | Self-update audit state — the agent owns controller updates now (doc 03 §11). | clean |
|
||||||
| `updater.go` | **DELETE (→agent)** | Resolved (doc 03 §11): the controller is **agent-managed** — the agent snapshots → redeploys → health-gates → rolls back the controller. The controller's old self-update path (image pull + compose edit) is **removed**. | clean |
|
| `updater.go` | **DELETE (→agent)** | Resolved (doc 03 §11): the controller is **agent-managed** — the agent snapshots → redeploys → health-gates → rolls back the controller. The controller's old self-update path (image pull + compose edit) is **removed**. | clean |
|
||||||
|
|
||||||
@@ -387,7 +403,8 @@ dashboard alert is an `Alert` with two placement fields: `PageOnly` (the pages i
|
|||||||
page) and `Inline` (rendered by the page template in place, not by the layout's banner). `GetAlerts` returns
|
page) and `Inline` (rendered by the page template in place, not by the layout's banner). `GetAlerts` returns
|
||||||
the list (endpoint-drift, agent-channel and dead-app alerts first, then the rest sorted error > warning > info,
|
the list (endpoint-drift, agent-channel and dead-app alerts first, then the rest sorted error > warning > info,
|
||||||
capped at five plus an overflow line), and `layout.html` paints only those that are not `Inline` and match the
|
capped at five plus an overflow line), and `layout.html` paints only those that are not `Inline` and match the
|
||||||
page; `GetInlineAlerts(page)` hands the dashboard and monitoring pages their inline ones.
|
page *(as built since R-906 — noted 2026-10-24: pages use `GetBannerAlerts(page, lang)`, which filters per page BEFORE the
|
||||||
|
cap, `internal/web/alerts.go:353`, `handlers.go:144`)*; `GetInlineAlerts(page)` hands the dashboard and monitoring pages their inline ones.
|
||||||
|
|
||||||
**Exactly one warning is inline today:** the „storage is not on a separate drive" health warning. It is
|
**Exactly one warning is inline today:** the „storage is not on a separate drive" health warning. It is
|
||||||
`PageOnly: dashboard, monitoring` and `Inline: true`, so it sits quietly under the storage bars; **every other
|
`PageOnly: dashboard, monitoring` and `Inline: true`, so it sits quietly under the storage bars; **every other
|
||||||
@@ -821,7 +838,7 @@ R-438. The three facts below are the ones a reader needs before touching any of
|
|||||||
`Syncer.copyTemplates` (`sync/sync.go:319`) walks every directory in the catalog cache and copies
|
`Syncer.copyTemplates` (`sync/sync.go:319`) walks every directory in the catalog cache and copies
|
||||||
`docker-compose.yml` and `.felhom.yml` into the matching stack folder. **There is no test of whether
|
`docker-compose.yml` and `.felhom.yml` into the matching stack folder. **There is no test of whether
|
||||||
the app is deployed.** The only guard is a sha256 content compare (`copyIfChanged`) and the only
|
the app is deployed.** The only guard is a sha256 content compare (`copyIfChanged`) and the only
|
||||||
exclusion is `app.yaml`. Interval is `git.sync_interval`, default `15m` (`config/config.go:351`), plus
|
exclusion is `app.yaml`. Interval is `git.sync_interval`, default `15m` (`config/config.go:351` then, `:399` now; `copyTemplates` is now `sync.go:436`), plus
|
||||||
one immediate sync at controller start (`sync.go:98`).
|
one immediate sync at controller start (`sync.go:98`).
|
||||||
|
|
||||||
**SINCE v0.235.0** this walk still happens and `.felhom.yml` is still copied unconditionally, but the
|
**SINCE v0.235.0** this walk still happens and `.felhom.yml` is still copied unconditionally, but the
|
||||||
@@ -835,6 +852,7 @@ and they stay that way until something else acts.
|
|||||||
|
|
||||||
### 2. Every lifecycle path resolves that disagreement, silently, by upgrading
|
### 2. Every lifecycle path resolves that disagreement, silently, by upgrading
|
||||||
|
|
||||||
|
*(This section is the 2026-09 measurement; `UpdateStack` was deleted in v0.237.0 — the button calls `StartGuardedUpdate`, `09` §6.1 — noted 2026-10-24.)*
|
||||||
`StartStack`, `RestartStack` and `UpdateStack` (`stacks/manager.go:1029 / 1133 / 1170`) all end in
|
`StartStack`, `RestartStack` and `UpdateStack` (`stacks/manager.go:1029 / 1133 / 1170`) all end in
|
||||||
`docker compose up -d`. `up -d` makes the container match the file, **and pulls the image itself if it
|
`docker compose up -d`. `up -d` makes the container match the file, **and pulls the image itself if it
|
||||||
is absent** — measured at 18.3 s with a pull versus 0.5 s without, against a negative control
|
is absent** — measured at 18.3 s with a pull versus 0.5 s without, against a negative control
|
||||||
@@ -887,7 +905,9 @@ measured; that the restore writes to that path is read, not measured.
|
|||||||
beállítások csak olvashatók.", `deploy.html`). This is a design, not an oversight: a changed value would
|
beállítások csak olvashatók.", `deploy.html`). This is a design, not an oversight: a changed value would
|
||||||
need a guarded re-deploy that nothing performs. **Finding (2026-09-15):** the catalog field flag
|
need a guarded re-deploy that nothing performs. **Finding (2026-09-15):** the catalog field flag
|
||||||
`locked_after_deploy` (`stacks/metadata.go`) is parsed and read by **no** controller code — every field
|
`locked_after_deploy` (`stacks/metadata.go`) is parsed and read by **no** controller code — every field
|
||||||
is read-only after install whatever the catalog says. So a page must never tell a household to change
|
is read-only after install whatever the catalog says. *(Dated note 2026-10-24: it is now read into the app's
|
||||||
|
`LockedFields` (`internal/stacks/deploy.go:106,534`) and enforced only by `UpdateStackConfig`, which nothing calls — so
|
||||||
|
„read-only whatever the catalog says" still holds in practice.)* So a page must never tell a household to change
|
||||||
a value after install. **Vaultwarden (R-512)** follows the design instead of breaking it: registration
|
a value after install. **Vaultwarden (R-512)** follows the design instead of breaking it: registration
|
||||||
is closed by default and the household is invited from the admin panel, measured to work without mail
|
is closed by default and the household is invited from the admin panel, measured to work without mail
|
||||||
(stranger 400, invite 200, invited 200).
|
(stranger 400, invite 200, invited 200).
|
||||||
|
|||||||
@@ -47,7 +47,7 @@ Owns:
|
|||||||
1. **Proxmox lifecycle** — create/start/stop/destroy guests, snapshots, storage allocation. Via a scoped Proxmox API token (the **`FelhomAgent` operator role** — `proxmox-platform.md` §3.6, validated Phase 3 B3) for everything the API covers; raw host ops only where unavoidable.
|
1. **Proxmox lifecycle** — create/start/stop/destroy guests, snapshots, storage allocation. Via a scoped Proxmox API token (the **`FelhomAgent` operator role** — `proxmox-platform.md` §3.6, validated Phase 3 B3) for everything the API covers; raw host ops only where unavoidable.
|
||||||
2. **Storage management** — attach/classify targets, reconcile the storage manifest, mount USB-by-UUID, present mounts into guests.
|
2. **Storage management** — attach/classify targets, reconcile the storage manifest, mount USB-by-UUID, present mounts into guests.
|
||||||
3. **Backup/restore orchestration** — vzdump to the tiers, PBS, snapshot management, and the **self-restore-test**.
|
3. **Backup/restore orchestration** — vzdump to the tiers, PBS, snapshot management, and the **self-restore-test**.
|
||||||
4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub. **[FACT, 2026-10-04] The "cloudflared health" leg reads a host unit that does not exist:** `internal/hub/cloudflared.go` runs `systemctl is-active cloudflared` on the HOST, but cloudflared is a container in the guest (below), so every box reports `inactive` (`Unit cloudflared.service could not be found`, demo-hp). R-841. **[FACT, FIXED agent v0.141.0 / controller v0.292.0, R-841 CLOSED]** The agent now reads the guest's `cloudflared` container through the existing `pct exec [0-9]* -- docker inspect -f *` sudoers line: state, exit code and the Docker health status of a check the controller adds (`cloudflared tunnel --metrics localhost:20241 ready` → cloudflared's own `/ready`, 200 only with a connection). Three states: `running` (healthy), `not_running` (stopped, absent, or running but NOT connected), `unknown` (could not ask, or the check is still starting) — `unknown` never alarms. No new sudoers line. **[FACT, agent v0.142.0, R-852]** The host report gains `system`: the Proxmox version and kernel from the
|
4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub. **[FACT, 2026-10-04] The "cloudflared health" leg reads a host unit that does not exist:** `internal/hub/cloudflared.go` runs `systemctl is-active cloudflared` on the HOST, but cloudflared is a container in the guest (below), so every box reports `inactive` (`Unit cloudflared.service could not be found`, demo-hp). R-841. **[FACT, FIXED agent v0.141.0 / controller v0.292.0, R-841 CLOSED]** The agent now reads the guest's `cloudflared` container through the existing `pct exec [0-9]* -- docker inspect -f *` sudoers line: state, exit code and the Docker health status of a check the controller adds (`cloudflared tunnel --metrics localhost:20241 ready` → cloudflared's own `/ready`, 200 only with a connection). Four states (corrected 2026-10-24): `running` (healthy), `not_running` (stopped, or running but NOT connected), `unknown` (could not ask, or the check is still starting) — `unknown` never alarms — and since agent v0.161.0 `absent` (the guest has no cloudflared container at all, R-1070; the hub never alarms on it) — `internal/hub/cloudflared.go:12-23`. No new sudoers line. **[FACT, agent v0.142.0, R-852]** The host report gains `system`: the Proxmox version and kernel from the
|
||||||
Proxmox API, plus the OS wrapper's read-only `facts` (host Debian, next-boot kernel, held packages, taint, the crash
|
Proxmox API, plus the OS wrapper's read-only `facts` (host Debian, next-boot kernel, held packages, taint, the crash
|
||||||
guard; guest Debian, Docker engine, containerd, live-restore), at most every 10 min. **Still no new sudoers line** — the
|
guard; guest Debian, Docker engine, containerd, live-restore), at most every 10 min. **Still no new sudoers line** — the
|
||||||
facts, `live-restore-on` and the Docker layer all ride `FELHOM_OSAPPLY`. New ROOT-owned files instead (installer 1.30.0;
|
facts, `live-restore-on` and the Docker layer all ride `FELHOM_OSAPPLY`. New ROOT-owned files instead (installer 1.30.0;
|
||||||
@@ -64,7 +64,8 @@ Explicitly does **not**:
|
|||||||
- Serve application traffic or sit in the data path. **Control plane, not data plane**: if the agent dies, apps keep serving (Docker + LXC run without it); only *management* degrades — no new backups, no provisioning, hub loses the heartbeat.
|
- Serve application traffic or sit in the data path. **Control plane, not data plane**: if the agent dies, apps keep serving (Docker + LXC run without it); only *management* degrades — no new backups, no provisioning, hub loses the heartbeat.
|
||||||
- Hold or proxy customer application data.
|
- Hold or proxy customer application data.
|
||||||
- Run inside a guest. It is the thing that recovers guests and the host; it cannot be one of them.
|
- Run inside a guest. It is the thing that recovers guests and the host; it cannot be one of them.
|
||||||
- Manage **geo-restriction / the Cloudflare API**. Geo is hub-owned: the customer sets it in the controller UI, the controller reports the geo desired-state to the hub, and the **hub** (holding the CF API token) reconciles the WAF (S4). The agent manages only the *tunnel* service (`cloudflared`, §3/§5), never WAF rules.
|
- Manage **geo-restriction / the Cloudflare API**. Geo is hub-owned: the customer sets it in the controller UI, the controller reports the geo desired-state to the hub, and the **hub** (holding the CF API token) reconciles the WAF (S4). ~~The agent manages only the *tunnel* service (`cloudflared`, §3/§5), never WAF rules.~~ The agent manages no
|
||||||
|
Cloudflare piece: it only reads the guest's `cloudflared` container (§3) — corrected 2026-10-25, D45, R-1266.
|
||||||
|
|
||||||
## 3. Process model & host integration
|
## 3. Process model & host integration
|
||||||
|
|
||||||
@@ -129,13 +130,21 @@ the one trust root; the agent's API token is untouched; a box gets the new rule
|
|||||||
`agent_config_update` (the order: signed `agent_update` first, then the bundle — after the bundle, an agent below
|
`agent_config_update` (the order: signed `agent_update` first, then the bundle — after the bundle, an agent below
|
||||||
0.146.0 cannot update itself on that box).
|
0.146.0 cannot update itself on that box).
|
||||||
|
|
||||||
**[DESIGN, ruled 2026-10-22 15:47 (D4, R-982) — built as a ready patch, NOT shipped] Root SSH from the LAN by key only; the console password stays.** The config bundle gains one fixed file, `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`), checked and reloaded by `felhom-os-apply`; the installer's uninstall removes it. Why a drop-in: Debian's `sshd_config` Includes `sshd_config.d/*` FIRST and sshd keeps the first value it reads, so the drop-in wins over the stock line; the out-of-band operator sshd (`FELHOM_SSHD`, port 8822) has its own config with no Include and is not touched. Proven on a fresh ISO 1.30.0 install: password SSH refused after, key SSH and the PVE web password login (200) and the console login still work (`audits/night-2026-10-22/P2-D4/`). It ships with the next agent release + installer tag.
|
**[DESIGN, ruled 2026-10-22 15:47 (D4, R-982) — SHIPPED in agent v0.163.0 (`1109092`) and installer-v1.36.0 (`manifests/webpage.yaml:347`); corrected 2026-10-24] Root SSH from the LAN by key only; the console password stays.** The config bundle gains one fixed file, `/etc/ssh/sshd_config.d/10-felhom-root-key-only.conf` (`PermitRootLogin prohibit-password`), checked and reloaded by `felhom-os-apply`; the installer's uninstall removes it. Why a drop-in: Debian's `sshd_config` Includes `sshd_config.d/*` FIRST and sshd keeps the first value it reads, so the drop-in wins over the stock line; the out-of-band operator sshd (`FELHOM_SSHD`, port 8822) has its own config with no Include and is not touched. Proven on a fresh ISO 1.30.0 install: password SSH refused after, key SSH and the PVE web password login (200) and the console login still work (`audits/night-2026-10-22/P2-D4/`). It ships with the next agent release + installer tag.
|
||||||
|
**[DESIGN, operator ruling D43 = A, 2026-10-24 — committed agent `c8d3c17`, unreleased; R-1257]** **Appliances only.** The
|
||||||
|
drop-in's bundle policy is `appliance`: written only when the root-owned install record (`/var/lib/felhom-install/state.json`)
|
||||||
|
says `appliance` (the installer records the mode before it installs the bundle); on a `byo` record nothing is written and
|
||||||
|
a copy WE wrote (this path, beginning with our header line) is removed, then `sshd -t` and a reload; a box with no
|
||||||
|
readable record is left as it is. The „in force" check asks `sshd -T` for each of the host's own global IPv4 addresses
|
||||||
|
(a `Match Address <LAN>` block no longer reads as in force); a failing `sshd -t` puts back ONLY the SSH file and keeps
|
||||||
|
the rest of the bundle (the report says `sshd_step_undone`).
|
||||||
|
|
||||||
**[FACT, agent main `a0c88e5`, unreleased] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b).
|
**[FACT, agent `a0c88e5`, released in v0.163.0 — corrected 2026-10-24] The heartbeat reports a WireGuard handshake from the future (R-1204).** WireGuard refuses an initiation whose TAI64N time is not newer than the last one accepted from that peer, so a box that once ran with its clock ahead is locked out of ep0 until real time passes that moment. The agent used to clamp the negative age to 0 (it looked healthy); `wgtunnel.handshakeFuture` now sets `handshake_future_s` when the handshake is more than 60 s ahead (`TestR1204_AFutureHandshakeIsReported`). The hub alarms on it (`05` §15.4b).
|
||||||
|
|
||||||
**[DESIGN, operator ruling D30 2026-10-22; agent main `ae69ad4`, unreleased] The tunnel heals itself by a new key.** When
|
**[DESIGN, operator ruling D30 2026-10-22; agent `ae69ad4`, released in v0.163.0 — corrected 2026-10-24] The tunnel heals itself by a new key.** When
|
||||||
the latest handshake stays stamped in the future for 15 minutes, the agent makes a new WireGuard key pair (the old
|
the latest handshake stays stamped in the future for 15 minutes, the agent makes a new WireGuard key pair (the FIRST old
|
||||||
private key kept as `wg/private.key.prev`, 0600 — the escrow may hold it) and registers the new public key on the
|
private key kept as `wg/private.key.prev`, later ones as `wg/private.key.prev.<unix>`, the newest three kept, 0600 — the
|
||||||
|
escrow may hold one; the once-an-hour bound survives a restart in `wg/last-rekey` — `internal/wgtunnel/rekey.go`, corrected 2026-10-24) and registers the new public key on the
|
||||||
existing path, `POST /hosts/<id>/wg`: the hub re-keys the peer in place (same /32, generation bump) and its full-list
|
existing path, `POST /hosts/<id>/wg`: the hub re-keys the peer in place (same /32, generation bump) and its full-list
|
||||||
push to ep0 (`wgsync`, `wg syncconf`) drops the old key and adds the new one. A new key is a new peer for ep0, with no
|
push to ep0 (`wgsync`, `wg syncconf`) drops the old key and adds the new one. A new key is a new peer for ep0, with no
|
||||||
stored timestamp. Once an hour at most; not while the unit is down; never during a re-register already in flight. The
|
stored timestamp. Once an hour at most; not while the unit is down; never during a re-register already in flight. The
|
||||||
@@ -190,6 +199,49 @@ by verb**:
|
|||||||
neither lose nor invent one. New goldens run the controller with `--restart always`, which covers
|
neither lose nor invent one. New goldens run the controller with `--restart always`, which covers
|
||||||
only a Docker daemon restart.
|
only a Docker daemon restart.
|
||||||
|
|
||||||
|
- **`[DESIGN]` 2026-10-29 — the operator's „Restart controller" (agent `637c92c`, released v0.169.0; `04` §6.1).** The
|
||||||
|
host report's reply carries the AGENT's own closed list — `restart_controller <vmid>` only; the agent acts once per id
|
||||||
|
with the supervisor's act and guards (provisioned, running, not parked, no swap, not locked, no vzdump; an unreadable
|
||||||
|
guest list or vzdump state refuses), the brake counts these restarts with the supervisor's (3 in 15 minutes → refused,
|
||||||
|
then 30 minutes refused, `opBrakeUntil`), and the result rides the next report, requested at once. It works with the
|
||||||
|
controller stopped or stuck. **[FACT] Tester 1, 2026-10-29 13:01–13:02 CET:** four presses through the host page's
|
||||||
|
endpoint — poke to the agent in ~1 s, the controller healthy again after 9 s each, the result on the hub 15–17 s after
|
||||||
|
the press; the second press with the controller stopped by hand (`docker stop`) restarted it 5 s after the stop; the
|
||||||
|
fourth was refused („3 restarts in 15 minutes — presses are refused until 12:32 UTC") (`audits/release-2026-10-29b/`).
|
||||||
|
- **`[DESIGN]` 2026-10-29 — a controller that RUNS but does not answer (R-1344): ALARM ONLY.** Each sweep also reads
|
||||||
|
Docker's health status (the controller's own HEALTHCHECK, `curl -f /api/health` every 30 s, timeout 5 s); 6
|
||||||
|
consecutive „unhealthy" sweeps — on top of Docker's 3 failed probes, ~4.5 minutes in all — start an episode:
|
||||||
|
`unresponsive_since` in the stanza, the hub's operator-only `controller_unresponsive`. Not counted during a swap, a
|
||||||
|
parked or locked guest, or a vzdump. **It does not restart for it** until the R-1344 measurement (every probe, both
|
||||||
|
test boxes, through the 2026-10-29/30 night chain) shows a busy night never reads unhealthy that long.
|
||||||
|
- **`[DESIGN]` 2026-10-30 — R-1344 decided from the measurement: the alarm becomes a RESTART (agent `bd18347`,
|
||||||
|
released v0.171.0; hub 0.178.0 `controller_unresponsive_restart`).** The measurement (`audits/release-2026-10-30/
|
||||||
|
p1-r1344/`): 4 733 probes on Tester 1 and demo-hp, 2026-10-29 11:12 → 2026-10-30 07:00 UTC, through demo-hp's PBS
|
||||||
|
vzdump, Tester 1's local vzdump, both DB dumps, tier-2, the off-site push, a restore test and two controller updates —
|
||||||
|
no failed probe, no 5-s timeout, slowest 303 ms (demo-hp, during the PBS vzdump). So: the alarm stays at 6 unhealthy
|
||||||
|
sweeps; at **10** (2 minutes later, ~6.5 minutes after the first failed probe) the agent restarts the bootstrap unit
|
||||||
|
— unless the controller's busy file is fresh (a backup, a restore, an update, an export) or unreadable, or the
|
||||||
|
supervisor's brake refuses (3 restarts in 15 minutes → the crash-loop pause). A swap, a parked or locked guest and a
|
||||||
|
vzdump keep resetting the count. The restart counts in both brakes (fast and R-539's slow counter); after it the count
|
||||||
|
starts from zero. The stanza carries `unresponsive_restart_after_s` (the hub's alarm text says when the restart comes,
|
||||||
|
and says „does not restart" for an older agent) and `unresponsive_restart_at` (the hub mails
|
||||||
|
`controller_unresponsive_restart` once — not also `controller_restarted_by_agent`). Proven live on Tester 1
|
||||||
|
2026-10-30: the controller frozen (SIGSTOP) 09:24:50 UTC → Docker „unhealthy" 09:26:36 → the alarm 09:29:21 → the
|
||||||
|
restart 09:31:22 → healthy 09:31:24.
|
||||||
|
- **`[DESIGN]` 2026-10-29 — the operator's „Reboot guest" (D65 = A, agent `ca15a31`, released v0.170.0; `04` §6.1,
|
||||||
|
`09` §3 decision 284).** The agent's list gains `reboot_guest <vmid>`: the existing guest-reboot path (`pct reboot`
|
||||||
|
of THAT guest — never the host), for a stuck Docker or a stuck network inside the box. Guards, each refusing with the
|
||||||
|
reason: provisioned, running, not locked, not parked; the host-wide heavy-operation gate free (held from the checks
|
||||||
|
until the guest is up); `jobBlocker` (agent backup job, controller swap, drive/NAS job, escrow ceremony, vzdump); the
|
||||||
|
controller's busy file not fresh (the controller rewrites `data/ctl/busy` every 15 s during a backup, restore,
|
||||||
|
off-site run, night catch-up, app update or app export/import; older than 90 s = a controller that stopped writing,
|
||||||
|
not a busy one; any read failure but „no file" refuses); at most one per hour per guest (`<guests>/<vmid>/
|
||||||
|
operator-reboot.json`, persisted); the restart brake (a reboot counts as a restart). The guest-power watchdog and the
|
||||||
|
supervisor leave the guest alone while it reboots. The result names both times from the press („rebooted; the guest was
|
||||||
|
up after N s, the controller answered after M s"). The record is written BEFORE the reboot; `GET /guest/operator-reboot`
|
||||||
|
(self-scoped) hands it to the controller that starts after it, and the household's dashboard says „A Felhom
|
||||||
|
ügyfélszolgálata HH:MM-kor újraindította a dobozt." for a day.
|
||||||
|
|
||||||
**[RULING 2026-09-16, operator] The brake catches a FAST crash loop and is blind to a SLOW one — a
|
**[RULING 2026-09-16, operator] The brake catches a FAST crash loop and is blind to a SLOW one — a
|
||||||
second, slower counter is owed (R-539).** MEASURED the same day on the drill box: the controller was
|
second, slower counter is owed (R-539).** MEASURED the same day on the drill box: the controller was
|
||||||
killed four times, each kill 20 minutes after the last. Every one was restarted (dashboard back in
|
killed four times, each kill 20 minutes after the last. Every one was restarted (dashboard back in
|
||||||
@@ -232,10 +284,16 @@ alerts the operator when an agent misses its expected check-in window. This is t
|
|||||||
failure mode for a managed service, so it gets first-class treatment hub-side.
|
failure mode for a managed service, so it gets first-class treatment hub-side.
|
||||||
|
|
||||||
**Break-glass.** Standing inbound control is off. But when the poll loop *itself* is wedged
|
**Break-glass.** Standing inbound control is off. But when the poll loop *itself* is wedged
|
||||||
(agent hung, host sick) you cannot fix it through the poll loop. So there is an explicit,
|
(agent hung, host sick) you cannot fix it through the poll loop. So there is an explicit emergency path.
|
||||||
**off-by-default, customer-consented, fully-audited** emergency path: SSH to the host via
|
~~**off-by-default, customer-consented, fully-audited** emergency path: SSH to the host via
|
||||||
the Cloudflare Tunnel behind Cloudflare Access (or on-site). Enabling it is itself a signed,
|
the Cloudflare Tunnel behind Cloudflare Access (or on-site). Enabling it is itself a signed,
|
||||||
logged operation; it auto-expires.
|
logged operation; it auto-expires.~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** The path is the operator door: the operator's WireGuard
|
||||||
|
peer reaches a dedicated `felhom-sshd` on the host (a non-22 port, reachable only from the operator's `/32` over
|
||||||
|
`wg-felhom`, key-only login; `06` §4.5, TASK H1). It does not use the Cloudflare Tunnel. It is installed by default on
|
||||||
|
an appliance install, and a BYO install refuses it (the owner's call; `scripts/felhom-host-install.sh`). It stays on;
|
||||||
|
it is not opened per incident. When SSH itself is broken, the hub-vaulted `root@pam` password opens the Proxmox web
|
||||||
|
console (`runbooks/break-glass.md`). The website tells the household that the operator has root access for support
|
||||||
|
(R-985).
|
||||||
|
|
||||||
## 6. Agent ↔ controller local API
|
## 6. Agent ↔ controller local API
|
||||||
|
|
||||||
@@ -250,8 +308,9 @@ The controller (in its LXC) reaches the agent (on the host) over the local bridg
|
|||||||
- `POST /backup` — request a backup-now of *this* guest (enqueued; non-destructive).
|
- `POST /backup` — request a backup-now of *this* guest (enqueued; non-destructive).
|
||||||
- `GET /backup/due` — whether a policy-scheduled backup is due for *this* guest, so the controller can quiesce then call `POST /backup` (the app-consistent path, §8).
|
- `GET /backup/due` — whether a policy-scheduled backup is due for *this* guest, so the controller can quiesce then call `POST /backup` (the app-consistent path, §8).
|
||||||
- `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI. **`host_busy`
|
- `GET /backup/status`, `GET /restore-test/status` — read-only status for the controller's UI. **`host_busy`
|
||||||
(R-921, agent Unreleased `56aabc7`):** `GET /backup/status` also says whether the host-wide heavy-operation gate is
|
(R-921, agent `56aabc7`, released in v0.161.0 — corrected 2026-10-24):** `GET /backup/status` also says whether the host-wide heavy-operation gate is
|
||||||
held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off), the
|
held (`{busy, what}` — another guest's backup, the restore test, an OS step, the stale-lock sweep, a power-off, and since v0.164.0 the
|
||||||
|
weekly upload line test — gate name `line-test`), the
|
||||||
state in which `POST /backup` answers 409; absent when the gate is not wired. The controller (≥ 0.335.0) reads it
|
state in which `POST /backup` answers 409; absent when the gate is not wired. The controller (≥ 0.335.0) reads it
|
||||||
before it stops any app. A read, never a reservation.
|
before it stops any app. A read, never a reservation.
|
||||||
- **Crash-boot fact (R-856, agent v0.149.0):** `GET /host/crash-guard` — the host crash guard's last-boot record
|
- **Crash-boot fact (R-856, agent v0.149.0):** `GET /host/crash-guard` — the host crash guard's last-boot record
|
||||||
@@ -306,6 +365,49 @@ A controller can only `POST /rollback` (or snapshot/backup) **its own** guest
|
|||||||
token → guest and authorizes per guest, so a compromised controller's blast radius is
|
token → guest and authorizes per guest, so a compromised controller's blast radius is
|
||||||
**self-scoped and bounded** to its own guest.
|
**self-scoped and bounded** to its own guest.
|
||||||
|
|
||||||
|
**[DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265); every route listed below since
|
||||||
|
2026-10-25]:** the surface listed above is the slice-8 design; on `main` the local API registers **41** routes
|
||||||
|
(`felhom-agent/internal/localapi/server.go:562-634`). The list above names 15 of them; the other 26 are in the table
|
||||||
|
below (recounted 2026-10-25 — the review counted 24). Every route is behind the same per-guest token (`withGuest`).
|
||||||
|
Among them the controller can start a HOST action: `POST /host/line-test/run` (about 55 s; bounded by the due
|
||||||
|
date, the window, 6 tries a day and the heavy-operation gate — R-1250, unreleased). The local API also listens on the
|
||||||
|
R-50 island bridge (host `169.254.253.1`, guest `net1` `169.254.253.2/30`, `internal/config/config.go:260-269`). The
|
||||||
|
weekly line test keeps its state in the OOB state dir (`line-test.json`) and has an off switch, `line_test.disable`.
|
||||||
|
The served WireGuard block carries `oob_peer_ip` and `oob_operator_ssh_key` (`internal/hub/report.go`). The D30 re-key
|
||||||
|
runs only inside the tunnel watchdog, and its 15-minute count lives in memory — an agent restart starts it again (the
|
||||||
|
once-an-hour bound survives in `wg/last-rekey`).
|
||||||
|
|
||||||
|
The routes the list above does not name (read from `server.go:562-634` and each handler's own comment, 2026-10-25):
|
||||||
|
|
||||||
|
| Route | What it does |
|
||||||
|
|---|---|
|
||||||
|
| `GET /backup/tiers` | this guest's backup tiers, each with its cadence and whether its Proxmox storage exists now (`present` / `absent` / `unknown`) |
|
||||||
|
| `POST /backup/target` | moves the primary backup tier to a new storage: create → grant → config; answers `restart_required`, and the caller restarts the agent after its own in-flight check |
|
||||||
|
| `GET /host/line-test` | HOST: the weekly upload line test's last measurement; `present:false` when never measured (R-1247) |
|
||||||
|
| `POST /host/line-test/run` | HOST: starts a line test now — about 55 s, bounded as above (R-1250) |
|
||||||
|
| `GET /disks/candidates` | the host disks free for Felhom to enroll (`initialize`, and `attach` for those with a mountable file system) |
|
||||||
|
| `POST /disks/same-disk` | whether two of the guest's paths sit on the same physical disk, from the agent's own read (`16` §4) |
|
||||||
|
| `POST /disks/decommission` | takes the household's own drive out for good: a permanent intent, the guest bind pruned, unmounted; never formats; a system or backup mount is refused (403) |
|
||||||
|
| `GET /disks/format/status` | the newest or running format job, for a caller whose request timed out |
|
||||||
|
| `POST /disks/guest-attach` | binds an enrolled drive's `felhom-data` folder into this guest (read-write); the rest of the drive stays hidden |
|
||||||
|
| `GET /guest/memory` | this guest's RAM and its bounds |
|
||||||
|
| `POST /guest/memory` | a bounded RAM resize, applied live (R-24) |
|
||||||
|
| `POST /netstorage/add` | mounts a NAS share on the host and starts a verify job; a failed verify rolls it back |
|
||||||
|
| `GET /netstorage` | the NAS shares |
|
||||||
|
| `GET /netstorage/verify-status` | the verify job's state |
|
||||||
|
| `POST /netstorage/remove` | removes a NAS share |
|
||||||
|
| `POST /controller/swap` | swaps the controller image (202, then async); the agent verifies and rolls back |
|
||||||
|
| `GET /controller/swap/status` | the swap's state |
|
||||||
|
| `POST /escrow/stage-secret` | stages the off-site repository password for the escrow ceremony (0600, never logged) |
|
||||||
|
| `DELETE /escrow/stage-secret` | wipes the staged password |
|
||||||
|
| `POST /escrow/recover-offsite-password` | with the recovery code in the body, returns only the off-site repository password (§8a) |
|
||||||
|
| `GET /escrow/preflight` | the ceremony's checklist |
|
||||||
|
| `POST /escrow/ceremony` | starts the root ceremony job |
|
||||||
|
| `GET /escrow/ceremony/status` | the ceremony's state |
|
||||||
|
| `POST /escrow/ceremony/claim` | the one-shot hand-off of the new recovery code; a later claim is 410 |
|
||||||
|
| `POST /escrow/ceremony/confirm` | the household's two words; only then does the package reach the hub (R-1151) |
|
||||||
|
| `GET /debug/logs` | the agent's debug log ring, for the controller's Debug page |
|
||||||
|
|
||||||
### 6a. Implementation (slice 8A — implemented)
|
### 6a. Implementation (slice 8A — implemented)
|
||||||
|
|
||||||
**Status: implemented** (agent v0.10.0 `internal/localapi`; controller v0.35.0 `internal/bootstrap`
|
**Status: implemented** (agent v0.10.0 `internal/localapi`; controller v0.35.0 `internal/bootstrap`
|
||||||
@@ -314,7 +416,8 @@ token → guest and authorizes per guest, so a compromised controller's blast ra
|
|||||||
quiesce-on-due consumer is 8B), the rest wrap the existing slice-5/6/7 machinery.
|
quiesce-on-due consumer is 8B), the rest wrap the existing slice-5/6/7 machinery.
|
||||||
|
|
||||||
- **Transport / pin.** The agent serves a **persisted self-signed leaf** bound to the host bridge IP
|
- **Transport / pin.** The agent serves a **persisted self-signed leaf** bound to the host bridge IP
|
||||||
on a fixed port (default `:8443`). The controller pins the **leaf-cert SHA-256** (decision:
|
on a fixed port — `local_api.listen_addr` is REQUIRED, there is no default (`internal/config/config.go:317-319`); an
|
||||||
|
island install (R-50) listens on `169.254.253.1:8443` on its private bridge (corrected 2026-10-24). The controller pins the **leaf-cert SHA-256** (decision:
|
||||||
consistency with the agent's Proxmox/PBS cert pinning), carried in its bootstrap. The leaf is
|
consistency with the agent's Proxmox/PBS cert pinning), carried in its bootstrap. The leaf is
|
||||||
generated **once and persisted**, so its fingerprint is stable across agent restarts (a fresh cert
|
generated **once and persisted**, so its fingerprint is stable across agent restarts (a fresh cert
|
||||||
each boot would invalidate every already-issued bootstrap pin). Defense-in-depth: the listener
|
each boot would invalidate every already-issued bootstrap pin). Defense-in-depth: the listener
|
||||||
@@ -420,8 +523,8 @@ per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a ba
|
|||||||
> space on …" — so the hub raises `restore_test_failed`; it is never a pass and never dropped.
|
> space on …" — so the hub raises `restore_test_failed`; it is never a pass and never dropped.
|
||||||
> - **Leftovers on a timer.** A failed scratch teardown and the stale-lock sweep (R-673) run every 10
|
> - **Leftovers on a timer.** A failed scratch teardown and the stale-lock sweep (R-673) run every 10
|
||||||
> minutes, not only at agent start; the sweep holds the one-heavy-operation gate; after 3 failed
|
> minutes, not only at agent start; the sweep holds the one-heavy-operation gate; after 3 failed
|
||||||
> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent Unreleased
|
> teardown tries the operator is told. **A half-removed `vzdump` snapshot (R-1010, agent
|
||||||
> `56aabc7`):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's
|
> `56aabc7`, released in v0.161.0 — corrected 2026-10-24):** when the sweep's delete fails ONLY with „Failed to find logical volume …_vzdump" (one disk's
|
||||||
> snapshot already gone), it retries once with Proxmox's own `force=1` on the same API call (no sudoers line, no
|
> snapshot already gone), it retries once with Proxmox's own `force=1` on the same API call (no sudoers line, no
|
||||||
> root command); any other error is returned unchanged, and a failed forced delete is never forced again.
|
> root command); any other error is returned unchanged, and a failed forced delete is never forced again.
|
||||||
> - **A thin pool ≥ 90 % requests an immediate report**; the hub judges a thin pool on the worse of data
|
> - **A thin pool ≥ 90 % requests an immediate report**; the hub judges a thin pool on the worse of data
|
||||||
@@ -450,7 +553,7 @@ per Part 1: **snapshot** (LVM-thin, transient, whole-guest rollback — not a ba
|
|||||||
backup is due (`GET /backup/due`, §6) → **quiesces** (stops its app stacks) → `POST /backup` →
|
backup is due (`GET /backup/due`, §6) → **quiesces** (stops its app stacks) → `POST /backup` →
|
||||||
polls `GET /backup/status` to `done` → **unquiesces** (restarts exactly the stacks it stopped).
|
polls `GET /backup/status` to `done` → **unquiesces** (restarts exactly the stacks it stopped).
|
||||||
Implemented in `felhom-controller` v0.36.0 (`internal/quiesce`) + `felhom-agent` v0.11.0 (the
|
Implemented in `felhom-controller` v0.36.0 (`internal/quiesce`) + `felhom-agent` v0.11.0 (the
|
||||||
`/backup/due` cadence policy + `/backup/status` phases). **An agent-initiated vzdump is
|
`/backup/due` cadence policy + `/backup/status` phases). *[DESIGN] R-1232 (2026-10-26 night, agent main `3911cc6`, next agent release): on an encrypted tier the due-check counts only archives written with THIS storage's key (the R-727 discriminator — an archive carries no host id), so a reinstalled box does not read the old box's archive as its own last backup.* **An agent-initiated vzdump is
|
||||||
crash-consistent only** (there is no inbound-to-guest channel to trigger a quiesce — §3/§5); the
|
crash-consistent only** (there is no inbound-to-guest channel to trigger a quiesce — §3/§5); the
|
||||||
controller stopping its stacks first is what makes the captured state **clean-shutdown-consistent**
|
controller stopping its stacks first is what makes the captured state **clean-shutdown-consistent**
|
||||||
(validated live: a quiesced postgres restore comes up clean — "database system was shut down" — vs
|
(validated live: a quiesced postgres restore comes up clean — "database system was shut down" — vs
|
||||||
@@ -509,6 +612,16 @@ never opens, re-seals or logs it; the audit (`escrow_carried`) carries host ids
|
|||||||
hosts → no carry, `escrow_carry_refused` to the operator. The box then sees „identity present", its mint guard (R-241)
|
hosts → no carry, `escrow_carry_refused` to the operator. The box then sees „identity present", its mint guard (R-241)
|
||||||
holds, and it offers the recovery screen. Proven live 2026-10-21 (`audits/fixes-2026-10-21/P1-live/`: the right code
|
holds, and it offers the recovery screen. Proven live 2026-10-21 (`audits/fixes-2026-10-21/P1-live/`: the right code
|
||||||
opened the household's history, Nextcloud's 3 031 files came back byte-identical, the next off-site run appended).
|
opened the household's history, Nextcloud's 3 031 files came back byte-identical, the next off-site run appended).
|
||||||
|
**[DESIGN] 2026-10-28 night — WHICH retained row is carried (R-1255 H3; felhom.eu `1ee6d5a5`, rides the next hub
|
||||||
|
release):** the newest retained row that has an identity blob AND a repository-password hash; only when no row has a
|
||||||
|
hash, the newest row as before (a legacy package whose hash was never recorded may still seal the password — never fewer
|
||||||
|
ways back). Before, the newest row alone was taken: a newer package without the password was carried while the
|
||||||
|
household's real one stayed under the deleted host. **And the set-aside purge (R-1255 H4):** a set-aside is named
|
||||||
|
`<repo>.orphaned-<YYYYMMDDTHHMMSSZ>` (unique per run; was the date, which a same-day abandonment freed for the next
|
||||||
|
set-aside), and a retained row is purged only when it was tagged with the deleted path no later than that path's
|
||||||
|
deletion (`repo_tagged_at`; a row tagged before the column existed uses `superseded_at`) — a reused path's later copy is
|
||||||
|
still stored, and its packages stay. Store tests `TestR1255_*`, red before the fix; no live proof (a hub deploy is not
|
||||||
|
tonight's).
|
||||||
|
|
||||||
#### The separation principle (the rule that governs every posture)
|
#### The separation principle (the rule that governs every posture)
|
||||||
Reading customer data needs **BOTH** the encrypted chunks **AND** a usable key. **Zero-knowledge
|
Reading customer data needs **BOTH** the encrypted chunks **AND** a usable key. **Zero-knowledge
|
||||||
@@ -592,6 +705,15 @@ nothing waiting 404, hub failure 502 (still waiting). A controller that has not
|
|||||||
R-978 order. Proven on demo-hp 2026-10-19: ceremony 1 typed back → code 1 opens the hub's package, code 0 is a kept
|
R-978 order. Proven on demo-hp 2026-10-19: ceremony 1 typed back → code 1 opens the hub's package, code 0 is a kept
|
||||||
earlier one (422); ceremony 2 shown and abandoned → code 1 still opens it, code 2 opens nothing
|
earlier one (422); ceremony 2 shown and abandoned → code 1 still opens it, code 2 opens nothing
|
||||||
(`audits/day-2026-10-19/P1/escrow-*.txt`).
|
(`audits/day-2026-10-19/P1/escrow-*.txt`).
|
||||||
|
**[DESIGN] 2026-10-28 night — a lost or late answer (R-1254 H2; agent `5ddd124` + controller `e33cffa`, unreleased):**
|
||||||
|
the upload at the claim or the confirm can lose its answer AFTER the hub stored the package, and the household was then
|
||||||
|
told „the previous code still works" — false. Now, after a failed upload, the agent asks the hub which package it holds
|
||||||
|
(`GET /api/v1/hosts/<id>/escrow`, the identity blob compared byte for byte, ciphertext only, 15 s): THIS package → a
|
||||||
|
success; another package → 502, the old sentence (now true); no answer → **504**, a sentence that promises nothing about
|
||||||
|
either code, the wait goes on. The controller words 504 and its own failed call as „Nem tudjuk biztosan… Őrizd meg
|
||||||
|
mindkét kódot…" (`api.escrow.confirm_store_unknown`), and waits up to 60 s for the claim and the confirm (was the
|
||||||
|
client's 15 s — shorter than the agent's 30 s upload, so an abandoned claim could store a package whose code reached
|
||||||
|
nobody). Tests `TestR1254_*` in both repos, red first. Live proof with the agent release (the vouch).
|
||||||
|
|
||||||
#### Custody across generations — what the hub retains when a blob is superseded (hub v0.60.0; **contract corrected hub v0.93.0, R-198**)
|
#### Custody across generations — what the hub retains when a blob is superseded (hub v0.60.0; **contract corrected hub v0.93.0, R-198**)
|
||||||
|
|
||||||
@@ -677,7 +799,8 @@ the *archive* and the *back half*, **and in identity policy** (below).
|
|||||||
`swap`, and the agent never set it; on the 8 GB minimum machine (guest 4096 MB) the household's first uploads filled the
|
`swap`, and the agent never set it; on the 8 GB minimum machine (guest 4096 MB) the household's first uploads filled the
|
||||||
guest with its swap full, the kernel could evict only program code, and the box froze (memory „full" pressure 96 %, load
|
guest with its swap full, the kernel could evict only program code, and the box froze (memory „full" pressure 96 %, load
|
||||||
162, the dashboard dead 6½ minutes). `[DESIGN]` The size step sets `swap` = the guest's memory: at bring-up, at a RAM
|
162, the dashboard dead 6½ minutes). `[DESIGN]` The size step sets `swap` = the guest's memory: at bring-up, at a RAM
|
||||||
resize (`POST /guest/memory`), and once a minute after agent start for every pool LXC whose swap is below its RAM (live,
|
resize (`POST /guest/memory`), and ONCE, one minute after agent start, for every pool LXC whose swap is below its RAM (`cmd/felhom-agent/main.go`
|
||||||
|
`EnsureGuestSwap`; „once a minute" corrected 2026-10-24) (live,
|
||||||
no restart, never lowers). Proven on a fresh box the night of 2026-10-21 (`audits/night-2026-10-21/P1-soak/`: 512 →
|
no restart, never lowers). Proven on a fresh box the night of 2026-10-21 (`audits/night-2026-10-21/P1-soak/`: 512 →
|
||||||
4096 one minute after the fixture agent started; the 2-hour household load then answered every 10-second probe). **The
|
4096 one minute after the fixture agent started; the 2-hour household load then answered every 10-second probe). **The
|
||||||
host must have swap behind it:** our installer gives an 8 GB / 256 GB machine 7.75 GB (measured the same night); a BYO
|
host must have swap behind it:** our installer gives an 8 GB / 256 GB machine 7.75 GB (measured the same night); a BYO
|
||||||
|
|||||||
@@ -15,9 +15,9 @@
|
|||||||
|
|
||||||
|
|
||||||
|
|
||||||
> Status: design draft (decision content), grounded on `docs/tests/phase4-signing-findings.md`.
|
> Status: design draft (decision content), grounded on `documentation/tests/phase4-signing-findings.md`.
|
||||||
> To be reviewed by Claude Code against that spike + `03` §4, then placed at
|
> To be reviewed by Claude Code against that spike + `03` §4, then placed at
|
||||||
> `docs/architecture/04-control-plane-authorization.md`.
|
> `documentation/architecture/04-control-plane-authorization.md` — where it is now (paths corrected 2026-10-24).
|
||||||
>
|
>
|
||||||
> Builds on Part 1 (enrollment / trust), Part 3 (the agent verifies + the §4 reversibility gate).
|
> Builds on Part 1 (enrollment / trust), Part 3 (the agent verifies + the §4 reversibility gate).
|
||||||
> This doc defines the **mechanism** behind `03` §4's "an operator signature the hub can't forge."
|
> This doc defines the **mechanism** behind `03` §4's "an operator signature the hub can't forge."
|
||||||
@@ -75,7 +75,14 @@ recorded last**, only after everything else passes, so an invalid signature can
|
|||||||
nonce (DoS-safe). Each layer is mandatory and was proven to reject independently (Phase 4 §3–4):
|
nonce (DoS-safe). Each layer is mandatory and was proven to reject independently (Phase 4 §3–4):
|
||||||
- **target binding** — `target.host_id`/`guest_id` must equal *this* box/guest (a signature for box
|
- **target binding** — `target.host_id`/`guest_id` must equal *this* box/guest (a signature for box
|
||||||
A cannot be replayed at box B);
|
A cannot be replayed at box B);
|
||||||
- **time window** — `now ∈ [issued_at, expires_at]`;
|
- **time window** — `now ∈ [issued_at, expires_at]`; as built, `issued_at` may be up to **2 minutes** in the future
|
||||||
|
(clock skew), `expires_at` is never extended — the agent (`felhom-agent/internal/authz/verifier.go:19-22,148-155`) and
|
||||||
|
the root wrapper (`configs/felhom-os-apply`) alike (corrected 2026-10-24). **[DESIGN — the operator picked a code change
|
||||||
|
2026-10-25 (D44); row R-1280]:** neither caps the window's LENGTH (`expires_at − issued_at`); `felhom-opsign` defaults to 30 min
|
||||||
|
(`cmd/felhom-opsign/main.go:57`), and the nonce store keeps a record until its expiry. The pick: the agent and the
|
||||||
|
root wrapper refuse a window longer than 24 h. **Built 2026-10-25 night (agent `d912307`, unreleased — rides the next
|
||||||
|
agent release):** `authz.MaxWindow = 24h` (`ErrWindowTooLong`), the wrapper's `MAX_WINDOW_S`, and `felhom-opsign -ttl`
|
||||||
|
refuses more; exactly 24 h is accepted;
|
||||||
- **nonce** — unseen within the window (the nonce store **must be persistent across agent restarts**
|
- **nonce** — unseen within the window (the nonce store **must be persistent across agent restarts**
|
||||||
and expiry-pruned; a non-persistent store reopens the replay window after a restart).
|
and expiry-pruned; a non-persistent store reopens the replay window after a restart).
|
||||||
|
|
||||||
@@ -166,7 +173,7 @@ op the agent verifies** (same pipeline, §2.3) — never unauthenticated config.
|
|||||||
queue — then the agent polls, verifies, executes, and audits. One command + passphrase, from the
|
queue — then the agent polls, verifies, executes, and audits. One command + passphrase, from the
|
||||||
desk. **Never** a site visit.
|
desk. **Never** a site visit.
|
||||||
|
|
||||||
### 6.1 Operator actions in the report reply [DESIGN, R-314/R-279 — `09` §3 decision 185, controller + hub main 2026-10-08, unreleased]
|
### 6.1 Operator actions in the report reply [DESIGN, R-314/R-279 — `09` §3 decision 185; released in controller v0.304.0 + hub v0.144.0 — corrected 2026-10-24]
|
||||||
|
|
||||||
Routine, non-destructive operator requests need no signature (`03` §4 asks for one only to destroy or overwrite the
|
Routine, non-destructive operator requests need no signature (`03` §4 asks for one only to destroy or overwrite the
|
||||||
only copy). The hub stores a row, bumps the box's intent, and the report reply carries `operator_actions:[{id,action,arg}]`
|
only copy). The hub stores a row, bumps the box's intent, and the report reply carries `operator_actions:[{id,action,arg}]`
|
||||||
@@ -176,6 +183,32 @@ the deletion is the hub's), `run_job` (`fill-watch`, `offsite-integrity`, `offsi
|
|||||||
Anything else is refused at the hub's POST and again on the box. No action deletes data, starts a countdown or
|
Anything else is refused at the hub's POST and again on the box. No action deletes data, starts a countdown or
|
||||||
shortens one (`TestOpActions_ClosedList`, `TestOperatorActions_ClosedList`). Unanswered after 24 h: expired.
|
shortens one (`TestOpActions_ClosedList`, `TestOperatorActions_ClosedList`). Unanswered after 24 h: expired.
|
||||||
|
|
||||||
|
**`[DESIGN]` 2026-10-29 — D64 = B, and the agent's own list (operator ruling 12:10; `09` §3 decision 283).** The
|
||||||
|
controller's list gains **`clear_hold <app>`** for every hold reason, with no signature: the RUNNING controller clears the
|
||||||
|
hold in memory and saves at once (no later settings save can write it back — R-1343). The operator accepted the cost in
|
||||||
|
words: a compromised hub could release a data-protecting hold and start an app on half-restored data; the hub's confirm
|
||||||
|
step shows the hold's reason and, for a restore or put-back hold, that warning. **R-379 („only a person clears a hold")
|
||||||
|
now reads „a person — on the box, or the operator on the hub".** The hub refuses `clear_hold` for an app the box does not
|
||||||
|
report held now. **The AGENT has its own closed list on the host report's reply — `restart_controller <vmid>` only**
|
||||||
|
(`store.ValidateHostOperatorAction` / agent `localapi.OperatorActionNames`; never a shared verb, `TestHostOperatorActions_ClosedList`):
|
||||||
|
the supervisor's restart with its guards, the 3-in-15-minutes brake counting operator restarts too (then 30 minutes
|
||||||
|
refused), the result in the next report (requested at once), a vmid that is not a guest of that host refused at the hub.
|
||||||
|
Neither verb deletes data, starts a countdown or shortens one. ~~A guest reboot from the hub is NOT on either list.~~
|
||||||
|
(Superseded the same day by D65 below.)
|
||||||
|
|
||||||
|
**`[DESIGN]` 2026-10-29 — D65 = A: `reboot_guest <vmid>` joins the AGENT's list (operator ruling 15:50; `09` §3
|
||||||
|
decision 284).** No signature: a reboot destroys nothing (`03`). The agent's list is now `restart_controller` and
|
||||||
|
`reboot_guest`; the controller's list does not change and the two never share a verb (`TestHostOperatorActions_ClosedList`,
|
||||||
|
agent `TestOpActions_AgentClosedList`). The act is `pct reboot` of that guest over the existing exact sudo vector — never
|
||||||
|
the host. The agent refuses (reason in the result, nothing called) a guest it did not provision, a stopped, locked or
|
||||||
|
parked guest, a held heavy-operation gate, any agent job (`jobBlocker`: backup, controller swap, drive/NAS job, escrow
|
||||||
|
ceremony, vzdump), a fresh controller busy file (backup, restore, off-site run, night catch-up, app update, app
|
||||||
|
export/import — read over ONE new read-only grant, `pct exec <vmid> -- cat <the controller-data volume>/data/ctl/busy`;
|
||||||
|
any read failure but „no file" refuses), a second reboot within an hour (persisted), and the restart brake (a reboot
|
||||||
|
counts as a restart). It holds the heavy-operation gate until the guest is up. The hub expires an unpicked reboot after
|
||||||
|
30 minutes (a restart keeps the day). The closed list's rule stands: nothing on either list deletes data, starts a
|
||||||
|
countdown or shortens one.
|
||||||
|
|
||||||
### 6.2 Who may press the operator's buttons: session, Basic, or a scoped token [hub v0.152.0, `09` §3 decision 251]
|
### 6.2 Who may press the operator's buttons: session, Basic, or a scoped token [hub v0.152.0, `09` §3 decision 251]
|
||||||
|
|
||||||
The hub's operator surface has three ways in: a browser session, Basic auth with the operator password (scripts,
|
The hub's operator surface has three ways in: a browser session, Basic auth with the operator password (scripts,
|
||||||
|
|||||||
@@ -22,8 +22,9 @@
|
|||||||
> The hub is **not** greenfield — it's a mature service (felhom-hub **v0.11.0** as of 2026-06-13, Go +
|
> The hub is **not** greenfield — it's a mature service (felhom-hub **v0.11.0** as of 2026-06-13, Go +
|
||||||
> SQLite on k3s, `hub.felhom.eu`). This doc is the **deltas** to evolve it for the Proxmox model, plus
|
> SQLite on k3s, `hub.felhom.eu`). This doc is the **deltas** to evolve it for the Proxmox model, plus
|
||||||
> the new data model. Builds on Part 1 (trust/enrollment), Part 3 (the agent + reconcile), Part 4 (signing).
|
> the new data model. Builds on Part 1 (trust/enrollment), Part 3 (the agent + reconcile), Part 4 (signing).
|
||||||
> NOTE: this remains a design-draft for the deltas; the live hub is v0.11.0 (the prior "v0.6.3" reference
|
> NOTE: this remains a design-draft for the deltas. It was written against hub v0.11.0 (2026-06-13); the deployed
|
||||||
> was stale). Ground specifics against `felhom.eu/hub/` before relying on them.
|
> version is read from `manifests/hub.yaml`, never from this line (corrected 2026-10-24, review round 2: the old
|
||||||
|
> "the live hub is v0.11.0" was a current-state version literal). Ground specifics against `felhom.eu/hub/` before relying on them.
|
||||||
|
|
||||||
## 1. Source-of-truth model — two drivers, two directions
|
## 1. Source-of-truth model — two drivers, two directions
|
||||||
|
|
||||||
@@ -91,7 +92,7 @@ Evolves the existing staleness checker (60s **cadence**, a **configured** thresh
|
|||||||
|
|
||||||
- **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's
|
- **Primary = host-report recency → `host_stale` / `host_down`.** The agent heartbeat is the box's
|
||||||
liveness signal; a silent agent = the box is gone (the critical alert). **A hub restart does not re-mail a known
|
liveness signal; a silent agent = the box is gone (the critical alert). **A hub restart does not re-mail a known
|
||||||
silence (R-1001, hub 0.162.0 — built 2026-10-19, not yet deployed):** a host whose newest `host_stale`/`host_down`
|
silence (R-1001, hub 0.162.0 — deployed since; corrected 2026-10-24):** a host whose newest `host_stale`/`host_down`
|
||||||
event is newer than its last report is seeded with that state at start, silently; `restore_test_stale` reads its edge
|
event is newer than its last report is seeded with that state at start, silently; `restore_test_stale` reads its edge
|
||||||
from the DB on the first sweep (a signal for the tier inside the read window and newer than its last proof). A failed
|
from the DB on the first sweep (a signal for the tier inside the read window and newer than its last proof). A failed
|
||||||
lookup lets the alarm through.
|
lookup lets the alarm through.
|
||||||
@@ -138,8 +139,10 @@ walk is green; the golden is baked weekly and vouched by the operator; a new box
|
|||||||
itself to the floor (R-1052: not yet proven end to end). Setting the global floor today needs the operator page with the
|
itself to the floor (R-1052: not yet proven end to end). Setting the global floor today needs the operator page with the
|
||||||
operator's password — there is no scoped route for CC (R-1051). **An agent can be vouched mid-week; the golden stays
|
operator's password — there is no scoped route for CC (R-1051). **An agent can be vouched mid-week; the golden stays
|
||||||
weekly** (hub v0.158.0): the R-120 golden guard judges only a CHANGED golden selection, so a Day-0 save that keeps the
|
weekly** (hub v0.158.0): the R-120 golden guard judges only a CHANGED golden selection, so a Day-0 save that keeps the
|
||||||
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. **The agent list
|
vouched golden (same version and sha) saves a new agent even while that golden is behind the fleet. **„The fleet" is the
|
||||||
offers real agent packages only (R-1011, hub 0.162.0, not yet deployed):** the sha lookup asks for exactly the
|
newest RELEASED controller any box reported** — a test build (`-rc.N`, `-dev…`) never counts (R-1289, hub v0.171.0,
|
||||||
|
unreleased: the semver compare could not read a suffix and kept a dev build no box ran as the fleet's newest). **The agent list
|
||||||
|
offers real agent packages only (R-1011, hub 0.162.0, deployed since; corrected 2026-10-24):** the sha lookup asks for exactly the
|
||||||
`felhom-agent` file (the golden: `golden.tar.zst`) and never falls back to another file, so a step bundle
|
`felhom-agent` file (the golden: `golden.tar.zst`) and never falls back to another file, so a step bundle
|
||||||
(`0.156.1-step1`, only the config bundle) is left out of the list, and a vouch of one is refused
|
(`0.156.1-step1`, only the config bundle) is left out of the list, and a vouch of one is refused
|
||||||
(`artifact_file_missing`).
|
(`artifact_file_missing`).
|
||||||
@@ -295,6 +298,12 @@ What remains:
|
|||||||
PBS repo and namespace in the **DR recipe** (`dr_recipe`, reported every cycle — R-106 fixed it to match the
|
PBS repo and namespace in the **DR recipe** (`dr_recipe`, reported every cycle — R-106 fixed it to match the
|
||||||
backup); the endpoint's PBS fingerprint in **tenantsync** (`hub/internal/tenantsync`); the wrapped key in
|
backup); the endpoint's PBS fingerprint in **tenantsync** (`hub/internal/tenantsync`); the wrapped key in
|
||||||
**`host_escrow.blob`** (and the identity blob beside it).
|
**`host_escrow.blob`** (and the identity blob beside it).
|
||||||
|
**[DESIGN] 2026-10-24 (R-1254; committed felhom.eu `dcf4f84c`, hub next v0.170.0 — NOT deployed):** the escrow PUT
|
||||||
|
stores the blob, its sealed-password hash and the identity blob in ONE transaction (`Store.SaveHostEscrowWithIdentity`;
|
||||||
|
the superseded copy is made inside it). *[DESIGN] R-1218 (2026-10-26 night, hub main `09b5f02f`, next hub release): a package that seals the off-site repository password is never replaced by one that seals none — the PUT answers 409 and the good package stays current (it used to be demoted to operator-only custody, so the household's code no longer opened its off-site copy).* A failure leaves all of the old package or all of the new — never the new hash
|
||||||
|
beside the old identity package, which a box would read as „escrowed" while the written code opened nothing. A
|
||||||
|
malformed identity blob is refused before anything is stored. (The agent ceremony's „the previous code still works"
|
||||||
|
after a lost answer, R-1254's other half, stays open.)
|
||||||
|
|
||||||
Both existing infra-backup tables retire — `infra_backup_versions` (the current/live one, all readers
|
Both existing infra-backup tables retire — `infra_backup_versions` (the current/live one, all readers
|
||||||
hit it) **and** `infra_backups` (the deprecated legacy mirror). (The planned slim DR record on `hosts` was never
|
hit it) **and** `infra_backups` (the deprecated legacy mirror). (The planned slim DR record on `hosts` was never
|
||||||
@@ -326,8 +335,9 @@ customer instead of a single controller.
|
|||||||
- **DROP `reports` + CREATE `guest_reports`** (under the clean cutover this is drop+create with no data
|
- **DROP `reports` + CREATE `guest_reports`** (under the clean cutover this is drop+create with no data
|
||||||
migration, not an in-place rename); `guest_reports` adds `guest_id`, `host_id`; `cpu/memory` mean
|
migration, not an in-place rename); `guest_reports` adds `guest_id`, `host_id`; `cpu/memory` mean
|
||||||
guest-level; `backup_last_snapshot` goes quiet.
|
guest-level; `backup_last_snapshot` goes quiet.
|
||||||
- **ADD** desired-state JSON + `desired_generation` to `hosts`; `desired_spec_json` to `guests`; the
|
- **ADD** desired-state JSON + `desired_generation` to `hosts`; `desired_spec_json` to `guests`. ~~The
|
||||||
slim DR record (PBS namespace + repo fingerprint + wrapped escrow key) onto `hosts`.
|
slim DR record (PBS namespace + repo fingerprint + wrapped escrow key) onto `hosts`~~ — retired, never built (§9
|
||||||
|
above, R-105) *(corrected 2026-10-24, review round 2)*.
|
||||||
- **DROP both** `infra_backup_versions` (current/live) **and** `infra_backups` (legacy mirror) — the DR
|
- **DROP both** `infra_backup_versions` (current/live) **and** `infra_backups` (legacy mirror) — the DR
|
||||||
recipe, tenantsync and the escrow blob replace them (R-105).
|
recipe, tenantsync and the escrow blob replace them (R-105).
|
||||||
- **KEEP** `customer_configs`, `events`, `customer_notifications`, `notification_log`,
|
- **KEEP** `customer_configs`, `events`, `customer_notifications`, `notification_log`,
|
||||||
@@ -350,7 +360,9 @@ customer instead of a single controller.
|
|||||||
|
|
||||||
The box's console tells the volunteer to open „az e-mailben kapott link". That mail must exist whenever
|
The box's console tells the volunteer to open „az e-mailben kapott link". That mail must exist whenever
|
||||||
a customer is **waiting for a box**, without an operator press. `autoMintSelfBindLink` runs on four
|
a customer is **waiting for a box**, without an operator press. `autoMintSelfBindLink` runs on four
|
||||||
events: **customer creation**; **RESET completion**; **an e-mail set or changed on a customer with no
|
events (a fifth path mails a link too: the household's „new link" on a USED or EXPIRED link — one mail per customer
|
||||||
|
per hour, R-719; when the customer still has a linked host it shows the „oldbox" page in the customer's language and
|
||||||
|
tells the operator instead, R-1203 — `internal/web/selfbind.go` `handleBindResend`) *(corrected 2026-10-24, review round 2)*: **customer creation**; **RESET completion**; **an e-mail set or changed on a customer with no
|
||||||
bound host** (config save); **a host delete** that keeps the customer. The last two re-check that no
|
bound host** (config save); **a host delete** that keeps the customer. The last two re-check that no
|
||||||
host is bound, so a customer who has a box is never sent a pairing link. **Appliance registration is
|
host is bound, so a customer who has a box is never sent a pairing link. **Appliance registration is
|
||||||
not a trigger** — it knows no customer (`api/appliance.go`). Every send is stored as a hub-internal
|
not a trigger** — it knows no customer (`api/appliance.go`). Every send is stored as a hub-internal
|
||||||
@@ -379,6 +391,17 @@ the explicit Re-issue action") and the re-issue itself then refused with 400 —
|
|||||||
tier. **Not built:** releasing ONLY the token on host delete. The endpoint's only removal op
|
tier. **Not built:** releasing ONLY the token on host delete. The endpoint's only removal op
|
||||||
(`deprovision`) destroys the backups too, so a token-only release needs a new endpoint operation.
|
(`deprovision`) destroys the backups too, so a token-only release needs a new endpoint operation.
|
||||||
|
|
||||||
|
**[DESIGN — R-1295, hub v0.172.0 (rc.1 deployed for the proof) + tenantsync v1.4.0 (committed, not on ep0)] A new
|
||||||
|
token's self-check waits out PBS's permission cache, and a failure is never silent.** PBS's proxy caches permissions for
|
||||||
|
up to 5 s and an ACL-only change does not clear that cache; a request from another client between the token create and
|
||||||
|
its grant leaves the token refused (403) for up to ~5 s, and the script's one quick retry fell inside the window
|
||||||
|
(Tester4, 2026-10-26: rolled back, the box got no whole-guest off-site tier, only a log line). Measured on a throwaway PBS
|
||||||
|
4.2.5 under load: v1.3.0 failed 11 of 30, v1.4.0 0 of 30 (`audits/r1295-pbsdr-selfcheck-2026-10-26/`). The endpoint now
|
||||||
|
re-checks the same token (and its own admin token) for up to 15 s; the hub repeats a `self_check_failed` rollback (3
|
||||||
|
attempts, 7 s apart, a log line each) in the provision and the F-14 re-issue paths; a final failure of the WG hook raises
|
||||||
|
the operator-only `pbsdr_provision_failed` („… save the customer config to retry"). Live 2026-10-26 on hub rc.1: a fresh
|
||||||
|
Tester4 install provisioned the tier on the first try.
|
||||||
|
|
||||||
## 15. Customer e-mails — what the hub writes to a household [hub v0.118.0, R-558]
|
## 15. Customer e-mails — what the hub writes to a household [hub v0.118.0, R-558]
|
||||||
|
|
||||||
**The section this document did not have.** The hub composes every sentence a household reads before
|
**The section this document did not have.** The hub composes every sentence a household reads before
|
||||||
@@ -388,7 +411,7 @@ it has seen any box screen, and until v0.118.0 nothing here described that.
|
|||||||
|
|
||||||
| Mail | Trigger | Rendered by | Language source |
|
| Mail | Trigger | Rendered by | Language source |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| Event notification (39 event types) | a box event, or a hub checker | `FormatCustomerEmail` | reported → created-with → `hu` |
|
| Event notification (44 event types — the `mail.event.*` keys of `internal/i18n/locales/hu.json`, counted 2026-10-24) | a box event, or a hub checker | `FormatCustomerEmail` | reported → created-with → `hu` |
|
||||||
| Claim / reset / re-enroll / claimed | the claim arc | `FormatClaimEmail` | created-with (no box has reported yet) |
|
| Claim / reset / re-enroll / claimed | the claim arc | `FormatClaimEmail` | created-with (no box has reported yet) |
|
||||||
| Self-bind link | customer creation, or the operator's button | `FormatSelfBindEmail` | created-with |
|
| Self-bind link | customer creation, or the operator's button | `FormatSelfBindEmail` | created-with |
|
||||||
| The public bind PAGE at `/bind/<token>` | the customer opens the link | one template per language | created-with, **except `expired`** — see 15.4 |
|
| The public bind PAGE at `/bind/<token>` | the customer opens the link | one template per language | created-with, **except `expired`** — see 15.4 |
|
||||||
@@ -456,7 +479,7 @@ The operator mail renders the counts as one `Repair:` line. The household's side
|
|||||||
date taken) are shown on its own app page, on its box, until it dismisses them — file names never leave the box.
|
date taken) are shown on its own app page, on its box, until it dismisses them — file names never leave the box.
|
||||||
|
|
||||||
|
|
||||||
### 15.4b A box whose tunnel to ep0 is dead — the clock-ahead alarm [hub 0.166.0, built, NOT deployed; R-1204]
|
### 15.4b A box whose tunnel to ep0 is dead — the clock-ahead alarm [hub 0.166.0, deployed with 0.167.0; R-1204 — corrected 2026-10-24]
|
||||||
|
|
||||||
**[FACT, hub `e267c5e2`]** The host report carries the agent's `wireguard` block (`hostWireguard`). `checkWGTunnel` (`api/host_tunnel.go`, called after `checkHostSwap`) calls the tunnel stale when the box reports `handshake_future_s` > 0 (the clock-ahead lock-out) or a last handshake older than 6 h (healthy boxes measured at most 113–124 s over 24 h). It is edge-triggered against the host's latest tunnel event (`store.LatestHostTunnelEvent`): `wg_tunnel_stale` once on the way down, `wg_tunnel_ok` once on the way back; both are allow-listed and operator-only (`notify.operatorOnlyEvents`, `wg_tunnel_operator_only_test.go`) — the household cannot repair a tunnel. A box with no `wireguard` block (an agent before the field, or no tunnel) raises nothing.
|
**[FACT, hub `e267c5e2`]** The host report carries the agent's `wireguard` block (`hostWireguard`). `checkWGTunnel` (`api/host_tunnel.go`, called after `checkHostSwap`) calls the tunnel stale when the box reports `handshake_future_s` > 0 (the clock-ahead lock-out) or a last handshake older than 6 h (healthy boxes measured at most 113–124 s over 24 h). It is edge-triggered against the host's latest tunnel event (`store.LatestHostTunnelEvent`): `wg_tunnel_stale` once on the way down, `wg_tunnel_ok` once on the way back; both are allow-listed and operator-only (`notify.operatorOnlyEvents`, `wg_tunnel_operator_only_test.go`) — the household cannot repair a tunnel. A box with no `wireguard` block (an agent before the field, or no tunnel) raises nothing.
|
||||||
|
|
||||||
@@ -555,14 +578,38 @@ record unreadable: its serve paths answer 500, a save refuses it (never blanks i
|
|||||||
to a hub before v0.138.0** needs the columns opened first: `felhom-hub -unseal-box-secrets` (all or nothing; exit 1 =
|
to a hub before v0.138.0** needs the columns opened first: `felhom-hub -unseal-box-secrets` (all or nothing; exit 1 =
|
||||||
nothing changed) — run in the running pod, then put the old image back. `guests.api_key` is an unused, always-empty
|
nothing changed) — run in the running pod, then put the old image back. `guests.api_key` is an unused, always-empty
|
||||||
column and is not sealed. *Decided by CC unattended — operator may reverse (`09` §3 decision 132).*
|
column and is not sealed. *Decided by CC unattended — operator may reverse (`09` §3 decision 132).*
|
||||||
|
**Since v0.168.0 (R-975, R-1241) the last plain tokens are sealed too:** the Cloudflare tunnel token, the Cloudflare API
|
||||||
|
token and the git token live INSIDE `customer_configs.config_json` (`infrastructure.cf_tunnel_token`,
|
||||||
|
`infrastructure.cf_api_token`, `git.token`). Each of those JSON leaves holds the same `enc:v1:` seal; the rest of the JSON
|
||||||
|
stays readable. `SaveCustomerConfig` seals, every reader opens through `openCustomerSecrets` (a leaf that does not open is
|
||||||
|
blanked and marks the record unreadable, as above). Legacy rows are sealed at start-up (`SealLegacyConfigTokens`, without
|
||||||
|
a `config_version` bump — the box's config is byte-for-byte the same, so no box re-pulls or restarts), followed ONCE by
|
||||||
|
`VACUUM` + a WAL checkpoint, because an `UPDATE` leaves the old plaintext row in a freed block of the file
|
||||||
|
(`TestR975_DBFileWithoutKeyRevealsNoToken`). `-unseal-box-secrets` opens these leaves too, so the roll-back above also
|
||||||
|
covers a return to a hub before v0.168.0. The full list of secret-bearing columns, measured on a copy of the live database
|
||||||
|
2026-10-23: `audits/hub-token-seal-2026-10-23/README.md`.
|
||||||
|
**Deployed 2026-10-24 — and what the live file still holds (operator ruling 2026-10-24, accepted).** On the live hub the
|
||||||
|
start-up `VACUUM` did not finish: the Longhorn disk writes ~1.3 MB/s (the 6-hourly `VACUUM INTO` of 530 MB takes ~7 min)
|
||||||
|
and the liveness probe stopped the first start after ~90 s; the second start found nothing left to seal (R-1249). So the
|
||||||
|
rows are sealed, but the live `hub.db` and its `-wal` still carry the old token bytes in freed space (8 of 8, measured).
|
||||||
|
**Every copy that leaves the volume is clean**: the hub's snapshot and the ep0 copy are written by `VACUUM INTO` (the ep0
|
||||||
|
copy of 06:30Z: 0 of 8), and the volume is out of DooPlex's Longhorn backup (D40). Accepted, not cleaned: whoever can read
|
||||||
|
the live volume is root on DooPlex and can read `OFFSITE_SECRET_KEY` there too; a full wipe would also need a hub stop of
|
||||||
|
8–20 minutes, a disk trim and the volume's Longhorn snapshots deleted. **Rule: a hand copy of `hub.db` for analysis is
|
||||||
|
made with `VACUUM INTO` (or taken from ep0), never `cp`/`tar` of the live file** — a `tar` of the file carries the old
|
||||||
|
bytes (`audits/line-test-2026-10-24/hub-0168-seal-check.txt`).
|
||||||
**And the key is the other half:** a backup of the database restores a hub that can open the sealed columns only
|
**And the key is the other half:** a backup of the database restores a hub that can open the sealed columns only
|
||||||
with the same `OFFSITE_SECRET_KEY`; today that key exists only on DooPlex (the k8s Secret, and the GPG secrets export
|
with the same `OFFSITE_SECRET_KEY`; today that key exists only on DooPlex (the k8s Secret, and the GPG secrets export
|
||||||
on the same machine). The off-site plan for the database and its key: `runbooks/RUNBOOK-hub-db-offsite-backup.md`
|
on the same machine). The off-site plan for the database and its key: `runbooks/RUNBOOK-hub-db-offsite-backup.md`
|
||||||
(R-173 — option A decided 2026-10-05, `09` decision 125; §16.3).
|
(R-173 — option A decided 2026-10-05, `09` decision 125; §16.3).
|
||||||
|
|
||||||
### 16.3 The nightly database snapshot (R-173, hub v0.136.0)
|
### 16.3 The database snapshot — every 6 hours since D34 (R-173, hub v0.136.0; D34 in hub 0.167.0)
|
||||||
|
|
||||||
Every night at **02:00 Budapest** the hub writes `<data>/snapshots/hub-<UTC stamp>.db` with SQLite's `VACUUM INTO`:
|
**Corrected 2026-10-24 (review round 2):** since D34 (operator yes 2026-10-23) the hub writes a snapshot at **02:00,
|
||||||
|
08:00, 14:00 and 20:00 Budapest** (`internal/dbsnap/dbsnap.go` `Slots`), catches up at start when the newest is older
|
||||||
|
than **6 h** (`CatchUpAge`), and DooPlex pushes the newest at **:30 after each slot** (`felhom-hub-db-backup.timer`),
|
||||||
|
refusing one older than **7 h** (`felhom-hub-db-backup` `MAX_AGE_H`). The paragraph below is the original nightly
|
||||||
|
design; its mechanics hold per slot. Originally: every night at **02:00 Budapest** the hub writes `<data>/snapshots/hub-<UTC stamp>.db` with SQLite's `VACUUM INTO`:
|
||||||
one statement, one point in time, every committed write included — the rows still in `hub.db-wal` too, which a file copy
|
one statement, one point in time, every committed write included — the rows still in `hub.db-wal` too, which a file copy
|
||||||
of `hub.db` loses (measured in `internal/dbsnap` tests: a plain copy held 0 of 150 fresh rows). Written as `.tmp`, mode
|
of `hub.db` loses (measured in `internal/dbsnap` tests: a plain copy held 0 of 150 fresh rows). Written as `.tmp`, mode
|
||||||
`0600`, then renamed, so a reader never sees half a file. The newest **2** stay (the volume grew from 1 GiB to 2 GiB
|
`0600`, then renamed, so a reader never sees half a file. The newest **2** stay (the volume grew from 1 GiB to 2 GiB
|
||||||
@@ -591,7 +638,7 @@ failure never fails or delays a request: lines go through a non-blocking buffer
|
|||||||
|
|
||||||
### 16.5 Daily jobs and the clock change [DESIGN — hub v0.163.0, operator ruling D24 2026-10-21, R-1183]
|
### 16.5 Daily jobs and the clock change [DESIGN — hub v0.163.0, operator ruling D24 2026-10-21, R-1183]
|
||||||
|
|
||||||
Every daily hub job (`scheduleDaily`: db-snapshot 02:00, prune, deadline-check 05:00, offsite-key-audit 07:10) runs at
|
Every daily hub job (`scheduleDaily`: the four db-snapshot slots 02:00/08:00/14:00/20:00, prune at its configured time, deadline-check 05:00, offsite-key-audit 07:10 — slots corrected 2026-10-24) runs at
|
||||||
its Budapest wall-clock time **once per calendar day**: the next run is the calendar day after (`time.Date`, day+1),
|
its Budapest wall-clock time **once per calendar day**: the next run is the calendar day after (`time.Date`, day+1),
|
||||||
never „now + 24 h", and the slot that ran last never runs again. On a 25-hour day (the last Sunday of October) the old
|
never „now + 24 h", and the slot that ran last never runs again. On a 25-hour day (the last Sunday of October) the old
|
||||||
arithmetic ran every job after 02:00 twice; on a 23-hour day it ran them an hour late. Pinned by `TestR1183_*` for
|
arithmetic ran every job after 02:00 twice; on a 23-hour day it ran them an hour late. Pinned by `TestR1183_*` for
|
||||||
@@ -712,7 +759,7 @@ method (405), so a token gets 403.
|
|||||||
| `/configs/{id}/edit` | `customers:read` | **session only** | Edit can change the household's e-mail — and a new e-mail auto-mails the self-bind link (ruling 13:20). |
|
| `/configs/{id}/edit` | `customers:read` | **session only** | Edit can change the household's e-mail — and a new e-mail auto-mails the self-bind link (ruling 13:20). |
|
||||||
| `/offsite/remove-unpinned/{id}` | — | **session only** | |
|
| `/offsite/remove-unpinned/{id}` | — | **session only** | |
|
||||||
| `/offsite/abandon-cancel/{id}` | — | **session only** | Cancelling a pending set-aside deletion: no group names it; until the operator places it, session only. |
|
| `/offsite/abandon-cancel/{id}` | — | **session only** | Cancelling a pending set-aside deletion: no group names it; until the operator places it, session only. |
|
||||||
| `/offsite/window-grant/{id}` | — | `offsite:write` | |
|
| `/offsite/window-grant/{id}` | — | `offsite:write` | The one-shot clean-up window: the script address AND the customer page's „Backup & DR" card (R-1299, hub v0.174.0): `from=page` + `confirm=1` from the form; `cancel=1` removes a waiting grant; a limit above the store's snapshot count is refused. |
|
||||||
| `/offsite/windows-enabled` | — | `offsite:write` | |
|
| `/offsite/windows-enabled` | — | `offsite:write` | |
|
||||||
| `/os/{id}` | `os:read` | `os:write` | |
|
| `/os/{id}` | `os:read` | `os:write` | |
|
||||||
| `/offsite/key-audit` | — | `offsite:read` | Runs the daily key check now: it reads keys and reports, it changes none. |
|
| `/offsite/key-audit` | — | `offsite:read` | Runs the daily key check now: it reads keys and reports, it changes none. |
|
||||||
@@ -769,3 +816,39 @@ the DB override. Now `main` always mounts `webServer.Handler()` (`TestWebMountAl
|
|||||||
answers **503** on every operator route; `/bind/` stays public, `/login` renders but cannot succeed
|
answers **503** on every operator route; `/bind/` stays public, `/login` renders but cannot succeed
|
||||||
(`TestRequireAuth_NoPasswordRefusesEveryOperatorRoute`). **The live hub was never open:** 2026-10-15, hub 0.150.0, an
|
(`TestRequireAuth_NoPasswordRefusesEveryOperatorRoute`). **The live hub was never open:** 2026-10-15, hub 0.150.0, an
|
||||||
unauthenticated `GET /`, `/configuration`, `/hosts` each answered 302 → `/login`, and the ConfigMap carries a hash.
|
unauthenticated `GET /`, `/configuration`, `/hosts` each answered 302 → `/login`, and the ConfigMap carries a hash.
|
||||||
|
|
||||||
|
|
||||||
|
**`[DESIGN]` 2026-10-29 — two buttons on the host page (hub v0.176.0/0.176.1; `04` §6.1, D64 = B).** „Restart controller"
|
||||||
|
sits in the guest table, one per guest of a customer host (0.176.0 gated it on a reported controller version, which the
|
||||||
|
agent does not report — found live, fixed in 0.176.1); it stores an AGENT row (`operator_actions.host_id`), pokes the box
|
||||||
|
and shows the result under Operator Actions. „Clear hold" sits on each Held apps row; its confirm step names the app,
|
||||||
|
the reason and since when, and for a restore or put-back hold warns that the app may start on half-restored data; the hub
|
||||||
|
refuses an app the box does not report held now. Both are `hosts:write`, CSRF like every form (a token press needs none
|
||||||
|
and is recorded as `access token "<name>"`), and neither is reachable without the confirm fold-out in the browser.
|
||||||
|
|
||||||
|
**`[DESIGN]` 2026-10-29 — „Reboot guest" beside it (hub v0.177.0; D65 = A, `04` §6.1).** One per guest of a customer
|
||||||
|
host, behind its own confirm step (the whole guest reboots, every app stops for about a minute, when it is refused); it
|
||||||
|
stores an AGENT row `reboot_guest <vmid>` and pokes the box like the restart. **A reboot row the agent did not pick up in
|
||||||
|
30 minutes expires** (`HostOperatorActionRebootTTL`; a restart keeps the 24 h) — a box that was off must not reboot by
|
||||||
|
itself when it comes back. The agent's result closes the row with one `operator_action` event. **R-1346:** the access-log
|
||||||
|
middleware also writes `[WARN] slow request: <method> <path> <status> <ms>` for a request over 5 s (path only;
|
||||||
|
`/api/v1/wait` exempt), and the artifact vouch save logs each leg's time.
|
||||||
|
|
||||||
|
**[DESIGN] 2026-10-30 — the hub's database speed (R-1346, hub 0.178.0).** Measured on a copy of the live database
|
||||||
|
inside the hub pod (`audits/release-2026-10-30/p2-r1346/`): the pod had 256 MiB for a 570 MB database, so nearly every
|
||||||
|
page read went to Longhorn, and five statements read whole tables. `controller_version` and `agent_version` sit after
|
||||||
|
`report_json` in their rows, so reading them walks each row's JSON. Four indexes (`idx_reports_controller_version`,
|
||||||
|
`idx_host_reports_agent_version`, `idx_reports_received`, `idx_host_reports_received`) and one hint (`INDEXED BY
|
||||||
|
idx_app_telemetry_prune` on the apps page) cut the pages read 35 623 → 90 (the vouch's fleet check), 38 570 → 118 (the
|
||||||
|
OS alarm's agent versions), 30 097 → 1 013 (the 04:30 prune, inside its write lock) and 15 404 → 1 437 (the apps page);
|
||||||
|
the SQL lives in `store/r1346_speed.go` and each plan is pinned by `TestR1346_HeavyStatementsUseTheirIndex`. The
|
||||||
|
vouch's manifest is one transaction. Every statement over 1 s is logged as `[WARN] slow statement: <ms> (over 1s):
|
||||||
|
<SQL>` (`store/timeddriver.go`; the SQL only, never a value). The memory limit is 1 GiB (operator yes 2026-10-30). The
|
||||||
|
6-hourly snapshot (`VACUUM INTO`) takes no write lock — 9 600 writes ran during one on a copy, the slowest 1.5 ms; its
|
||||||
|
cost is 3 minutes of disk load. The vouch save answers in the request: its database step is now milliseconds, so no
|
||||||
|
background save is needed. **R-1344 restart:** the hub mails `controller_unresponsive_restart` (warning, operator-only)
|
||||||
|
when the stanza's `unresponsive_restart_at` moves, and not also `controller_restarted_by_agent` for that act; the
|
||||||
|
`controller_unresponsive` text says when the agent restarts (`unresponsive_restart_after_s`), or that an older agent
|
||||||
|
does not. **R-1352 (2026-11-02, hub commit only):** the alarm of an episode whose restart arrives in the same report is not
|
||||||
|
emitted: the restart's mail says all of it. An episode that began after that restart still alarms; a time that cannot be read
|
||||||
|
counts as a new episode (`episodeRestarted`, `TestR1352_*`).
|
||||||
@@ -19,7 +19,8 @@
|
|||||||
> decisions; grounded against felhom.eu @ `bf099f6` and felhom-agent @ `4ba1b14` (v0.63.0).
|
> decisions; grounded against felhom.eu @ `bf099f6` and felhom-agent @ `4ba1b14` (v0.63.0).
|
||||||
> Evidence base: `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` (all
|
> Evidence base: `documentation/audits/SPIKE-connectivity-wireguard-2026-07-03.md` (all
|
||||||
> measurements cited below as "spike Pn"). This document ends in an implementation slice roadmap
|
> measurements cited below as "spike Pn"). This document ends in an implementation slice roadmap
|
||||||
> (§8); **no slice is implemented yet** — each is a separate future task. Consistent with and
|
> (§8); **no slice is implemented yet** — each is a separate future task. *(Dated note 2026-10-24, review round 2: true when
|
||||||
|
> written; S1–S4 and S5's safe halves have since shipped — §8 and the agent's `internal/wgtunnel` say what is built.)* Consistent with and
|
||||||
> extending Part 01 (`01-topology-and-trust.md`); where this doc names a component's duty it uses
|
> extending Part 01 (`01-topology-and-trust.md`); where this doc names a component's duty it uses
|
||||||
> Part 01 §3 vocabulary.
|
> Part 01 §3 vocabulary.
|
||||||
|
|
||||||
@@ -140,7 +141,9 @@ before this ships (the drilled `--rescope-acl` posture is unaffected).
|
|||||||
4. **Hub → endpoint VM:** adds `{pubkey, /32}` to the WG server peer list (slice-1 surface).
|
4. **Hub → endpoint VM:** adds `{pubkey, /32}` to the WG server peer list (slice-1 surface).
|
||||||
5. **Box (agent):** next heartbeat sees the generation advance → fetches desired-state → writes
|
5. **Box (agent):** next heartbeat sees the generation advance → fetches desired-state → writes
|
||||||
the WG config → `install` + `systemctl enable --now wg-felhom` (the sudoers `*.mount` pattern
|
the WG config → `install` + `systemctl enable --now wg-felhom` (the sudoers `*.mount` pattern
|
||||||
of `felhom-agent.sudoers:16-18`, extended to this unit — slice-3).
|
of `felhom-agent.sudoers:16-18`, extended to this unit — slice-3). *(As built since R-861: no `install` sudoers
|
||||||
|
line; the conf is installed 0600 to its fixed path by the root wrapper `felhom-priv-apply wg` —
|
||||||
|
`felhom-agent/configs/felhom-agent.sudoers:223-229`; corrected 2026-10-24.)*
|
||||||
6. **Box:** tunnel handshakes (outbound, keepalive); PBS storage is (re)pointed at
|
6. **Box:** tunnel handshakes (outbound, keepalive); PBS storage is (re)pointed at
|
||||||
`pbs_tunnel_ip:8007` (§3.4); first backup runs.
|
`pbs_tunnel_ip:8007` (§3.4); first backup runs.
|
||||||
|
|
||||||
@@ -169,7 +172,8 @@ tenant ACL (`DatastoreBackup` on `/datastore/<ds>/<ns>` to BOTH the user and the
|
|||||||
### 3.5 Key custody + DR re-establishment
|
### 3.5 Key custody + DR re-establishment
|
||||||
|
|
||||||
The WG private key joins the **escrowed IdentityBundle**
|
The WG private key joins the **escrowed IdentityBundle**
|
||||||
(`internal/escrow/identity.go:24-27`, today `{TunnelToken, PBSToken}` → `+ WGPrivateKey`) — the
|
(`internal/escrow/identity.go`, then `{TunnelToken, PBSToken}` → `+ WGPrivateKey`; as built the bundle carries FOUR
|
||||||
|
fields — `TunnelToken`, `PBSToken`, `WGPrivateKey`, `ResticRepoPassword` — `identity.go:30-40`, corrected 2026-10-24) — the
|
||||||
same R-wrapped, zero-knowledge age blob the DR recipe already consumes. On host-loss
|
same R-wrapped, zero-knowledge age blob the DR recipe already consumes. On host-loss
|
||||||
re-provisioning (01 §9 restore mode): the box recovers the bundle with the customer's recovery
|
re-provisioning (01 §9 restore mode): the box recovers the bundle with the customer's recovery
|
||||||
code, restores the WG key, and re-establishes the tunnel; hub-side the peer entry still exists
|
code, restores the WG key, and re-establishes the tunnel; hub-side the peer entry still exists
|
||||||
@@ -199,7 +203,8 @@ disables the service, KEEPS its marker, and never re-registers on its own — re
|
|||||||
revoked until the operator re-adds the peer (the heartbeat report carries the box's pubkey as
|
revoked until the operator re-adds the peer (the heartbeat report carries the box's pubkey as
|
||||||
the recovery handle; **re-add via `POST /hosts/{id}/wg` with the global key** — the raw registry
|
the recovery handle; **re-add via `POST /hosts/{id}/wg` with the global key** — the raw registry
|
||||||
add does not bump the host's generation). The only automatic re-registration is the
|
add does not bump the host's generation). The only automatic re-registration is the
|
||||||
pubkey-mismatch (DR/re-key) path, backoff-bounded. Absent DATA (a failed fetch) is never a
|
pubkey-mismatch (DR/re-key) path, backoff-bounded — and, since D30 (agent v0.163.0), a key the agent makes itself after
|
||||||
|
a handshake stamped in the future for 15 min (`internal/wgtunnel/rekey.go`; corrected 2026-10-24). Absent DATA (a failed fetch) is never a
|
||||||
teardown signal. **Rollout gate:** `wg_tunnel.enabled` defaults FALSE agent-side until the
|
teardown signal. **Rollout gate:** `wg_tunnel.enabled` defaults FALSE agent-side until the
|
||||||
production endpoint exists.
|
production endpoint exists.
|
||||||
|
|
||||||
@@ -285,7 +290,9 @@ production endpoint exists.
|
|||||||
`wg-felhom` AllowedIPs so it survives the agent's self-heal ([OF-1]); PBS unaffected.
|
`wg-felhom` AllowedIPs so it survives the agent's self-heal ([OF-1]); PBS unaffected.
|
||||||
- **4.6 Tunnel health → hub.** The tunnel is a storage dependency, so it reports like one: no handshake
|
- **4.6 Tunnel health → hub.** The tunnel is a storage dependency, so it reports like one: no handshake
|
||||||
within ~3 keepalive periods → the offsite target reports unreachable → the existing alerting path
|
within ~3 keepalive periods → the offsite target reports unreachable → the existing alerting path
|
||||||
carries it. No new alarm channel.
|
carries it. No new alarm channel. *(Since hub 0.166.0 there is one: `wg_tunnel_stale` / `wg_tunnel_ok`,
|
||||||
|
operator-only — a handshake older than 6 h or stamped in the future, D30 / R-1204, `hub/internal/api/host_tunnel.go`;
|
||||||
|
corrected 2026-10-24.)*
|
||||||
**EXTENDED 2026-07-05 (TASK H1).** The agent's heartbeat now also carries an **`oob` stanza**
|
**EXTENDED 2026-07-05 (TASK H1).** The agent's heartbeat now also carries an **`oob` stanza**
|
||||||
(`felhom_sshd_active`, `felhom_sshd_port`, `reachable`, `config_invalid`, `operator_peer_configured`,
|
(`felhom_sshd_active`, `felhom_sshd_port`, `reachable`, `config_invalid`, `operator_peer_configured`,
|
||||||
`operator_key_configured`, `wg_handshake_age_s`) — the operator's "can I get into this box right
|
`operator_key_configured`, `wg_handshake_age_s`) — the operator's "can I get into this box right
|
||||||
@@ -297,6 +304,11 @@ production endpoint exists.
|
|||||||
|
|
||||||
## 5. The endpoint VM (the one new managed-infra piece)
|
## 5. The endpoint VM (the one new managed-infra piece)
|
||||||
|
|
||||||
|
**AMENDED 2026-10-24 (R-1247, operator ruling):** a fourth, small thing — `felhom-iperf3.service`, an iperf3 server bound to
|
||||||
|
`10.77.0.1:5201` and opened in nftables for `wg0` from `10.77.0.0/24` only, so each box's agent can measure its upload
|
||||||
|
line once a week (felhom.eu `scripts/ep0-iperf3/`; agent `internal/linetest`). It never answers on the internet (proven
|
||||||
|
2026-10-24, `audits/line-test-2026-10-24/`).
|
||||||
|
|
||||||
Runs three things, nothing else: the WG server (peer list = hub-written), PBS (one datastore,
|
Runs three things, nothing else: the WG server (peer list = hub-written), PBS (one datastore,
|
||||||
per-customer namespaces), and the minimal peer-sync surface the hub drives. **How** the hub
|
per-customer namespaces), and the minimal peer-sync surface the hub drives. **How** the hub
|
||||||
mutates the peer list — a tiny authenticated push API on the VM vs the VM pulling a signed peer
|
mutates the peer list — a tiny authenticated push API on the VM vs the VM pulling a signed peer
|
||||||
@@ -318,12 +330,16 @@ traverse `/root`) — create datastores under a world-traversable parent (`/srv`
|
|||||||
## 6. Trust-model conformance (01 §3/§4 preserved)
|
## 6. Trust-model conformance (01 §3/§4 preserved)
|
||||||
|
|
||||||
- The box **only ever dials out** (UDP to the endpoint; the same posture as hub-poll and
|
- The box **only ever dials out** (UDP to the endpoint; the same posture as hub-poll and
|
||||||
PBS-push, 01 §4/§7). No inbound listener is added to any customer box; the hub still never
|
PBS-push, 01 §4/§7). ~~No inbound listener is added to any customer box; the hub still never
|
||||||
initiates into a box.
|
initiates into a box.~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** Two listeners sit on the box's tunnel side, reachable only through
|
||||||
|
the tunnel: the agent's poke listener (UDP 51822 on the box's WireGuard address — a poke carries no command, the box
|
||||||
|
still pulls) and, on an appliance install, the operator door `felhom-sshd` (only from the operator's `/32`, §4.5) —
|
||||||
|
the documented break-glass (`03` §5). The hub still never sends a command into a box.
|
||||||
- The endpoint VM cannot reach into boxes either: it terminates tunnels whose per-peer routes
|
- The endpoint VM cannot reach into boxes either: it terminates tunnels whose per-peer routes
|
||||||
(§4.5) admit only box→PBS traffic; it holds no box credentials.
|
(§4.5) admit only box→PBS traffic; it holds no box credentials.
|
||||||
- **Out of scope, explicitly:** (a) operator-inbound access to boxes — a *separate* future
|
- **Out of scope, explicitly:** (a) ~~operator-inbound access to boxes — a *separate* future
|
||||||
problem with its own trust decision; nothing here creates or presumes it; (b) game-server
|
problem with its own trust decision; nothing here creates or presumes it;~~ operator-inbound access to boxes was
|
||||||
|
added on 2026-07-05 (TASK H1) and rides this tunnel — §4.5 (corrected 2026-10-25, D45, R-1266); (b) game-server
|
||||||
ingress — public players are not tunnel peers; that is a different ingress problem (public
|
ingress — public players are not tunnel peers; that is a different ingress problem (public
|
||||||
reachability), not a backup-transport one.
|
reachability), not a backup-transport one.
|
||||||
|
|
||||||
@@ -350,7 +366,8 @@ venue; it gates nothing. Runbook:
|
|||||||
**Open sub-decisions (deferred by design):**
|
**Open sub-decisions (deferred by design):**
|
||||||
- Endpoint peer-sync mechanism (push-API vs signed-manifest pull) — **slice-1 design point** (§5).
|
- Endpoint peer-sync mechanism (push-API vs signed-manifest pull) — **slice-1 design point** (§5).
|
||||||
- WG key rotation cadence/mechanics — the reconcile channel can carry it (new pubkey → hub →
|
- WG key rotation cadence/mechanics — the reconcile channel can carry it (new pubkey → hub →
|
||||||
endpoint swap); design when fleet size warrants, possibly the moment the "separable fleet
|
endpoint swap); design when fleet size warrants *(one automatic rotation exists since D30: after a future-stamped
|
||||||
|
handshake for 15 min, at most once an hour — `rekey.go`; a scheduled rotation is still open; noted 2026-10-24)*, possibly the moment the "separable fleet
|
||||||
layer" (D1) conversation reopens.
|
layer" (D1) conversation reopens.
|
||||||
- Endpoint scaling/sharding (N customers per VM, second region, HU-hosted option) — capacity
|
- Endpoint scaling/sharding (N customers per VM, second region, HU-hosted option) — capacity
|
||||||
planning, not architecture; the peer registry already keys per-host.
|
planning, not architecture; the peer registry already keys per-host.
|
||||||
@@ -373,6 +390,8 @@ venue; it gates nothing. Runbook:
|
|||||||
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; ~~§4.3 MTU decision~~ **RESOLVED (client MTU 1280, agent v0.65.0, §4.3)**; residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
|
| **S6 — monitoring + isolation hardening** | endpoint-side alerting, per-peer firewall audit; ~~CGNAT smoke-test closure~~ **DONE 2026-07-04 (§7)** — mobile-carrier NAT traversal + 32-min keepalive hold proven; ~~§4.3 MTU decision~~ **RESOLVED (client MTU 1280, agent v0.65.0, §4.3)**; residual = true-CGNAT-SIM retest (low risk) | tunnel-down and peer-anomaly both alert; isolation verified adversarially |
|
||||||
|
|
||||||
**MVP that puts offsite backup in a real customer's hands = S1 → S2 → S3; then S4.** S5/S6
|
**MVP that puts offsite backup in a real customer's hands = S1 → S2 → S3; then S4.** S5/S6
|
||||||
complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — today a
|
complete the story. The cloudflared service (`internal/hub/cloudflared.go:9-12` — then a
|
||||||
read-only prober that names itself "the seam for the tunnel-management slice") adopts the
|
read-only prober that named itself "the seam for the tunnel-management slice") ~~adopts the
|
||||||
S3-piloted agent-managed-service pattern in a later, separate slice.
|
S3-piloted agent-managed-service pattern in a later, separate slice.~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** This plan is withdrawn:
|
||||||
|
cloudflared is a GUEST container that the controller renders and keeps up; the agent only reads it (four states,
|
||||||
|
`cloudflared.go:12-23`) and will not manage it — `03` §2/§3 wins.
|
||||||
@@ -253,7 +253,22 @@ It carries guest sizing, `pve_storage[]`, and an app half with per-app `storage_
|
|||||||
the identity bundle's shape is `{tunnel_token, pbs_token, wg_private_key, restic_repo_password}`
|
the identity bundle's shape is `{tunnel_token, pbs_token, wg_private_key, restic_repo_password}`
|
||||||
(`felhom-agent/internal/escrow/identity.go:26-39`).
|
(`felhom-agent/internal/escrow/identity.go:26-39`).
|
||||||
|
|
||||||
**[DESIGN] Off-site deletion custody (decisions 68–69, 2026-10-03) — BUILT hub v0.127.0 / controller v0.289.1, live on both demo boxes.** The restic repository password stays on the box only. The box's off-site key is append-only (pinned in `authorized_keys`, written by the hub — the key registrar); the box never receives the sub-account password, which the hub stores encrypted at rest. Old snapshots are pruned by the box itself, only in a weekly window the hub opens, behind a fake-snapshot guard (R-822); the window has run live on both demo boxes since 2026-10-05 (R-95, closed). The guard's residual — past-dated fakes steering the keeps, and a window check that trusts the box's own counts — is R-822 and R-895. **Hub v0.158.0 (built 2026-10-18, deploy waits on the operator) adds the hub's own check:** the 07:10 key audit compares the repository's `snapshots/` listing, over the hub's own login, with yesterday's — files gone that no clean-up window explains raise `offsite_snapshots_vanished` (error), files no reported box run explains raise `offsite_snapshots_unexplained` (warning); both operator-only (R-895 slice 2). A Felhom-side pruner holding repository passwords is rejected.
|
**[DESIGN] Off-site deletion custody (decisions 68–69, 2026-10-03) — BUILT hub v0.127.0 / controller v0.289.1, live on both demo boxes.** The restic repository password stays on the box only. The box's off-site key is append-only (pinned in `authorized_keys`, written by the hub — the key registrar); the box never receives the sub-account password, which the hub stores encrypted at rest. Old snapshots are pruned by the box itself, only in a weekly window the hub opens, behind a fake-snapshot guard (R-822); the window has run live on both demo boxes since 2026-10-05 (R-95, closed). The guard's residual — past-dated fakes steering the keeps, and a window check that trusts the box's own counts — is R-822 and R-895. **Hub v0.158.0 (built 2026-10-18; deployed the same day on the operator's yes — corrected 2026-10-24) adds the hub's own check:** the 07:10 key audit compares the repository's `snapshots/` listing, over the hub's own login, with yesterday's — files gone that no clean-up window explains raise `offsite_snapshots_vanished` (error), files no reported box run explains raise `offsite_snapshots_unexplained` (warning); both operator-only (R-895 slice 2). A Felhom-side pruner holding repository passwords is rejected.
|
||||||
|
|
||||||
|
**[FINDING — R-1294, 2026-10-26, read only; a rule change waits for the operator (STATUS D54)] A store that once goes
|
||||||
|
over the window's cap never comes back by itself.** The cap is half the store (hub `offsitekeys.MaxRemove`); each refused
|
||||||
|
window adds a week of snapshots and removes none, so the plan grows faster than the cap. demo-hp: one test day
|
||||||
|
(2026-10-16, 152 snapshots) aged past 7 days and the 10-26 window refused 209 > 179 — an HONEST plan (every removal 7–30
|
||||||
|
days old, every app keeps ≥ 9 or its last copies; `audits/r1294-offsite-cap-2026-10-26/`). The existing way out is the
|
||||||
|
operator's one-shot raised cap (R-833, `GrantOffsiteWindowOnceMax`).
|
||||||
|
**[DESIGN — R-1299, built hub v0.174.0/v0.174.1 2026-10-27, `d85198b8` + `de57c090`] The one-shot window has a button.**
|
||||||
|
The customer page's „Backup & DR" tab carries the „Off-site clean-up window" card: the weekly switch, the normal cap,
|
||||||
|
a waiting grant and the last window; an optional limit, a confirm step that names the customer and the limit, then the
|
||||||
|
same route and rule as before (`POST /offsite/window-grant/<customer>`, `offsite:write` or a session). A raised limit
|
||||||
|
above the store's measured snapshot count is refused (a window cannot remove more than exist; the 500 ceiling stays). A
|
||||||
|
waiting grant can be cancelled before the box uses it. Every grant and cancel is a stored operator-only info event
|
||||||
|
`offsite_window_grant` (who, customer, limit). Live: granted and cancelled on tester-1 by the operator, nothing removed
|
||||||
|
(`audits/r1299-window-button-2026-10-27/`).
|
||||||
|
|
||||||
**[DESIGN, built hub 2026-10-07 — R-32 option A, `09` §3 decision 167] What RESET does to the household's off-site
|
**[DESIGN, built hub 2026-10-07 — R-32 option A, `09` §3 decision 167] What RESET does to the household's off-site
|
||||||
copy.** On the shared pool box a sub-account is a LOGIN, not the data: deleting it leaves its folder (measured
|
copy.** On the shared pool box a sub-account is a LOGIN, not the data: deleting it leaves its folder (measured
|
||||||
@@ -380,7 +395,7 @@ never moved by an update or an undo, and the unit restore accepts it.
|
|||||||
|
|
||||||
| Tier | Location | Captures | Cadence (LIVE) | Retention (LIVE) | Encrypted |
|
| Tier | Location | Captures | Cadence (LIVE) | Retention (LIVE) | Encrypted |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
| **Tier-1** recovery unit | `<nsRoot>/backups/primary/<app>/` on the app's own drive | compose + `.felhom.yml` + **secret-stripped** `app.yaml`, `db-dumps/*.sql`, `volume-dumps/*.tar`, `manifest.json` | nightly at W, plus a checksum-gated refresh on the periodic status pass | one point per app (`restore_points.go:14-18`) | no |
|
| **Tier-1** recovery unit | `<nsRoot>/backups/primary/<app>/` on the app's own drive | compose + `.felhom.yml` + `app.yaml` (0600 — since D5 / schema 2 it carries the PORTABLE secrets, `recovery_unit.go:25`; „secret-stripped" corrected 2026-10-24), `db-dumps/*.sql`, `volume-dumps/*.tar`, `manifest.json` | nightly at W, plus a checksum-gated refresh on the periodic status pass | one point per app (`restore_points.go:14-18`) | no |
|
||||||
| **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no |
|
| **Tier-2** cross-drive | `<target nsRoot>/backups/secondary/<app>/` | **always** a full mirror of the unit (`tier2.go:368-369`) **plus** `mandatory + optional` file legs, v2 layout `hdd/<rel>` + `userdata/<rel>` | nightly at W+60m | mirror (rsync) | no |
|
||||||
| **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) |
|
| **Tier-3** restic offsite | Hetzner Storage Box over SFTP, one multi-path snapshot per app per run | the unit **plus** `mandatory` legs only; a separate `_shares` snapshot | nightly at W+105m | `--keep-daily 7 --keep-weekly 4 --keep-monthly 6` | **yes** (restic) |
|
||||||
| **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/felhom` (its `docker/` and `sys_drive/` folders, bound in the guest to `/var/lib/docker` and `/mnt/sys_drive` — §7.5.1; `mp1` no longer exists, corrected 2026-10-15) | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` → `--prune-backups keep-daily=3`: one copy per day, 3 days (keep-last until agent 0.160.0, R-1149) | **no** |
|
| **Plane-2** whole-guest, local | `local:` → `/var/lib/vz/dump` on the host | rootfs + `mp0 /var/lib/felhom` (its `docker/` and `sys_drive/` folders, bound in the guest to `/var/lib/docker` and `/mnt/sys_drive` — §7.5.1; `mp1` no longer exists, corrected 2026-10-15) | **24 h** (`backup_cadence_seconds: 0`) | `local_backup_retention: 3` → `--prune-backups keep-daily=3`: one copy per day, 3 days (keep-last until agent 0.160.0, R-1149) | **no** |
|
||||||
@@ -396,6 +411,8 @@ never moved by an update or an undo, and the unit restore accepts it.
|
|||||||
> copy protected the app's files. Since v0.245.0 every dashboard page carries the reminder (the R-241
|
> copy protected the app's files. Since v0.245.0 every dashboard page carries the reminder (the R-241
|
||||||
> bar, second instance) and the Tier-1 sentence renders by state — „védené … szünetel" while paused.
|
> bar, second instance) and the Tier-1 sentence renders by state — „védené … szünetel" while paused.
|
||||||
> Measured on a fresh box 2026-09-16: zero snapshots, and the page said the files were protected.
|
> Measured on a fresh box 2026-09-16: zero snapshots, and the page said the files were protected.
|
||||||
|
>
|
||||||
|
> *[DESIGN] R-1219 / R-1218 (2026-10-26 night, controller main `1170b48`, next release): during a recovery offer the bar does NOT urge the ceremony (a new code would replace the key that opens the old off-site copy) and the start answers 409; a ceremony stages the off-site password whenever its FILE is on the box, also with off-site switched off or cleared.*
|
||||||
|
|
||||||
> **R-191 (2026-08-04) — this row was RIGHT and the configuration disagreed with it, weekly, for as long as R-89 has been in force.** The contract has not changed: offsite retention is ep0's, the box's token is write-only, and the box cannot delete its own history. What had not followed was the installer's `keep_last: 2` on the offsite tier, so every weekly run uploaded its snapshot successfully and then failed the whole JOB on a prune the token is refused — `whole_guest_backup_failed` in the operator's inbox about a backup that had already succeeded. Fixed in installer **1.25.0** (`keep_last: 0`) and on both live boxes; a gate now asserts it. **Verified before changing it:** ep0's two prune jobs have run every day since 2026-07-27, 18 tasks, all OK. A doc that states the contract does not enforce it — the gate does.
|
> **R-191 (2026-08-04) — this row was RIGHT and the configuration disagreed with it, weekly, for as long as R-89 has been in force.** The contract has not changed: offsite retention is ep0's, the box's token is write-only, and the box cannot delete its own history. What had not followed was the installer's `keep_last: 2` on the offsite tier, so every weekly run uploaded its snapshot successfully and then failed the whole JOB on a prune the token is refused — `whole_guest_backup_failed` in the operator's inbox about a backup that had already succeeded. Fixed in installer **1.25.0** (`keep_last: 0`) and on both live boxes; a gate now asserts it. **Verified before changing it:** ep0's two prune jobs have run every day since 2026-07-27, 18 tasks, all OK. A doc that states the contract does not enforce it — the gate does.
|
||||||
|
|
||||||
@@ -437,7 +454,7 @@ container must keep its id). Ring 1 never takes it in the night leg. Measured: t
|
|||||||
|
|
||||||
**[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
|
**[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
|
||||||
offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
|
offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
|
||||||
(`cmd/controller/main.go:604-607`). Both boxes run W = `02:30`.
|
(`backupwindow.LegTimes`, called at `cmd/controller/main.go:1258-1264`; line corrected 2026-10-24). Both boxes run W = `02:30`.
|
||||||
|
|
||||||
**[FACT, controller v0.271.0, 2026-09-25] A fourth leg: automatic app updates, CHAINED after the off-site
|
**[FACT, controller v0.271.0, 2026-09-25] A fourth leg: automatic app updates, CHAINED after the off-site
|
||||||
leg** (`09` §3 decisions 11 and 20, §6.4 part 7). The `offbox-backup` job runs the update leg when its off-site
|
leg** (`09` §3 decisions 11 and 20, §6.4 part 7). The `offbox-backup` job runs the update leg when its off-site
|
||||||
@@ -448,8 +465,10 @@ night the whole-guest backup starts later inside its own window; it is never ski
|
|||||||
numbers: `audits/DRILL-night-2026-09-25.md` Part D.
|
numbers: `audits/DRILL-night-2026-09-25.md` Part D.
|
||||||
|
|
||||||
**[FACT, 2026-09-25] demo-hp's local whole-guest target is its host ROOT disk; retention there is now
|
**[FACT, 2026-09-25] demo-hp's local whole-guest target is its host ROOT disk; retention there is now
|
||||||
keep-last=1** (operator ruling 2026-09-24, option A; R-684). PVE prunes AFTER a successful backup, so the target
|
keep-last=1** (operator ruling 2026-09-24, option A; R-684) — *since R-1149 (operator D17, 2026-10-19) the local tier's
|
||||||
must hold keep-last + 1 archives during the run — measured: a 9201 archive is 8.18 GB, the root disk 39 GB. The
|
spec is `keep-daily=N` (`felhom-agent/internal/config/config.go:553-559`), so demo-hp is keep-daily=1; corrected
|
||||||
|
2026-10-24*. PVE prunes AFTER a successful backup, so the target
|
||||||
|
must hold N + 1 archives during the run — measured: a 9201 archive is 8.18 GB, the root disk 39 GB. The
|
||||||
warning that should precede such a failure is R-685.
|
warning that should precede such a failure is R-685.
|
||||||
|
|
||||||
**[FACT, 2026-10-14] A power cut during the whole-guest backup leaves a half-written archive, and from agent v0.156.1 the
|
**[FACT, 2026-10-14] A power cut during the whole-guest backup leaves a half-written archive, and from agent v0.156.1 the
|
||||||
@@ -731,7 +750,8 @@ definition is written or a named volume replaced — a failed placement, volume
|
|||||||
the live definition back and HOLDS the app stopped (`restore_mixed`, operator-cleared); the household reads that it
|
the live definition back and HOLDS the app stopped (`restore_mixed`, operator-cleared); the household reads that it
|
||||||
needs our help; the operator gets a `backup_run_failures` line. The hold is persisted before the stop. Placed files
|
needs our help; the operator gets a `backup_run_failures` line. The hold is persisted before the stop. Placed files
|
||||||
ALONE (no version change, no volume replaced) still restart after a successful rollback (`TestR379_ScenarioA`): that
|
ALONE (no version change, no volume replaced) still restart after a successful rollback (`TestR379_ScenarioA`): that
|
||||||
mix, and the undo of everything, is option A — „put back exactly as it was", the next slice on R-893. **`[FACT]`
|
mix, and the undo of everything, is option A — „put back exactly as it was", the next slice on R-893 *(shipped since:
|
||||||
|
D20 in controller v0.336.0 and D22 in v0.338.0, below — noted 2026-10-24)*. **`[FACT]`
|
||||||
2026-10-19 (measured on 9202, controller 0.335.0, `audits/night-2026-10-19/P3/`):** a forced replay failure after the
|
2026-10-19 (measured on 9202, controller 0.335.0, `audits/night-2026-10-19/P3/`):** a forced replay failure after the
|
||||||
volume was replaced left the NEWEST rows (the undo dump) on top of the backup's volume — a table only the backup had
|
volume was replaced left the NEWEST rows (the undo dump) on top of the backup's volume — a table only the backup had
|
||||||
stayed (37 tables, the live app had 36); the definition did not move (same version); the app was held and a start
|
stayed (37 tables, the live app had 36); the definition did not move (same version); the app was held and a start
|
||||||
@@ -747,13 +767,17 @@ the snapshot added is removed, the live definition is written back and the app i
|
|||||||
restore failed and nothing changed (with placed files: its database and volumes are back, the copied files stay); the
|
restore failed and nothing changed (with placed files: its database and volumes are back, the copied files stay); the
|
||||||
operator gets `backup_run_failures` leg `restore-put-back`. The hold (`restore_mixed`) remains only for a put-back that
|
operator gets `backup_run_failures` leg `restore-put-back`. The hold (`restore_mixed`) remains only for a put-back that
|
||||||
fails — then the copy is kept for support — and the double failure (replay AND rollback) holds as before. On success the
|
fails — then the copy is kept for support — and the double failure (replay AND rollback) holds as before. On success the
|
||||||
copy is removed. **`[FACT]` 2026-10-20 (9202, controller 0.336.0-rc.1, `audits/night2-2026-10-20/P2/live/`):** the same
|
copy is removed — **since controller 0.343.0 (fix A, 2026-10-25) only after a STABLE start, and the verdict and the release
|
||||||
|
are two different tests (§8.2e):** the verdict „started" needs the stack running on every sample of a 45 s window, no
|
||||||
|
container restart in it and every healthcheck `healthy`; the copy is released only after a further 3 minutes of the same
|
||||||
|
(10 s samples, watched up to 30 minutes; a new restore of the app ends the watch and leaves the copy to it). A restore that
|
||||||
|
passes the verdict and then flaps keeps its copy. **`[FACT]` 2026-10-20 (9202, controller 0.336.0-rc.1, `audits/night2-2026-10-20/P2/live/`):** the same
|
||||||
forced failure as on 2026-10-19 → 36 tables, the backup-only table absent, schema/data/row-count md5 equal to before,
|
forced failure as on 2026-10-19 → 36 tables, the backup-only table absent, schema/data/row-count md5 equal to before,
|
||||||
the app running, no hold, the copy (69 MB, 0.9 s) gone; a normal restore of the same snapshot then worked. The R-893
|
the app running, no hold, the copy (69 MB, 0.9 s) gone; a normal restore of the same snapshot then worked. The R-893
|
||||||
known limit above is closed for named volumes; placed FILES are still not taken back (the plain-files mix, as ruled).
|
known limit above is closed for named volumes; placed FILES are still not taken back (the plain-files mix, as ruled).
|
||||||
|
|
||||||
**`[DESIGN]` 2026-10-21 — the put-back covers EVERY restore, and a power cut (operator ruling D22 = yes, 2026-10-21
|
**`[DESIGN]` 2026-10-21 — the put-back covers EVERY restore, and a power cut (operator ruling D22 = yes, 2026-10-21
|
||||||
20:05; controller `main`, unreleased).** (1) **The local unit restore** — the app's own unit, the second drive's unit,
|
20:05; released in controller v0.338.0 — corrected 2026-10-24).** (1) **The local unit restore** — the app's own unit, the second drive's unit,
|
||||||
the second drive's „Teljes visszaállítás" and a kept load all run `RestoreFromRecoveryUnitAtWith` — now takes the same
|
the second drive's „Teljes visszaállítás" and a kept load all run `RestoreFromRecoveryUnitAtWith` — now takes the same
|
||||||
measure → copy → put-back (`offbox_restore_undo.go`, one implementation, no second copy of the logic): the copy must fit
|
measure → copy → put-back (`offbox_restore_undo.go`, one implementation, no second copy of the logic): the copy must fit
|
||||||
beside the unit before anything stops; ANY failure after the stop (volume replay, definition write, DB-only start, DB
|
beside the unit before anything stops; ANY failure after the stop (volume replay, definition write, DB-only start, DB
|
||||||
@@ -775,6 +799,86 @@ than the unit), the app running, the record reads the D22 sentence. Known limit
|
|||||||
named volume the unit replaces (none in the catalog today — a drive-file app's unit restore is refused by R-538) is not
|
named volume the unit replaces (none in the catalog today — a drive-file app's unit restore is refused by R-538) is not
|
||||||
covered by the volume copy.
|
covered by the volume copy.
|
||||||
|
|
||||||
|
**`[DESIGN]` 2026-10-24 — a pending put-back blocks EVERY start (R-1251; review round 2 HUNT-1 H1, H2, H7; committed in
|
||||||
|
controller `bb2809d` + `062f9b4`, unreleased — rides the next controller release).** For a drive-backed app the copy lives on
|
||||||
|
the app's drive, and at start the drive is often not bound yet. A copy that cannot be reached while its drive is not
|
||||||
|
bound is **waiting, never „gone"**: the crash recovery writes a restore hold (`putback_pending`, the copy's folder
|
||||||
|
in it) and starts nothing. The hold is the one store every start path already honours — the household's
|
||||||
|
Start/Restart/Update, the boot recreate, the drive return, the media restart, the boot sweep, the night chain, the
|
||||||
|
update leg — so no path has its own check. When the drive binds, the put-back runs FIRST (the boot recreate and the
|
||||||
|
drive-return restart; the start and every drive return sweep the waiting holds), the hold is lifted, then the app
|
||||||
|
starts. The household reads „…A visszaállítás előtti adatai egy meghajtón vannak, amely most nem érhető el…" on the
|
||||||
|
app and „Ne indítsd újra a visszaállítást…" in the restore record. A put-back that FAILS (at start or when the drive
|
||||||
|
returns), or a copy gone while its drive is bound, leaves the app **held** with the D20 hold („félúton megszakadt"),
|
||||||
|
the copy kept, and the marker lets go of the app — never stopped-and-unheld (H2). Stop pressed during the restore still
|
||||||
|
gets the put-back; only the start is skipped (H7). A removal clears a waiting put-back; a later restore that completes
|
||||||
|
drops it (the household chose that data). The copy's folders are synced to disk BEFORE the marker names the copy.
|
||||||
|
**`[DESIGN]` 2026-11-02 — a failed put-back at start-up reaches the operator; a restore that lifts an update hold forgets the
|
||||||
|
in-memory flag (R-1348, R-1347; controller `c4b6844`, commit only).** The hold from `Recover` is written before any notifier exists, so
|
||||||
|
the guard keeps the name and `SetHeldNotify` (main.go, where the notifier is wired) sends the R-1340 sentence once; a later failure, a
|
||||||
|
drive that returned, mails at once. `clearUpdateHoldAfterRestore` also calls `ForgetUpdateHeld` through `SetUpdateHeldForgetter`: the
|
||||||
|
hold store and the flag go together, as in the operator's clear (R-1343), so the crash-loop watcher and the night leg take the
|
||||||
|
restored app again. Live proof: `audits/security-2026-11-02/part3/`.
|
||||||
|
**`[FACT]` 2026-10-24 night, Tester 1 (VM 341; MeTube on the data drive, 1.5 GB in its volume;
|
||||||
|
`audits/restore-holes-2026-10-24/P1-tester1/`):** a local restore cut by powering the VM off 2.1 s after „the restore
|
||||||
|
copy is recorded", the data disk taken off, the box booted: held as waiting, Start and Update refused (409, the
|
||||||
|
sentence); the disk plugged back: the agent re-mounted it, the boot recreate put the copy back BEFORE the start, the
|
||||||
|
household's newest data („gen2") and its download on the drive were there, and a later controller restart poured
|
||||||
|
nothing. Run 1 of the same proof found the journal left as `undo.json.tmp` (the rename lived only in the page cache) —
|
||||||
|
the app was held for support, never started on the cut data; the folder sync above fixed it, run 2 passed.
|
||||||
|
|
||||||
|
**`[DESIGN]` 2026-10-28 night — four restore edges (R-1256, review round 2 HUNT-1 H4/H5/H6/H8; controller `342a394`,
|
||||||
|
unreleased — the morning list).** (H4) An off-site restore whose definition write fails after the compose was written
|
||||||
|
writes the LIVE definition back before it starts the app; a write-back that fails too HOLDS the app (`restore_mixed`) —
|
||||||
|
never a start on the snapshot's definition over the live data. (H5) A put-back whose stop fails pours nothing, keeps the
|
||||||
|
copy and holds the app (off-site, local unit, and a restore cut by a power loss); a failed definition write-back keeps the
|
||||||
|
copy (D20); a `docker volume rm -f` that fails (the volume is in use) fails that volume — the `volume create` after it was
|
||||||
|
a no-op and the tar was unpacked ON TOP of the old content. (H6) A restore op begins in ONE step with the check for every
|
||||||
|
other operation (a restore, the backup run, a second-drive copy — under the lock the mirror asks under); every restore
|
||||||
|
button and the kept loads begin that way, and the install-time „use my kept data" asks about the second-drive copy too.
|
||||||
|
(H8) An unused restore copy goes after 14 days (every 6 h): only when the app has no restore hold of any kind, the
|
||||||
|
app-stop marker is readable and does not name the app, and no restore runs. **Decided by CC unattended 2026-10-28 —
|
||||||
|
operator may reverse:** *How long does an unused restore copy stay?* (a) 14 days; (b) for ever (as before); (c) 3 days.
|
||||||
|
(a) holds up to two weeks of disk per restore and gives support two weeks; (b) grows without end; (c) is shorter than a
|
||||||
|
support reply. A held or waiting copy is never touched under any option. Tests `TestR1256_*`, each red first.
|
||||||
|
**`[DESIGN]` 2026-10-29 — D62 replaces the age, and the household can delete a copy (operator ruling 08:17; `09` §3
|
||||||
|
decision 281).** An unused copy stays **30 days**. The restore page's kept-copies list („Megőrzött másolatok") shows each
|
||||||
|
restore copy: the app, the date of the restore it protects, its size and „automatikusan törlődik: <date>". „Törlés most"
|
||||||
|
deletes one earlier after the warning („Ez a(z) <app> adatainak másolata a <dátum>-i visszaállítás előttről. Ha törlöd, a
|
||||||
|
visszaállítás többé nem vonható vissza, és ez a másolat nem hozható vissza.") and the app's name typed. Never deletable: a
|
||||||
|
copy a hold or a pending put-back needs (the same rule as the sweep; the button is off with the reason, and the handler
|
||||||
|
refuses it), any copy while a backup or restore runs or the app updates (409), and — R-1342, controller `652a690`,
|
||||||
|
unreleased — a copy the 30-minute stable-start watch still holds (it reads „checking"; an app removed during that watch
|
||||||
|
ends it quietly instead of alarming). The check and the removal are one step
|
||||||
|
under the backup manager's lock. One `[WARN]` log line and one operator-only event `restore_copy_deleted` (who, app,
|
||||||
|
restore date, size; hub `19afefab`) per delete. Controller `e3fce09`, tests `TestD62_*`. **D63** (`09` §3 decision 282):
|
||||||
|
H4 and H5's failed stop ship only with a live proof, staged with a fault switch that exists only in a `-tags faultinject`
|
||||||
|
test build.
|
||||||
|
**`[FACT]` 2026-10-29, Tester 1, test build `0.347.0-rc.1-d63fault` (`audits/d63-2026-10-29/`):** (H4, 08:10 UTC) a
|
||||||
|
throwaway Radicale ran the true 3.8.0 (digest `0d60d544…`) while its newest off-site snapshot held 3.8.1's data; the full
|
||||||
|
restore said „it comes back at the snapshot's version", the fault failed the definition write after the compose was
|
||||||
|
written, the log then read „the live definition is written back before the start" and only then „Starting stack"; the app
|
||||||
|
ran 3.8.0 again, no hold, and an event made before the snapshot and one made after it both read back. (H5, 08:18 UTC) a
|
||||||
|
created-only container held the data volume („Failed to remove volume … (not written into)"); the restore's own stop
|
||||||
|
passed, the put-back's stop failed — „nothing is put back; the copy stays at …; holding the app"; the app page showed the
|
||||||
|
hold sentence; on the restore page the held copy's „Törlés most" was off with „Nem törölhető: az alkalmazás le van
|
||||||
|
állítva, és ez a másolat az út vissza." (D62, 08:15 UTC) a copy kept because a second restore began inside the first one's
|
||||||
|
stable-start window was listed with „visszaállítás: 2026. 10. 29. 09:11 · 37,3 KB · automatikusan törlődik: 2026. 11.
|
||||||
|
28."; a wrong typed name → 400, the right one → 200, the copy gone, one `[WARN]` line; the event got HTTP 400 from hub
|
||||||
|
0.174.1, which does not know the type yet (hub 0.175.0 does). The release image carries no fault switch (`grep -c "FAULT
|
||||||
|
INJECTED"` = 0 on `0.347.0-rc.1`, 2 on the test image).
|
||||||
|
**`[FACT]` 2026-10-29 night, scratch 9202 (`audits/night-2026-10-28/r1256-restore/h5-live/`):** vikunja's first volume
|
||||||
|
held by a created-only container, the household's restore pressed through the page: „Failed to remove volume
|
||||||
|
vikunja_vikunja_data before the restore … (not written into)", the restore failed, the put-back poured the live copy of
|
||||||
|
both volumes, the app ran, no hold, and a project made before the restore point AND one made after it both read back. H4
|
||||||
|
(off-site only, and a fault there races the write-back) and H5's failed stop are proven by tests only; their live proof
|
||||||
|
is owed to an attended session.
|
||||||
|
|
||||||
|
**`[FACT]` 2026-10-29 03:29 UTC, scratch 9202 (`audits/night-2026-10-28/r1256-restore/h8-live/`):** three restore
|
||||||
|
copies planted by hand (dated 2026-10-01, 2026-10-05 and 2026-10-28); the first six-hour sweep after the controller start
|
||||||
|
removed the two older than 14 days, one of them under the standing app paperless, and kept the young one; the job took
|
||||||
|
8 ms. The young copy was then removed by hand.
|
||||||
|
|
||||||
**[DESIGN] 2026-08-22 — the failure ladder of a database restore: replay → rollback → hold.**
|
**[DESIGN] 2026-08-22 — the failure ladder of a database restore: replay → rollback → hold.**
|
||||||
Recorded here rather than only in a closed register row, because a decision that survives only inside
|
Recorded here rather than only in a closed register row, because a decision that survives only inside
|
||||||
a closed work item is a decision nobody will find.
|
a closed work item is a decision nobody will find.
|
||||||
@@ -885,6 +989,10 @@ run makes it longer.
|
|||||||
|
|
||||||
### 6.5 Kept data — what a removed app leaves on the drive (controller v0.274.0, `09` §3 decision 36)
|
### 6.5 Kept data — what a removed app leaves on the drive (controller v0.274.0, `09` §3 decision 36)
|
||||||
|
|
||||||
|
*Second-drive copies that are KEPT (D25) and the rule that a rejected backup block never deletes one: §8.2a (R-1252, 2026-10-24).*
|
||||||
|
|
||||||
|
*[DESIGN] R-1220 (2026-10-26 night, controller main `7e15a37`, ships with the next controller release): an app whose `app.yaml` EXISTS but does not parse counts as INSTALLED — every folder its compose binds, on every registered drive, is live and never listed as kept data (fail closed; one WARN names the app).*
|
||||||
|
|
||||||
**What it is.** "Remove the app, keep my data" leaves the app's drive folder (`<drive>/appdata/<app>`, or the folder
|
**What it is.** "Remove the app, keep my data" leaves the app's drive folder (`<drive>/appdata/<app>`, or the folder
|
||||||
its definition binds through `${HDD_PATH}`) in place. A reinstall over it no longer runs silently into the old files
|
its definition binds through `${HDD_PATH}`) in place. A reinstall over it no longer runs silently into the old files
|
||||||
(R-657): the install asks „A megőrzött adataimat használom" / "Use my kept data" (the database from the newest copy of
|
(R-657): the install asks „A megőrzött adataimat használom" / "Use my kept data" (the database from the newest copy of
|
||||||
@@ -910,6 +1018,26 @@ holds no database, or **does not record its data version** (§6.6 — a unit wri
|
|||||||
a mixed or mismatched unit is refused by §6.6's own rule. Then a dated folder's files move back and the one unit restore
|
a mixed or mismatched unit is refused by §6.6's own rule. Then a dated folder's files move back and the one unit restore
|
||||||
runs. The downloaded copy is removed on every path. A failed download or a refusal leaves the kept files as they were.
|
runs. The downloaded copy is removed on every path. A failed download or a refusal leaves the kept files as they were.
|
||||||
|
|
||||||
|
**[FACT] 2026-10-25 — why the „another drive" check let another box's copy through (R-1271; catalog sweep 2026-10-24,
|
||||||
|
Tester4: Paperless-ngx, Nextcloud, Immich).** The check (`KeptCopyAt`, `internal/backup/kept_load.go`) compares the
|
||||||
|
unit's `app.yaml` `HDD_PATH` with the drive the kept files sit on. That is a mount PATH, and every box mounts its first
|
||||||
|
data drive under the same name (`/mnt/felhom-drives/hdd_1`) — so a copy taken of another box's disk matches. It is a
|
||||||
|
path check, not a physical identity, and it stays one: loading an old box's data on a new box is what off-site is for.
|
||||||
|
What chose the wrong copy was the DATE: the rehearsal box's clock had been set to 2026-10-28, its off-site snapshot
|
||||||
|
carried that time, this box had no data time of its own to cap it (`offsiteDataTime`), and „newer than every local
|
||||||
|
copy" took it over the family's own copy of the same evening. Immich's 20 minutes of 404 were the ROUTING, not a half
|
||||||
|
load: the unit brought the old box's `SUBDOMAIN` (`photos`), so the app answered at `photos.<domain>` and the probe
|
||||||
|
asked `immich.<domain>`; its containers were healthy.
|
||||||
|
|
||||||
|
**[DESIGN] 2026-10-25 — a copy dated in the future is never „the newest" (R-1271; brief 2026-10-25 Part 4, the safest
|
||||||
|
reading; controller `bd9f4f7`, unreleased, rides v0.343.0).** A copy whose date lies more than **10 minutes** past this box's clock (`KeptClockSkew`: NTP keeps a box
|
||||||
|
within a second; ten minutes covers a box that synced late after a power cut) is future-dated. It never counts as the
|
||||||
|
newest: the newest copy with a true date wins (a tie goes local); the household's own local copy wins over any
|
||||||
|
off-site copy that is not newer by a true date. A future-dated copy is offered only when no other copy exists — then
|
||||||
|
the question says its date is in the future and does not recommend it (the „use" button is not the primary one). The
|
||||||
|
question always names the copy it will use. An off-site copy is never refused only because another box of the
|
||||||
|
household made it.
|
||||||
|
|
||||||
**It is NOT backed up.** No tier captures `<drive>/kept` or a leftover `appdata/<x>` of a removed app; the page says
|
**It is NOT backed up.** No tier captures `<drive>/kept` or a leftover `appdata/<x>` of a removed app; the page says
|
||||||
so („Erről nem készül mentés." / "This is not backed up."). What brings it back into an app is the unit it carries
|
so („Erről nem készül mentés." / "This is not backed up."). What brings it back into an app is the unit it carries
|
||||||
(or the app's own unit on the drive), listed per row as „Visszatölthető innen" / "Can be loaded from".
|
(or the app's own unit on the drive), listed per row as „Visszatölthető innen" / "Can be loaded from".
|
||||||
@@ -973,6 +1101,7 @@ both. Files written under different pins (a failed leg) make the unit `mixed`.
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| own unit (Tier 1), second-drive mirror (Tier 2), kept-data Load | present, `compose/` names the data's pins | the data starts under that definition; if it differs from what ran, the restore pages' first sentence names the backup and the version (the kept-data Load has its own sentence and does not add this one) |
|
| own unit (Tier 1), second-drive mirror (Tier 2), kept-data Load | present, `compose/` names the data's pins | the data starts under that definition; if it differs from what ran, the restore pages' first sentence names the backup and the version (the kept-data Load has its own sentence and does not add this one) |
|
||||||
| same | present, `compose/` names OTHER pins, or `mixed` | **refused before anything is touched** (`ErrUnitVersionMismatch`): „…ezért a visszaállítás nem indult el. Az alkalmazás érintetlen" |
|
| same | present, `compose/` names OTHER pins, or `mixed` | **refused before anything is touched** (`ErrUnitVersionMismatch`): „…ezért a visszaállítás nem indult el. Az alkalmazás érintetlen" |
|
||||||
|
| off-site (Tier 3) | present, `mixed`, or a definition that does not belong to its data | **refused before anything is touched** like the rows above (`unitVersionCheck`, `offbox_reconstitute.go:826-833`; row added 2026-10-24) |
|
||||||
| off-site (Tier 3) | present, the snapshot's version differs from what runs | the snapshot unit's definition is written into the stack dir (and pinned) right after the stop, before any file, volume or database is touched; the database service is resolved from it |
|
| off-site (Tier 3) | present, the snapshot's version differs from what runs | the snapshot unit's definition is written into the stack dir (and pinned) right after the stop, before any file, volume or database is touched; the database service is resolved from it |
|
||||||
| any | absent — a unit or snapshot written before v0.275.0, or with an unstamped file | restores **as before** (its captured definition; the off-site path the live one), with a WARN naming it |
|
| any | absent — a unit or snapshot written before v0.275.0, or with an unstamped file | restores **as before** (its captured definition; the off-site path the live one), with a WARN naming it |
|
||||||
|
|
||||||
@@ -999,13 +1128,30 @@ requires a database dump written after the conversion whose recorded engine is t
|
|||||||
image is re-pulled on restore. A version its maker has deleted cannot start. Measured 2026-09-26: the 42 ladder
|
image is re-pulled on restore. A version its maker has deleted cannot start. Measured 2026-09-26: the 42 ladder
|
||||||
digests all resolve today (`audits/version-travel-2026-09-26/A7/`). **Accepted as a known limit — operator ruling
|
digests all resolve today (`audits/version-travel-2026-09-26/A7/`). **Accepted as a known limit — operator ruling
|
||||||
2026-10-12 (S8, `09` §3 decision 203; R-698 closed).** What a household meets: a restore whose version the maker has
|
2026-10-12 (S8, `09` §3 decision 203; R-698 closed).** What a household meets: a restore whose version the maker has
|
||||||
deleted stops at the image pull, the app stays as it was, and the household uses another copy or a newer version.
|
deleted stops at the image pull, the app stays as it was (since D48 by ORDER — the pull comes before the stop; before it,
|
||||||
|
by the D22 put-back), and the household uses another copy or a newer version.
|
||||||
The operator's follow-up — *„will need to mirror the apps we support later, just to be sure"* — is **R-933**: mirror
|
The operator's follow-up — *„will need to mirror the apps we support later, just to be sure"* — is **R-933**: mirror
|
||||||
the images of every catalog app we support into our own registry, before scale. Until it is built, this limit holds.
|
the images of every catalog app we support into our own registry, before scale. Until it is built, this limit holds.
|
||||||
**Since decision 53 (`09` §3, 2026-09-30 evening):** a box keeps only the image each app service runs and the one before
|
**Since decision 53 (`09` §3, 2026-09-30 evening):** a box keeps only the image each app service runs and the one before
|
||||||
it; older images of the app are deleted. A restore that needs an older version re-pulls it — as before; the limit
|
it; older images of the app are deleted. A restore that needs an older version re-pulls it — as before; the limit
|
||||||
above is unchanged, and kept data (decision 40) is not touched by the image clean-up.
|
above is unchanged, and kept data (decision 40) is not touched by the image clean-up.
|
||||||
|
|
||||||
|
**[DESIGN] 2026-10-25 — a restore downloads first (operator ruling D48, `09` §3 decision 257; controller `2d71270`,
|
||||||
|
unreleased — rides v0.343.0).** „The app stays as it was" in the paragraph above is now true BY ORDER, not by the D22
|
||||||
|
put-back. After the existing refusals and before the first stop or change, every restore path resolves every image the
|
||||||
|
definition that will RUN names — present on the box, or pulled now — under the image-work lock held from the first
|
||||||
|
check until the restore's last start (R-863: the image clean-up cannot delete a pulled image before the start). A pull
|
||||||
|
that fails is a REFUSAL: the app is not stopped, its data is not moved, and the household reads which version, and
|
||||||
|
why: „Ennek a mentésnek a verzióját (%s) nem lehet letölteni: a készítője már nem kínálja…" when Docker says the version
|
||||||
|
does not exist (`manifest unknown`, `not found`, `pull access denied`), „…most nem lehet letölteni… Próbáld újra
|
||||||
|
később…" for everything else (a busy or unreachable registry). The operator gets one `backup_run_failures` line (leg
|
||||||
|
`restore-image-unavailable`). The paths: the local unit restore — and with it the second drive's unit restore, the
|
||||||
|
second drive's whole restore (its `CheckOnly` pass carries the pull, so its file half never runs first), a kept-data
|
||||||
|
load from a unit and from off-site, and a set-aside unit (D47) — the off-site restore, the second drive's file restore
|
||||||
|
and the no-unit fallback. Not covered: the `.fab` import — it removes the running app's volumes before its definition is pulled (R-1279).
|
||||||
|
|
||||||
|
*[DESIGN] The `.fab` route, R-1187 (2026-10-26 night, controller main, next release): the export carries a symlink AS A LINK and stops on a FIFO, socket or device naming its path; a drive folder that cannot be packed fails the export (never left out). The import writes only through `os.Root`, keeps owners and modes, restores a link only when it points DOWNWARD inside the folder (no absolute target, no „..”, no link on the way — others refused and listed), refuses while another operation holds the box and refuses an empty or relative `HDD_PATH`; its compose runs get the controller's own `USERDATA_PATH`/`IMPORT_PATH`; the database loads through the backup side (`appbackup.ImportDump`: mariadb / mariadb-admin, psql ON_ERROR_STOP) — the import's own `mysqladmin`/`mysql` calls failed on every MariaDB 11+ image. Live: `audits/night-fixes-2026-10-26/`.*
|
||||||
|
|
||||||
### 6.7 Why an off-site run failed — the failure classifier (R-571)
|
### 6.7 Why an off-site run failed — the failure classifier (R-571)
|
||||||
|
|
||||||
**[FACT, read from source 2026-10-05, felhom-controller `114ff27`]** When an off-site (restic) run fails, the
|
**[FACT, read from source 2026-10-05, felhom-controller `114ff27`]** When an off-site (restic) run fails, the
|
||||||
@@ -1341,12 +1487,13 @@ Added 2026-08-02 from `audits/SPIKE-recovery-unit-space-2026-08-02.md` and
|
|||||||
which is true and is the whole story for a drive-resident app. **It is not the whole story for an app
|
which is true and is the whole story for a drive-resident app. **It is not the whole story for an app
|
||||||
with no data drive, and that case was undocumented until now.**
|
with no data drive, and that case was undocumented until now.**
|
||||||
|
|
||||||
**The unit is RETAINED, not staged.** `GetAppDrivePath` (`felhom-controller/internal/backup/backup.go:245-255`)
|
**The unit is RETAINED, not staged.** `GetAppDrivePath` (`felhom-controller/internal/backup/backup.go`, now near line 441)
|
||||||
returns the app's `HDD_PATH` if it has one and **`systemDataPath` otherwise** — what
|
returns the app's `HDD_PATH` if it has one and **`systemDataPath` otherwise** — what
|
||||||
`internal/appbackup/paths.go:26-27` calls *"the SSD-only system-data fallback"*. Nothing deletes a unit
|
`internal/appbackup/paths.go:26-27` calls *"the SSD-only system-data fallback"*. Nothing deletes a unit
|
||||||
after it is copied onward: the only prune is F5 (`backup.go:1053-1112`), which removes residue on an
|
after it is copied onward: the only prune is F5 (`pruneStalePrimaryDirs`, `backup.go`, now near line 1514), which removes residue on an
|
||||||
**old** drive when an app **moves**. So for every app without a data drive, `mp1` holds the kept copy
|
**old** drive when an app **moves**. So for every app without a data drive, `mp1` holds the kept copy
|
||||||
indefinitely.
|
indefinitely. *(Dated note 2026-10-24: `mp1` no longer exists — §6.1 / line 386; the system data now lives under the
|
||||||
|
one `mp0` volume's `sys_drive/` folder, `felhom-agent/configs/build-golden.sh`.)*
|
||||||
|
|
||||||
**What a unit contains, which bounds the problem** (`internal/backup/recovery_unit.go:20-25`):
|
**What a unit contains, which bounds the problem** (`internal/backup/recovery_unit.go:20-25`):
|
||||||
`compose/` + `db-dumps/` + `volume-dumps/` (named-volume tars) + `manifest.json`. **It never contains
|
`compose/` + `db-dumps/` + `volume-dumps/` (named-volume tars) + `manifest.json`. **It never contains
|
||||||
@@ -1495,7 +1642,10 @@ crosses the line — **R-158**.
|
|||||||
> Eleven rows carry a blank RTO — 4, 5, 8, 9, 10, 11, 11b, 12, 13, 14, 15 — and only six of those are
|
> Eleven rows carry a blank RTO — 4, 5, 8, 9, 10, 11, 11b, 12, 13, 14, 15 — and only six of those are
|
||||||
> deferred work. **Row 5** has no recovery to time (a derived copy), **row 13** has no route by
|
> deferred work. **Row 5** has no recovery to time (a derived copy), **row 13** has no route by
|
||||||
> design, **row 14** is proven and merely never stopwatched, **row 11b** is a note not a row, and
|
> design, **row 14** is proven and merely never stopwatched, **row 11b** is a note not a row, and
|
||||||
> **row 15 is an open DEFECT (R-104) that this line does NOT cover.**
|
> **row 15 was an open DEFECT (R-104) that this line did NOT cover** — *R-104 was CLOSED 2026-10-05 (controller
|
||||||
|
v0.297.0: a lock that survives the self-heal gets the class `locked`, its own note and a retry, `offbox.go:156-162`;
|
||||||
|
CLOSED-ITEMS.md; corrected 2026-10-24).* **Row 16** (added 2026-10-25, D44 #6, R-1265 — after this banner) has a blank RTO
|
||||||
|
too: a registry rebuild from the ep0 release copy was never timed.
|
||||||
>
|
>
|
||||||
> **NO STATUS MOVED ON THE DAY THIS WAS WRITTEN, because nothing was proven that day.** A stopping
|
> **NO STATUS MOVED ON THE DAY THIS WAS WRITTEN, because nothing was proven that day.** A stopping
|
||||||
> line that promotes a row is a stopping line that lies. Full text, and the conditions that reopen
|
> line that promotes a row is a stopping line that lies. Full text, and the conditions that reopen
|
||||||
@@ -1528,13 +1678,14 @@ crosses the line — **R-158**.
|
|||||||
| 7 | **Guest stopped and does not come back** | everything | guest-power watchdog (60 s, `onboot` as the deliberate-stop discriminator) | automatic | **120 s** | | **PROVEN** | agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human |
|
| 7 | **Guest stopped and does not come back** | everything | guest-power watchdog (60 s, `onboot` as the deliberate-stop discriminator) | automatic | **120 s** | | **PROVEN** | agent v0.107.0 replay — 120 s unattended vs the incident's 587 s with a human |
|
||||||
| 8 | **Host dies (hardware), drives intact** | the data drives; ep0's PBS namespace; the hub's Recipe + Escrow | **install a new host, then** `--selftest=bring-up -mode dr` per guest, **then** re-attach drives by `durable_id` | **operator** (SSH) | | 7 d | **IMPLEMENTED** **`[BETA-DEFERRED]`** | bring-up code exists and has **never been executed** (`CAMPAIGN-8…:522`); the drive half of the plan is empty on every box (**R-105**) |
|
| 8 | **Host dies (hardware), drives intact** | the data drives; ep0's PBS namespace; the hub's Recipe + Escrow | **install a new host, then** `--selftest=bring-up -mode dr` per guest, **then** re-attach drives by `durable_id` | **operator** (SSH) | | 7 d | **IMPLEMENTED** **`[BETA-DEFERRED]`** | bring-up code exists and has **never been executed** (`CAMPAIGN-8…:522`); the drive half of the plan is empty on every box (**R-105**) |
|
||||||
| 9 | **Whole box lost (fire/theft) — host and drives gone** | ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow | new hardware → day-0 → escrow-consume with **R** → restore guests from PBS → app data from Tier-3 | **operator + customer** (R) | | 7 d (guest) · 24 h (app data) | **IMPLEMENTED / UNPROVEN** **`[BETA-DEFERRED]`** | every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (`06-offsite-connectivity.md:327`) **`[FACT]` 2026-10-03 (decision 70): ep0's datastore now has a nightly copy on DooPlex** (PBS pull-sync `ep0-felhom-offsite`, `remove-vanished false`, weekly verify, failures mailed to the operator); first pull 201 s, 12 GB, 4 of 4 snapshots, matching ep0 namespace for namespace. Restore route: `runbooks/ep0-datastore-copy.md` — never walked. `audits/offsite-lock-build-2026-10-03/partF/`. **`[FACT]` 2026-10-04: the restore route from the DooPlex copy was WALKED** on demo-hp (scratch VMID): listed in 2 s, restored in 186 s (15 GB logical), data read by `pct mount`; the restored config carries `onboot: 1` and the host's real drive binds — strip them on a restore beside the original (R-834). The copy now keeps 8 weekly copies (decision 71). `runbooks/ep0-datastore-copy.md` route 1. |
|
| 9 | **Whole box lost (fire/theft) — host and drives gone** | ep0 PBS namespace; the restic repo; the hub's Recipe + Escrow | new hardware → day-0 → escrow-consume with **R** → restore guests from PBS → app data from Tier-3 | **operator + customer** (R) | | 7 d (guest) · 24 h (app data) | **IMPLEMENTED / UNPROVEN** **`[BETA-DEFERRED]`** | every leg exists; the composed path has never been run. The destructive S5 drill is operator-gated and unrun (`06-offsite-connectivity.md:327`) **`[FACT]` 2026-10-03 (decision 70): ep0's datastore now has a nightly copy on DooPlex** (PBS pull-sync `ep0-felhom-offsite`, `remove-vanished false`, weekly verify, failures mailed to the operator); first pull 201 s, 12 GB, 4 of 4 snapshots, matching ep0 namespace for namespace. Restore route: `runbooks/ep0-datastore-copy.md` — never walked. `audits/offsite-lock-build-2026-10-03/partF/`. **`[FACT]` 2026-10-04: the restore route from the DooPlex copy was WALKED** on demo-hp (scratch VMID): listed in 2 s, restored in 186 s (15 GB logical), data read by `pct mount`; the restored config carries `onboot: 1` and the host's real drive binds — strip them on a restore beside the original (R-834). The copy now keeps 8 weekly copies (decision 71). `runbooks/ep0-datastore-copy.md` route 1. |
|
||||||
| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** **`[BETA-DEFERRED]`** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. **2026-09-01 (SPIKE R-95, `audits/SPIKE-r95-offsite-delete-2026-09-01.md`) — THIS ROW'S STATUS IS UNCHANGED, but the seven-day bound beside it is now known to be unverified.** Measured on BOTH boxes over their own SFTP credential, with controls: **no `.snapshots` is visible to either sub-account**, and the account is jailed. Either none exist or a sub-account cannot see them — and a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel, not a product capability (R-429; the confirming field needs the provider API, fenced by §11-D). **The PBS shape does NOT transfer**: the sub-account API has one permission axis, `readonly`, and a backup target cannot be read-only — PBS is a server that can refuse, a Storage Box is a filesystem that runs nothing. **What the spike removed as a fear:** withdrawing delete does NOT wedge the store (measured — restic treats a dead owner's lock as stale and proceeds). **What it added:** `unlock --remove-all` reports success while deleting nothing (R-430). **2026-09-01, LATER THE SAME DAY — THE PROBE ABOVE LOOKED FOR THE WRONG NAME AND THIS ROW'S SECOND CLAUSE IS NOW HALF WRONG.** It searched `.snapshots`; the vendor documents `/.zfs/snapshot`. Re-probed at the documented path on BOTH boxes, with controls, the two halves separate cleanly: **(a) the LIVE repository is deletable by the box — unchanged, R-95 stands;** **(b) the daily SNAPSHOTS of it are not writable by anything — PROVEN, not cited:** a write into `/.zfs/snapshot` is refused (`dest open …: Failure`) while the identical write to the account home succeeds. Seven daily snapshots confirmed in the panel (R-429). **So this row's "the restic repo is NOT protected the same way" is true of the repository and FALSE of its snapshots** — a deletion costs at most the day since the last snapshot, recoverable per-file (vendor). **Two limits kept honest:** a sub-account sees the snapshot directory EMPTY, so per-file recovery is operator-only today (R-432); and a panel-driven restore rolls back the WHOLE box. **The status is NOT moved** — evidence (b) is measured, but the recovery ROUTE has never been walked, which is what PARTIAL means. **Detection shipped hub v0.111.0 (R-431):** an unexplained fall in the snapshot count is noticed within a day — **but only a fall of MORE than half (R-435).** **2026-09-01, THE DRILL — CLAUSE (b) OF THE RE-SCOPE ABOVE IS WITHDRAWN; THE STATUS IS STILL NOT MOVED.** The re-scope said a deletion is *recoverable per-file*. **It is not, from the box: no snapshot is reachable by ANY name.** MEASURED on demo-hp, read-only, no delete verb issued: **777,600** exact names in the vendor form `YYYY-MM-DDTHH-MM-SS` (nine full days, second granularity) plus 126 alternative shapes — **zero hits**, against a control where the identical 600-name batch returns a path that exists (6/6). **Structural cause:** `/home` (`u629488-sub3`) is **st_dev 0,82**, `/.zfs/snapshot` is **st_dev 0,276**, and `/home/.zfs` does not exist — a snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding the repository. **Clause (a) — the box cannot WRITE into the snapshot area — is unchanged and re-confirmed.** So this row's *"recoverable per-file (vendor)"* and its limit *"per-file recovery is operator-only today (R-432)"* both overstate what exists: the remaining routes are a panel rollback of the WHOLE Storage Box and the provider API (fenced, and the hub's client has no snapshot method at all). **R-432 ANSWERED negatively; R-433 opened. RTO still blank — nothing was recoveredLine truncated
|
| 10 | **Ransomware / malicious deletion inside the guest** | PBS offsite (the box **cannot** delete its own snapshots); **the restic repo is NOT protected the same way** | whole-guest restore from PBS to a point before the event | **operator** (SSH) | | 7 d | **PARTIAL** **`[BETA-DEFERRED]`** | R-89 proved the box is refused when deleting its own PBS snapshot (`CAMPAIGN-8…:514-515`). **R-95 (open, ranked #1):** the restic credential **can delete** — `readonly=False`, `forget --prune` runs from the box, and SFTP cannot express append-only (`OPEN-ITEMS.md:13`) **2026-08-30 (R-359): the store is now VERIFIED on a cadence** — a daily `offsite-integrity` job runs `restic check` when the last successful one is over 7 days old. **This is a readability check, not a restore-test (R-87 stays open).** The depth that ships ON now re-reads **100%** of the pack data (R-399 CLOSED, controller v0.228.0) — the sentence here previously said the opposite and was stale. **2026-08-31 (SPIKE R-87):** a scratch restore of every app on demo-hp was measured at **25 s for 8 snapshots / 774 MB**, against 40.3 s for one weekly check — but restic 0.14.0's `--verify` checks size and mtime, **not content**, so nothing available today can vouch for the restored BYTES. Row 4's verdict is UNCHANGED by the spike. `audits/SPIKE-restic-restore-test-2026-08-31.md`. **2026-09-01 (SPIKE R-95, `audits/SPIKE-r95-offsite-delete-2026-09-01.md`) — THIS ROW'S STATUS IS UNCHANGED, but the seven-day bound beside it is now known to be unverified.** Measured on BOTH boxes over their own SFTP credential, with controls: **no `.snapshots` is visible to either sub-account**, and the account is jailed. Either none exist or a sub-account cannot see them — and a snapshot the box cannot see is one the box cannot restore from, so recovery would be an operator act at the Hetzner panel, not a product capability (R-429; the confirming field needs the provider API, fenced by §11-D). **The PBS shape does NOT transfer**: the sub-account API has one permission axis, `readonly`, and a backup target cannot be read-only — PBS is a server that can refuse, a Storage Box is a filesystem that runs nothing. **What the spike removed as a fear:** withdrawing delete does NOT wedge the store (measured — restic treats a dead owner's lock as stale and proceeds). **What it added:** `unlock --remove-all` reports success while deleting nothing (R-430). **2026-09-01, LATER THE SAME DAY — THE PROBE ABOVE LOOKED FOR THE WRONG NAME AND THIS ROW'S SECOND CLAUSE IS NOW HALF WRONG.** It searched `.snapshots`; the vendor documents `/.zfs/snapshot`. Re-probed at the documented path on BOTH boxes, with controls, the two halves separate cleanly: **(a) the LIVE repository is deletable by the box — unchanged, R-95 stands;** **(b) the daily SNAPSHOTS of it are not writable by anything — PROVEN, not cited:** a write into `/.zfs/snapshot` is refused (`dest open …: Failure`) while the identical write to the account home succeeds. Seven daily snapshots confirmed in the panel (R-429). **So this row's "the restic repo is NOT protected the same way" is true of the repository and FALSE of its snapshots** — a deletion costs at most the day since the last snapshot, recoverable per-file (vendor). **Two limits kept honest:** a sub-account sees the snapshot directory EMPTY, so per-file recovery is operator-only today (R-432); and a panel-driven restore rolls back the WHOLE box. **The status is NOT moved** — evidence (b) is measured, but the recovery ROUTE has never been walked, which is what PARTIAL means. **Detection shipped hub v0.111.0 (R-431):** an unexplained fall in the snapshot count is noticed within a day — **but only a fall of MORE than half (R-435).** **2026-09-01, THE DRILL — CLAUSE (b) OF THE RE-SCOPE ABOVE IS WITHDRAWN; THE STATUS IS STILL NOT MOVED.** The re-scope said a deletion is *recoverable per-file*. **It is not, from the box: no snapshot is reachable by ANY name.** MEASURED on demo-hp, read-only, no delete verb issued: **777,600** exact names in the vendor form `YYYY-MM-DDTHH-MM-SS` (nine full days, second granularity) plus 126 alternative shapes — **zero hits**, against a control where the identical 600-name batch returns a path that exists (6/6). **Structural cause:** `/home` (`u629488-sub3`) is **st_dev 0,82**, `/.zfs/snapshot` is **st_dev 0,276**, and `/home/.zfs` does not exist — a snapshot under `/.zfs/snapshot` belongs to a different dataset than the one holding the repository. **Clause (a) — the box cannot WRITE into the snapshot area — is unchanged and re-confirmed.** So this row's *"recoverable per-file (vendor)"* and its limit *"per-file recovery is operator-only today (R-432)"* both overstate what exists: the remaining routes are a panel rollback of the WHOLE Storage Box and the provider API (fenced, and the hub's client has no snapshot method at all). **R-432 ANSWERED negatively; R-433 opened. RTO still blank — nothing was recoveredLine truncated
|
||||||
| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its Longhorn volume backup | operator (`kubectl`) | | 24 h + weekly (Longhorn `RETAIN 1` each) | **UNPROVEN** **`[BETA-DEFERRED]`** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) |
|
| 11 | **Hub lost** | every box's data plane, every tier, every Lane-1 route | none needed for recovery **of a box**; the hub itself restores from its encrypted database copy on ep0 (6-hourly, `scripts/hub-db-backup`; **since 2026-10-24 the hub volume is no longer in DooPlex's Longhorn backup**, D40 / R-1243) | operator (`kubectl`) | | 6 h (ep0, 14 daily + 8 weekly) | **UNPROVEN** **`[BETA-DEFERRED]`** | a hub restore has never been performed. The backup target is `nfs://192.168.0.180` — **DooPlex itself** — and exactly **2** restore points exist (LIVE, INV Part D2.2) |
|
||||||
| 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** **`[BETA-DEFERRED]`** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) |
|
| 11b | *consequences while the hub is gone* | — | — | — | | | **[FACT]** **`[BETA-DEFERRED]`** | **day:** nothing customer-visible breaks; events queue (`settings.go:1466-1485`). **week:** the operator alarm plane is dark, no claim/reset codes, no config or floor convergence, no PBS-secret re-issue. **permanently:** escrow custody and break-glass credentials are gone (INV Part D2.4) |
|
||||||
| 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** **`[BETA-DEFERRED]`** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D **`[FACT]` 2026-10-03:** the PBS (whole-guest) off-site copy now also exists OFF Hetzner — DooPlex pulls ep0's datastore nightly (decision 70). The restic tier is still Hetzner-only. |
|
| 12 | **Offsite provider lost (Hetzner)** | everything on-premises: both drives, both whole-guest tiers | none needed — on-premises recovery is unaffected. Re-provision a new offsite target. | operator | | | **[FACT]** **`[BETA-DEFERRED]`** | **restic and PBS share the provider** (INV Part E.1). Whether they share an account and payment method is **UNKNOWN** → §11-D **`[FACT]` 2026-10-03:** the PBS (whole-guest) off-site copy now also exists OFF Hetzner — DooPlex pulls ep0's datastore nightly (decision 70). The restic tier is still Hetzner-only. |
|
||||||
| 13 | **Customer loses R** | every byte, every tier, the Recipe | **on-premises recovery is unaffected** — Tier-1/2/3 and the whole-guest tiers all work while the guest and host live. What is lost: the ability to re-establish **host identity** and to open the escrowed offsite key material after a host loss | — | | | **NONE for host-loss** | R exists in **zero** system copies by design (INV Part C, row 24). Whether the operator should be able to recover is **open** → §11-A, §11-B. **D5 narrows this further:** Tier-1/2 app recovery needs only the drive, so losing R now costs the offsite route and host identity, never local app recovery (§7.4) |
|
| 13 | **Customer loses R** | every byte, every tier, the Recipe | **on-premises recovery is unaffected** — Tier-1/2/3 and the whole-guest tiers all work while the guest and host live. What is lost: the ability to re-establish **host identity** and to open the escrowed offsite key material after a host loss | — | | | **NONE for host-loss** | R exists in **zero** system copies by design (INV Part C, row 24). Whether the operator should be able to recover is **open** → §11-A, §11-B. **D5 narrows this further:** Tier-1/2 app recovery needs only the drive, so losing R now costs the offsite route and host identity, never local app recovery (§7.4) |
|
||||||
| 14 | **Host SSH management plane dies** | everything | 3-layer break-glass: tmpfiles → 60 s watchdog → hub-vaulted `root@pam` on the PVE console | operator | | | **PROVEN** | `runbooks/break-glass.md`; the original incident and its fix |
|
| 14 | **Host SSH management plane dies** | everything | 3-layer break-glass: tmpfiles → 60 s watchdog → hub-vaulted `root@pam` on the PVE console | operator | | | **PROVEN** | `runbooks/break-glass.md`; the original incident and its fix |
|
||||||
| 15 | **Interrupted offsite run leaves a stale lock** | everything | manual `restic unlock --remove-all` | **operator** (SSH) | | | **DEFECT** | the self-heal exists (`offbox.go:634-648`) but the probe fails first and `classifyResticProbe` has no lock case → fail-fast, operator told *„ismeretlen okból"* → **R-104** |
|
| 15 | **Interrupted offsite run leaves a stale lock** | everything | manual `restic unlock --remove-all` | **operator** (SSH) | | | **FIXED 2026-10-05 (R-104, v0.297.0; was DEFECT — corrected 2026-10-24)** | the self-heal exists (`offbox.go:634-648`) but the probe fails first and `classifyResticProbe` has no lock case → fail-fast, operator told *„ismeretlen okból"* → **R-104** |
|
||||||
|
| 16 | **The container registry is lost with DooPlex** (the released images and agent packages) | the release copy on ep0 (D33): the newest 3 release tags of `felhom-controller`, `felhom-hub` and `felhom-samba`, and the newest 5 agent packages (binary + config bundle), in a plain folder outside PBS (`/srv/felhom-release-copy`) | push the copy back into a registry with the same name — `runbooks/total-loss-of-dooplex.md` „Restore the release copy” (same digests, no rebuild; `scripts/release-copy/`) | **operator** (ep0 root) | | 24 h (nightly 01:40) | **PARTIAL** | `audits/day-2026-10-23/08-release-copy-D33.txt`: the copy was pulled back, every sha256 and digest matched, and a controller image from it ran on a scratch box. Not yet walked: the push into a real new registry and the agent-package upload. Added 2026-10-25 (D44 = all picks, R-1265) |
|
||||||
|
|
||||||
### 8.2 The ONE exception to row 5's derived-copy rule (R-403, controller v0.230.0)
|
### 8.2 The ONE exception to row 5's derived-copy rule (R-403, controller v0.230.0)
|
||||||
|
|
||||||
@@ -1590,6 +1741,19 @@ or pattern. **Why:** the data-loss hunt of 2026-10-20 proved two nights that del
|
|||||||
(R-1175: one share group's reconcile removed the other groups' copies, and a dead drive's copy; R-1181: a missing or
|
(R-1175: one share group's reconcile removed the other groups' copies, and a dead drive's copy; R-1181: a missing or
|
||||||
emptied folder took its copy within one night). Controller: `internal/backup/tier2_kept.go`.
|
emptied folder took its copy within one night). Controller: `internal/backup/tier2_kept.go`.
|
||||||
|
|
||||||
|
**`[DESIGN]` 2026-10-24 — a rejected backup block never deletes a copy (R-1252; review round 2 HUNT-2 H1, H2, H9;
|
||||||
|
committed in controller `518a135`, unreleased).** The night's reconcile (deleting dest dirs the legs no longer name)
|
||||||
|
runs ONLY under a valid classification. An app whose block is rejected (one bad line rejects the whole block) or absent
|
||||||
|
falls back to the legacy legs, and nothing is reconciled that night — every existing copy stays. A REJECTED block puts
|
||||||
|
„Az alkalmazás mentési beállításai most nem olvashatók…" on the app's second-drive status and tells the operator once a
|
||||||
|
day (the operator-only digest `backup_run_failures`, leg `tier2-settings-unreadable`). A kept copy (D25) is always in
|
||||||
|
the keep set, also when its leg later leaves the declared set (H2); Remove with „remove backups" drops the kept-copy
|
||||||
|
records of what it deleted (H9). **`[FACT]` 2026-10-24 night, 9202 (`audits/restore-holes-2026-10-24/P2-9202/`):**
|
||||||
|
audiobookshelf with a household file in `media/audiobooks`; two control runs kept the copy; one bogus path in the
|
||||||
|
block → „the backup block is REJECTED … every existing copy on the second drive is KEPT", the copy there, the status
|
||||||
|
sentence set, the operator digest sent (9202 has no hub: logged as dropped); the review's run on 0.342.0 the same day
|
||||||
|
deleted that copy (`review-round-2-2026-10-24/proofs/H2-bad-backup-block/`).
|
||||||
|
|
||||||
### 8.2b The second-drive copy runs in a memory-capped container that never outlives the copy [DESIGN + FACT, R-1214, 2026-10-22/23 nights; rides controller 0.340.0]
|
### 8.2b The second-drive copy runs in a memory-capped container that never outlives the copy [DESIGN + FACT, R-1214, 2026-10-22/23 nights; rides controller 0.340.0]
|
||||||
|
|
||||||
The copy (`rsync -a --delete`) runs in a container of the controller's own image, capped at 512 MB, so its page cache
|
The copy (`rsync -a --delete`) runs in a container of the controller's own image, capped at 512 MB, so its page cache
|
||||||
@@ -1599,12 +1763,98 @@ refuses a missing one — then the in-process copy runs). **The container must n
|
|||||||
fix, 2026-10-23 night): a fixed name and the label `felhom.tier2-copy=1`; on any error, the timeout included, it is
|
fix, 2026-10-23 night): a fixed name and the label `felhom.tier2-copy=1`; on any error, the timeout included, it is
|
||||||
removed by name; at controller start, before the scheduler and the web server, every labelled container is removed;
|
removed by name; at controller start, before the scheduler and the web server, every labelled container is removed;
|
||||||
and a restore is refused while any second-drive copy runs or a leftover may still run (the other direction of R-1180 —
|
and a restore is refused while any second-drive copy runs or a leftover may still run (the other direction of R-1180 —
|
||||||
a `--delete` beside a restore removes what the restore brings back). **`[FACT]` 9202, rc.5
|
a `--delete` beside a restore removes what the restore brings back). *(As built — noted 2026-10-24: the refusal is in the
|
||||||
|
web layer's `restoreOpBlocked` (`internal/web/restore_wizard.go:226`); the install-time „use my kept data" load checks
|
||||||
|
only `RestoreStatus`/`IsRunning` (`internal/api/kept_install.go:89-92`) — a register row of review round 2.)* **`[FACT]` 9202, rc.5
|
||||||
(`audits/night-2026-10-23/P1-straycopy/`):** a controller restart in the middle of a 16 GB copy → the new controller
|
(`audits/night-2026-10-23/P1-straycopy/`):** a controller restart in the middle of a 16 GB copy → the new controller
|
||||||
removed the old container at start; a restore probe during the copy was refused; the redone copy equals the source; a
|
removed the old container at start; a restore probe during the copy was refused; the redone copy equals the source; a
|
||||||
15-second test limit → the container removed, the partial copy stopped growing. Controller:
|
15-second test limit → the container removed, the partial copy stopped growing. Controller:
|
||||||
`internal/backup/tier2_copy_lifecycle.go`.
|
`internal/backup/tier2_copy_lifecycle.go`.
|
||||||
|
|
||||||
|
### 8.2d An earlier install's unit is set aside, never rewritten [DESIGN, operator ruling D47, 2026-10-25 08:38; `09` §3 decision 256; R-1269 — controller `02daebe`, unreleased, rides v0.343.0]
|
||||||
|
|
||||||
|
**[FACT] Measured before (controller 0.342.0, which is `e1613ae` for these files):** the catalog sweep 2026-10-24
|
||||||
|
(three times) and 9202 2026-10-25 (`audits/restore-truth-2026-10-25/before/`): Tandoor, one account, „Mentés most",
|
||||||
|
remove keeping data and backups, install again. Within five minutes the periodic refresh rewrote ONLY the unit's
|
||||||
|
`compose/app.yaml` (checksum `cbfcb538…` → `e19d2610…`) — the new install's database password — beside the OLD install's
|
||||||
|
data. Its one „same definition" test was the image pins (`recovery_unit.go:194`, the R-696 `frozen` rule), and a
|
||||||
|
reinstall of the same version has the same pins. A restore then paired the new password with the old data directory:
|
||||||
|
`password authentication failed` ×44 (app) / ×23 (postgres), the app stayed `starting`, the account was gone from
|
||||||
|
view. The next night's dump legs would have overwritten the old data as well, and the stale-drive prune deleted a unit
|
||||||
|
on another drive whatever install it belonged to.
|
||||||
|
|
||||||
|
**[DESIGN] The rule.** A recovery unit's definition and its data are written only together, by a backup of the same
|
||||||
|
install. Every install has an identity (`install_id` in its `app.yaml`): minted at a fresh install and at a `.fab`
|
||||||
|
import, carried by every load-then-save, taken FROM the unit by a restore (the app then holds that unit's data), and
|
||||||
|
minted once for an app installed before D47. The unit records it in its own `app.yaml`. Every writer of a unit asks
|
||||||
|
first — the capture and its refresh, the night database dump, the pre-backup and the update's pre-backup dumps, the
|
||||||
|
volume dump, the second drive's unit copy, the stale-drive prune. A unit of ANOTHER install (different identities; for
|
||||||
|
a unit written before D47, a carried secret that differs — a reinstall mints new ones) is **moved whole** (a rename on
|
||||||
|
its own drive; never a copy, never an edit) to `<namespace>/backups/set-aside/<app>/<UTC stamp>/` and recorded as a
|
||||||
|
kept copy (§8.2a's record: kind `unit`, reason `earlier_install`); the new install then gets its own unit from its
|
||||||
|
first backup. A move that cannot be done **refuses the write** and leaves the old unit byte for byte (an ERROR line
|
||||||
|
names the app and both identities). Nothing moves a unit while that app is being restored.
|
||||||
|
|
||||||
|
**What the household sees.** The restore page lists the set-aside unit under „Megőrzött másolatok" („…· egy korábbi
|
||||||
|
telepítés mentése") with **„Visszaállítás"** — the one unit-restore body, with the unit's own database password — and
|
||||||
|
„Eltávolítás". Only those two, and the app's remove with „delete its backups too", ever remove it (the §8.2a / R-1252 H9
|
||||||
|
rule). After a restore from it, the app is that unit's install; the app's own unit of the install it replaced is then
|
||||||
|
set aside ~~at the next backup~~ **at the restore itself** — never lost. **[DESIGN — R-1288, 2026-10-25 night, the brief's safest reading; controller `b5863a9`, unreleased]:** the restore of a set-aside unit moves the replaced install's unit aside whole right after it (success, or „not started yet" — the data stays, D49 = A), one log line „replaced by the restored install", **no operator alarm** — the household chose this restore; before, the next backup's writer set it aside with a second alarm and a second kept copy (measured on Tester 1, 0.343.0). **[FACT]** on the night's rig (rc.2): the line, the account back, the next backup set nothing aside, one set-aside mail only (`audits/night-2026-10-25/rig/r1288-live.txt`). A set-aside's operator mail (R-1287, hub v0.171.0, unreleased) reads „⚠️ … a backup of an earlier install was set aside — nothing failed", a warning.
|
||||||
|
|
||||||
|
**Known limit, stated.** A box that reinstalled and refreshed BEFORE this release already holds a rewritten unit; its
|
||||||
|
old password is gone and nothing can bring it back. A unit from before D47 of an app with no carried secret has no
|
||||||
|
signal: it is treated as this install's (the old behaviour) and gets the identity at its next write.
|
||||||
|
**Why a volume app's reinstall asks no kept-data question:** the question (§6.5, decision 36) is about the app's drive
|
||||||
|
FOLDER (`${HDD_PATH}` binds under `appdata/`); a volume app keeps nothing there — „remove, keep my data" removes its named
|
||||||
|
volumes and keeps only its backups — so the install starts empty, and the kept backup is now the set-aside unit on the
|
||||||
|
restore page. **[DESIGN — R-1278, controller `fbec5cb`, unreleased]:** the install page now says so for an app not installed — „the new install starts empty; load the backup afterwards on the Restore page" — with a link (set-aside and removed-app units counted).
|
||||||
|
|
||||||
|
### 8.2e A restore whose app does not start is never „success" [DESIGN, R-1270, 2026-10-25 — controller `b680db8` + fix A; D49 = A (operator 2026-10-25); released in controller 0.343.0]
|
||||||
|
|
||||||
|
**[FACT] Measured before:** the health check after a restore was a WARN on every path, so a Tandoor restore whose app
|
||||||
|
stayed `starting` was recorded `ok: true`, „…visszaállítva — az alkalmazás újraindult." (catalog sweep 2026-10-24; 9202
|
||||||
|
2026-10-25, `audits/restore-truth-2026-10-25/before/`). **[DESIGN]** Every restore path — the unit restore (so the
|
||||||
|
second drive's unit and whole restore, a kept load, a set-aside unit), the off-site restore, the second drive's file
|
||||||
|
restore and the no-unit fallback — returns „not started" when the app does not reach running within the health wait
|
||||||
|
(90 s). The household reads „A(z) %s: az adatokat visszatöltöttük, de az alkalmazás még nem indult el. Várj pár percet,
|
||||||
|
majd nyisd meg újra. Ha így marad, szólj nekünk." (the record `ok: false`, `not_started: true`); the operator gets one
|
||||||
|
`backup_run_failures` line (leg `restore-not-started`). **The safest reading, the operator may change it:** a timeout
|
||||||
|
does NOT pour the old data back — a slow first start (a database migration) is common, and a put-back would throw away a
|
||||||
|
correct restore. The data stays restored and the restore's undo copy is KEPT — the same „kept for support" state a failed
|
||||||
|
put-back leaves (§8 D20/D22), so no new retention rule. A watcher looks 30 more minutes: when the app runs, the record
|
||||||
|
turns to success („…elindult: a visszaállítás kész.") and the copy is removed, as after any successful restore; if it
|
||||||
|
never runs, the copy stays for support.
|
||||||
|
|
||||||
|
**D49 = A (operator ruling 2026-10-25, attended release run; `09` §3 decision 258):** the safest reading above is the rule —
|
||||||
|
the box keeps the restored data, keeps the old copy, tells the operator, and never puts the old data back by itself.
|
||||||
|
**Fix A (controller 0.343.0, R-1270 follow-up):** „started" is one helper on every path — the stack running on every
|
||||||
|
sample of a 45 s window, no container's `RestartCount` grown in it, every container with a healthcheck `healthy` (a
|
||||||
|
container without one is judged by running alone). An app that starts and then crash-loops is „not started". The undo
|
||||||
|
copy's RELEASE is a separate, longer test: 3 minutes of the same after the verdict (10 s samples, watched up to 30
|
||||||
|
minutes). **[FACT]** 9202 rc.6: an image that exits after 20 s → not started, copy kept, one operator line; the original
|
||||||
|
unit back → success, copy released 3 minutes later. Tester 1 on 0.343.0: a unit restore and a set-aside restore, each
|
||||||
|
„ran stably for 3m0s … the restore copy is released" (`audits/release-2026-10-26/`).
|
||||||
|
|
||||||
|
### 8.2c Decisions in code that no document recorded [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
|
||||||
|
|
||||||
|
Read at `file:line` on `main` (controller `321df1f`, hub `44d19042`) by the main session:
|
||||||
|
|
||||||
|
- **A second-drive copy is limited to 60 minutes**; a stray copy container that cannot be removed blocks every
|
||||||
|
restore and is retried every minute; copies into one destination run one at a time and are never killed by name
|
||||||
|
while running (`felhom-controller/controller/internal/backup/tier2_copy_lifecycle.go:40,113-121,150-172`).
|
||||||
|
- **The household's „Eltávolítás" of a kept copy (D25) is refused while any backup, restore or second-drive copy runs**
|
||||||
|
(`internal/web/kept_copies.go:54-58`).
|
||||||
|
- **A start that fails AFTER a successful put-back** leaves the app stopped and NOT held, keeps the copy, and tells the
|
||||||
|
household `restore_failed_put_back_not_started` (`internal/backup/offbox_restore_undo.go:245-247`).
|
||||||
|
- **The hub's off-site delivery checker** (`hub/internal/monitor/offsite_delivery.go`): a box that took no off-site
|
||||||
|
credential is reported `offsite_delivery_stuck` (operator) and healed by `maybeHeal` (a burned vs a stuck shape);
|
||||||
|
a box that declares it waits for the household's recovery code (`awaiting_recovery_key`) is skipped — no event, no
|
||||||
|
heal (R-1230, `:103-110`).
|
||||||
|
- **The release copy (D33)** — nightly 01:40, the newest 3 image tags and 5 agent packages to ep0 under its own
|
||||||
|
limited user, capped by the sender (`felhom.eu/scripts/release-copy/`). ~~has no row in this matrix~~ It is row
|
||||||
|
**16** of the matrix since 2026-10-25 (D44 = all picks, R-1265); its other homes are the poster facts and
|
||||||
|
`runbooks/total-loss-of-dooplex.md`.
|
||||||
|
|
||||||
### 8.1 The blank cells, listed explicitly
|
### 8.1 The blank cells, listed explicitly
|
||||||
|
|
||||||
Per the rule that a blank is a finding, here they are:
|
Per the rule that a blank is a finding, here they are:
|
||||||
@@ -1622,6 +1872,7 @@ Per the rule that a blank is a finding, here they are:
|
|||||||
| 13 | RTO, RPO | not a timed recovery; a capability loss |
|
| 13 | RTO, RPO | not a timed recovery; a capability loss |
|
||||||
| 14 | RTO | the break-glass path is proven but was never timed |
|
| 14 | RTO | the break-glass path is proven but was never timed |
|
||||||
| 15 | RTO | the manual unlock was performed but not timed |
|
| 15 | RTO | the manual unlock was performed but not timed |
|
||||||
|
| 16 | RTO | a registry rebuild from the ep0 release copy has never been timed (row added 2026-10-25, D44 #6, R-1265) |
|
||||||
|
|
||||||
**Also unmeasured, and not representable as a row** (INV Part F.3): any restore larger than 155.5 MB
|
**Also unmeasured, and not representable as a row** (INV Part F.3): any restore larger than 155.5 MB
|
||||||
from restic; any restore of a customer data drive (no tier holds one whole); the weekly PBS
|
from restic; any restore of a customer data drive (no tier holds one whole); the weekly PBS
|
||||||
@@ -1875,7 +2126,7 @@ to now *implement* D5 remains an open scheduling decision, not a blocked one.
|
|||||||
| host-loss plan → an actual restore | **EXECUTES NOTHING BY CONSTRUCTION** | `felhom-agent/internal/dr/plan.go:1-4` |
|
| host-loss plan → an actual restore | **EXECUTES NOTHING BY CONSTRUCTION** | `felhom-agent/internal/dr/plan.go:1-4` |
|
||||||
| host rebuilt as its former self | **NEVER DONE** | INV Part D1 |
|
| host rebuilt as its former self | **NEVER DONE** | INV Part D1 |
|
||||||
| escrow **consume** in a real recovery | **SPIKE-LEVEL ONLY** | `06-offsite-connectivity.md:327` |
|
| escrow **consume** in a real recovery | **SPIKE-LEVEL ONLY** | `06-offsite-connectivity.md:327` |
|
||||||
| hub DB restore from its Longhorn backup | **NEVER DONE** | INV Part D2.2 |
|
| hub DB restore from its ep0 copy into a running hub (the weekly restore-test opens the copy and checks it; a hub has never been brought up on it) | **NEVER DONE** | INV Part D2.2; the Longhorn backup it used to cite is off since 2026-10-24 (D40) |
|
||||||
| a **customer** performing a restore unassisted | **MISSING AS EVIDENCE** | `00-capability-map.md:75` |
|
| a **customer** performing a restore unassisted | **MISSING AS EVIDENCE** | `00-capability-map.md:75` |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|||||||
@@ -127,6 +127,11 @@ live 2026-08-23 — one event across 22 scans.
|
|||||||
|
|
||||||
- **Operator/hub event + dashboard banner.** `app_start_failed` is **not** in
|
- **Operator/hub event + dashboard banner.** `app_start_failed` is **not** in
|
||||||
`settings.DefaultEnabledEvents`, so by default it does **not** e-mail the customer.
|
`settings.DefaultEnabledEvents`, so by default it does **not** e-mail the customer.
|
||||||
|
- **`[DESIGN]` 2026-10-29 — no alarm after the operator's hold clear (R-1350).** The clear starts nothing, so a held app
|
||||||
|
stays stopped; the clear now records that stop as WANTED (desired state „stopped") for an app that is not up — once the
|
||||||
|
hold is found, before it is lifted — so the scan's tri-state reads „the household asked" and stays quiet. The
|
||||||
|
household's Start records „running" and ends it. Measured before the fix on Tester 1: `app_start_failed` 30 s after a
|
||||||
|
clear from the hub (controller, R-1350).
|
||||||
- **F-OBS heartbeat**, every 20 scans (~10 min), at `[INFO]`:
|
- **F-OBS heartbeat**, every 20 scans (~10 min), at `[INFO]`:
|
||||||
`[deadapp] check alive: N scans since boot, M deployed app(s) evaluated, K currently down`.
|
`[deadapp] check alive: N scans since boot, M deployed app(s) evaluated, K currently down`.
|
||||||
This line exists because an absent alarm and a stopped detector look identical in a log.
|
This line exists because an absent alarm and a stopped detector look identical in a log.
|
||||||
@@ -135,7 +140,9 @@ live 2026-08-23 — one event across 22 scans.
|
|||||||
|
|
||||||
**The vocabulary is the HUB's and it is exact: `{info, warning, error, critical}`.** Anything else is
|
**The vocabulary is the HUB's and it is exact: `{info, warning, error, critical}`.** Anything else is
|
||||||
**coerced to `info` at ingest**, and `info` is dropped by `severityNotifies` before *both* delivery
|
**coerced to `info` at ingest**, and `info` is dropped by `severityNotifies` before *both* delivery
|
||||||
legs. So a severity outside the set means the event is stored, answers `200`, shows on the dashboard —
|
legs — except the paired recoveries in `recoveredPairedDownTypes` (`node_recovered`, `host_recovered`,
|
||||||
|
`whole_guest_backup_recovered`, `pbsdr_box_recovered`, `offsite_box_recovered`, `storage_reconnected`,
|
||||||
|
`health_recovered`), which run before the gate and are mailed (`notify/dispatcher.go` — corrected 2026-10-24, review round 2). So a severity outside the set means the event is stored, answers `200`, shows on the dashboard —
|
||||||
and is e-mailed to **nobody**.
|
and is e-mailed to **nobody**.
|
||||||
|
|
||||||
**This shipped twice.** `DiskAlertKind.Severity` emitted `"warn"` until controller v0.215.0;
|
**This shipped twice.** `DiskAlertKind.Severity` emitted `"warn"` until controller v0.215.0;
|
||||||
@@ -155,8 +162,10 @@ Three things now hold it:
|
|||||||
3. **The dispatcher's `unrecognized severity` branch is kept**, because the hub's own monitor checkers
|
3. **The dispatcher's `unrecognized severity` branch is kept**, because the hub's own monitor checkers
|
||||||
call `ProcessEvent` directly and never pass the ingest handler. For them it is the only guard.
|
call `ProcessEvent` directly and never pass the ingest handler. For them it is the only guard.
|
||||||
|
|
||||||
**Who gets it.** `processOperator` consults only `operatorOn`, the address and a 1-hour cooldown —
|
**Who gets it.** `processOperator` consults only `operatorOn`, the address and a cooldown —
|
||||||
**never customer preferences** — so a valid severity always reaches the operator. `processCustomer`
|
1 hour, except 5 min for node/host liveness and 6 h for a storage-fill warning (`operatorCooldownFor`); it records a
|
||||||
|
`backup_tier_skipped` in a box's FIRST hour as „suppressed" instead of mailing it (R-723), and the mail hold applies —
|
||||||
|
**never customer preferences** (cooldowns and R-723 corrected 2026-10-24) — so a valid severity always reaches the operator. `processCustomer`
|
||||||
consults `operatorOnlyEvents` and then the customer's `enabled_events`.
|
consults `operatorOnlyEvents` and then the customer's `enabled_events`.
|
||||||
|
|
||||||
**`app_start_failed` is customer-switchable but OFF by default** [DESIGN, operator ruling 2026-08-23]:
|
**`app_start_failed` is customer-switchable but OFF by default** [DESIGN, operator ruling 2026-08-23]:
|
||||||
@@ -180,7 +189,10 @@ rather than assumed (2026-08-30):
|
|||||||
- **not** in `operatorOnlyEvents` — the operator leg ignores customer preferences anyway, so the
|
- **not** in `operatorOnlyEvents` — the operator leg ignores customer preferences anyway, so the
|
||||||
operator is always mailed; putting it here would only remove the customer's ability to opt in.
|
operator is always mailed; putting it here would only remove the customer's ability to opt in.
|
||||||
- **not** in `DefaultEnabledEvents` — customer-switchable, default OFF, the same ruling as
|
- **not** in `DefaultEnabledEvents` — customer-switchable, default OFF, the same ruling as
|
||||||
`app_start_failed`. The checkbox already exists at `settings_notifications.html:34`.
|
`app_start_failed`. **Superseded by operator Q1 (2026-10-12, R-388 slice 1):** since controller v0.309.0
|
||||||
|
`backup_integrity_failed` (with `crossdrive_failed`, `storage_reconnected`, `health_recovered`) IS in
|
||||||
|
`DefaultEnabledEvents` — the default for a household with no saved list (`internal/settings/settings.go`, pinned by
|
||||||
|
`TestR388_DefaultListAsGroups`; dated note 2026-10-24, review round 2). The checkbox already exists at `settings_notifications.html:34`.
|
||||||
|
|
||||||
**And a caveat that belongs in the alarm ladder rather than only in the backup document:** a
|
**And a caveat that belongs in the alarm ladder rather than only in the backup document:** a
|
||||||
`backup_integrity_ok` at the shipped depth means *the index, the pack inventory and the snapshot graph
|
`backup_integrity_ok` at the shipped depth means *the index, the pack inventory and the snapshot graph
|
||||||
@@ -244,6 +256,19 @@ today's engine, and its build (read the counter of every container; „probably
|
|||||||
before any code, by the brief's measure-first rule — `09` §3 decision 155 stands for the operator to confirm or drop.
|
before any code, by the brief's measure-first rule — `09` §3 decision 155 stands for the operator to confirm or drop.
|
||||||
Cost measured for the record: one `docker exec … cat memory.events` across demo-hp's 21 containers takes 1.6 s wall
|
Cost measured for the record: one `docker exec … cat memory.events` across demo-hp's 21 containers takes 1.6 s wall
|
||||||
(0.47 s user, 0.44 s sys); cloudflared has no `cat` and reads as unknown. `audits/design-build-2026-10-06/C/`.
|
(0.47 s user, 0.44 s sys); cloudflared has no `cat` and reads as unknown. `audits/design-build-2026-10-06/C/`.
|
||||||
|
**[DESIGN] 2026-10-28 night — the host's counter beside Docker's flag (R-528, as designed in the row 2026-10-26; agent
|
||||||
|
`6f933d6` + controller `eed23e7`, unreleased):** the agent serves `GET /guest/oom-kills` — the kernel's `oom_kill` of
|
||||||
|
each of the asking guest's containers, read on the host from `/sys/fs/cgroup/lxc/<vmid>/ns/system.slice/docker-<id>.scope/memory.events`
|
||||||
|
(0444; the non-root agent reads it — no sudo); the 30-s scan asks once per pass (5 s cap; an agent without the route
|
||||||
|
404s and is asked again after 10 minutes; no agent = Docker's flag only). A running container whose CURRENT run counts a
|
||||||
|
kill (its scope is new at each start) raises the same `app_oom` (once per run), whatever Docker's flag says; the
|
||||||
|
event's details carry `source` (`docker`, `cgroup` or both) and the host's count is the run's kill count — so a
|
||||||
|
container with a false flag now also reaches the storm rung. **[FACT] live 2026-10-28 23:27 on Tester 1** (test builds
|
||||||
|
of both, put back afterwards): a throwaway Radicale capped at 128 MB, `tail /dev/zero` inside it → the hog killed
|
||||||
|
(exit 137), the container kept running, the host counter 0 → 1; the controller logged the OOM line naming Radicale,
|
||||||
|
„seen by docker,cgroup", and pushed ONE `app_oom` (none in the next two minutes); the operator's mailbox (a second
|
||||||
|
channel) held exactly one mail, details `"source":"docker,cgroup"`. Docker 29.8.2 reported this kill too, so the
|
||||||
|
cgroup-only path is proven by the tests, not live. `audits/night-2026-10-28/r528-oom/`.
|
||||||
|
|
||||||
**The box STOPS a crash loop or an out-of-memory storm (decision 28 of `09` §3, R-667, controller v0.269.0 /
|
**The box STOPS a crash loop or an out-of-memory storm (decision 28 of `09` §3, R-667, controller v0.269.0 /
|
||||||
hub v0.123.0, 2026-09-24).** Alarming was not enough: gokapi crash-looped for hours at 385 → 546 restarts
|
hub v0.123.0, 2026-09-24).** Alarming was not enough: gokapi crash-looped for hours at 385 → 546 restarts
|
||||||
@@ -287,12 +312,15 @@ bands.
|
|||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| `controller_slow_crashloop` | warning | **operator only** | hub, from the agent's `slow_crashloop_since` moving (agent v0.132.0, hub v0.117.0) | host ids and vmids; the household's side of it is the dashboard coming back each time |
|
| `controller_slow_crashloop` | warning | **operator only** | hub, from the agent's `slow_crashloop_since` moving (agent v0.132.0, hub v0.117.0) | host ids and vmids; the household's side of it is the dashboard coming back each time |
|
||||||
| `restore_interrupted` | warning | **the household** (and the operator) | controller v0.246.0 at startup, once per interruption | they pressed restore, were told it started, and can run it again — Hungarian `customerMessages` entry |
|
| `restore_interrupted` | warning | **the household** (and the operator) | controller v0.246.0 at startup, once per interruption | they pressed restore, were told it started, and can run it again — Hungarian `customerMessages` entry |
|
||||||
|
| `controller_unresponsive` (2026-10-29, R-1344) | warning | **operator only** | hub, from the agent's `unresponsive_since` moving (agent v0.169.0, hub v0.176.0) | the controller runs but does not answer; the agent only alarms — the operator decides (Restart controller) |
|
||||||
|
| `restore_copy_deleted` (2026-10-29, D62) | info | **operator only** | controller v0.347.0, per „Törlés most" | the household did it after the warning; the record (who, app, date, size) is support's |
|
||||||
|
|
||||||
| Family | Grain | Key carries | Why |
|
| Family | Grain | Key carries | Why |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| app down (`app_start_failed`) | **per APP** | `…:<stack_name>` | no digest exists for it — see below |
|
| app down (`app_start_failed`) | **per APP** | `…:<stack_name>` | no digest exists for it — see below |
|
||||||
| update outcome (`app_update_undone`, `app_update_held`) | **per APP**, both legs | `…:<stack_name>` | hub v0.120.0 — one mail per app per outcome; the household leg has its own register (`perAppCustomerCooldownEvents`) |
|
| update outcome (`app_update_undone`, `app_update_held`) | **per APP**, both legs | `…:<stack_name>` | hub v0.120.0 — one mail per app per outcome; the household leg has its own register (`perAppCustomerCooldownEvents`) |
|
||||||
| OOM storm (`app_oom_storm`) | **per APP**, operator | `…:<stack_name>` | hub v0.121.0 — no digest; the controller already sends it at most once per container run |
|
| OOM storm (`app_oom_storm`) | **per APP**, operator | `…:<stack_name>` | hub v0.121.0 — no digest; the controller already sends it at most once per container run |
|
||||||
|
| more per-APP types (added 2026-10-24 from `dispatcher.go`'s per-stack set) | **per APP** | `…:<stack_name>` | `app_hold_no_whole_copy`, `app_stopped_unhealthy` (both legs), `app_repaired_after_crash`, `app_unexplained_upload`, `app_unexplained_cpu` |
|
||||||
| backup run (`backup_run_failures`) | per RUN | `…:<run_id>` | a digest already lists every failing app; one per run |
|
| backup run (`backup_run_failures`) | per RUN | `…:<run_id>` | a digest already lists every failing app; one per run |
|
||||||
| tiered backup (`whole_guest_backup_failed`, …) | per TIER | `…:<tier>` | the tiers fail independently and mean different things |
|
| tiered backup (`whole_guest_backup_failed`, …) | per TIER | `…:<tier>` | the tiers fail independently and mean different things |
|
||||||
| everything else, incl. `crossdrive_failed` and `backup_integrity_failed` | per TYPE, per hour | — | coarse **on purpose** |
|
| everything else, incl. `crossdrive_failed` and `backup_integrity_failed` | per TYPE, per hour | — | coarse **on purpose** |
|
||||||
@@ -367,6 +395,14 @@ one info line beside `host_crash_restart`; `operatorOnlyEvents`, pinned by
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
**More operator-only box alarms, listed 2026-10-24 (review round 2; each is in `notify/dispatcher.go`'s
|
||||||
|
`operatorOnlyEvents`, and each has its home elsewhere):** `wg_tunnel_stale` / `wg_tunnel_ok` — the box's tunnel to ep0:
|
||||||
|
a handshake older than 6 h or stamped in the future, edge-triggered (D30, R-1204; `api/host_tunnel.go`, `05` §15.4b);
|
||||||
|
`host_swap_short` / `host_swap_ok` — the host's swap below 90 % of min(the guests' swap allowance, 8192 MB) (R-1192;
|
||||||
|
`api/host_swap.go`); `bind_old_host_linked` — a household asked for a new bind link while its old host is still linked
|
||||||
|
(R-1203; `05` §13); `golden_not_vouched` — a published golden the hub does not vouch (decision 199, R-242);
|
||||||
|
`recovery_older_package` — a household's code opens, or may open, an older sealed package (decision 183, R-304 C).
|
||||||
|
|
||||||
## 6.4 A box that is not always on: the missed-backup deadline and the household's outage mail [DESIGN, hub v0.134.0, 2026-10-05]
|
## 6.4 A box that is not always on: the missed-backup deadline and the household's outage mail [DESIGN, hub v0.134.0, 2026-10-05]
|
||||||
|
|
||||||
Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *operator may reverse*).
|
Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *operator may reverse*).
|
||||||
@@ -376,7 +412,8 @@ Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *op
|
|||||||
never judged (measured 2026-10-05: `1 skipped (down)`). Now a down box is judged on longer lines:
|
never judged (measured 2026-10-05: `1 skipped (down)`). Now a down box is judged on longer lines:
|
||||||
`expected_dbdump_missed` after **48 h** without a `db_dump_completed`, `expected_backup_missed` after **72 h**
|
`expected_dbdump_missed` after **48 h** without a `db_dump_completed`, `expected_backup_missed` after **72 h**
|
||||||
without a whole-guest backup in any retained host report — never for a box first seen less than 48 h ago. A box that
|
without a whole-guest backup in any retained host report — never for a box first seen less than 48 h ago. A box that
|
||||||
died last night still raises only its staleness alarm. A `disabled` box is still skipped (R-321). Pinned by
|
died last night still raises only its staleness alarm. A `disabled` box is still skipped (R-321), and so is a box the household switched off on purpose — by its staleness
|
||||||
|
state or by the host row's hold (R-998, `monitor/deadline.go`; corrected 2026-10-24). Pinned by
|
||||||
`monitor/r872_down_box_test.go`.
|
`monitor/r872_down_box_test.go`.
|
||||||
- **R-873 — "your server cannot be reached" reaches the HOUSEHOLD at most once per 7 days** (`node_stale`,
|
- **R-873 — "your server cannot be reached" reaches the HOUSEHOLD at most once per 7 days** (`node_stale`,
|
||||||
`node_down`, `host_stale`, `host_down`, read from the persisted notification log, so a hub restart does not reset
|
`node_down`, `host_stale`, `host_down`, read from the persisted notification log, so a hub restart does not reset
|
||||||
@@ -387,7 +424,7 @@ Design home: `07` §6.1.1 (`09` decisions 109–110; CC decisions 115–116, *op
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 6.5 One lost off-site copy on a pinned tier [DESIGN, R-435 — `09` §3 decision 191, hub main 2026-10-08, unreleased]
|
## 6.5 One lost off-site copy on a pinned tier [DESIGN, R-435 — `09` §3 decision 191; released in hub v0.144.0 — corrected 2026-10-24]
|
||||||
|
|
||||||
On a **pinned** off-site tier (the hub holds a confirmed append-only key, decision 69) the only legitimate way the
|
On a **pinned** off-site tier (the hub holds a confirmed append-only key, decision 69) the only legitimate way the
|
||||||
snapshot count can fall is a clean-up window the hub opened (decision 68). So the hub raises
|
snapshot count can fall is a clean-up window the hub opened (decision 68). So the hub raises
|
||||||
@@ -470,7 +507,7 @@ was ruled that way on its own merits and does not pre-judge the redesign.
|
|||||||
**Filed as a PRODUCT DECISION, not a defect** — see the register. It is the operator's call to take
|
**Filed as a PRODUCT DECISION, not a defect** — see the register. It is the operator's call to take
|
||||||
separately, and no part of it was implemented here.
|
separately, and no part of it was implemented here.
|
||||||
|
|
||||||
### 8.1 The design — „only what you can act on" [DESIGN, R-388 — `09` §3 decision 200, 2026-10-12; NOT BUILT]
|
### 8.1 The design — „only what you can act on" [DESIGN, R-388 — `09` §3 decision 200, 2026-10-12; slice 1 BUILT in controller v0.309.0 — three group switches, details under Részletek (`internal/settings/notif_groups.go`); corrected 2026-10-24]
|
||||||
|
|
||||||
*Operator ruling 2026-10-12 (S5): design now, build in slices later. Written by CC from the direction above and the
|
*Operator ruling 2026-10-12 (S5): design now, build in slices later. Written by CC from the direction above and the
|
||||||
live page (controller, `settings_notifications.html`: **18 event switches** plus the address, the cooldown and the
|
live page (controller, `settings_notifications.html`: **18 event switches** plus the address, the cooldown and the
|
||||||
@@ -531,3 +568,183 @@ list is shown as custom and saved back unchanged; the default list maps to group
|
|||||||
multi-day backup fault reads as „nobody noticed" if the household ever finds out.
|
multi-day backup fault reads as „nobody noticed" if the household ever finds out.
|
||||||
2. **The cooldown field („hány óránként legfeljebb"):** keep it on the page or move it under „Részletek"? Pick:
|
2. **The cooldown field („hány óránként legfeljebb"):** keep it on the page or move it under „Részletek"? Pick:
|
||||||
**under Részletek** — it is an engineering choice, which is what this redesign removes.
|
**under Részletek** — it is an engineering choice, which is what this redesign removes.
|
||||||
|
|
||||||
|
## 9. The upload and CPU watch — „does the app explain its own traffic?" [DESIGN — built; released in controller v0.342.0 and hub v0.169.0 (deployed) on the operator's yes, 2026-10-24 — corrected 2026-10-24, review round 2]
|
||||||
|
|
||||||
|
**Why** (operator ruling 2026-10-23 19:22, `CONTEXT.md`): DooPlex's hacked Plex sent ~474 GB in a day with no viewer and
|
||||||
|
used ~10 cores with no film being converted (`audits/plex2shell-felhom-impact-2026-10-23/`). Nothing on a Felhom box
|
||||||
|
would have noticed: the health check sees whole-box CPU at one instant, and nothing looks at upload at all. **The idea:**
|
||||||
|
when an app sends a lot (or burns CPU) for a long time, ask the APP whether that is its work. Work the app explains is
|
||||||
|
normal use; work it does not explain is the sign of a hacked app. Numbers and measurements:
|
||||||
|
`audits/SPIKE-upload-watch-2026-10-23.md`; the code: controller `internal/upwatch` (README §8e).
|
||||||
|
|
||||||
|
1. **The line.** Learnt from what the box already does: while its off-site push runs, the controller samples its own
|
||||||
|
sent bytes once a second; the best ≥ 10-second rate is kept per night (14 nights, settings.json `upwatch.line`); a
|
||||||
|
night counts when that window moved ≥ 10 MB; the line is the highest counting night. A **lower bound**, shown so
|
||||||
|
(„legalább N Mbit/s", system settings). **AMENDED 2026-10-24 (operator ruling 07:26, R-1247): the weekly line TEST.**
|
||||||
|
The night's one-second read showed how low that floor is (demo-hp „at least 10 Mbit/s"; spike §2.1). So the host's
|
||||||
|
agent (agent ≥ 0.164.0, `internal/linetest`) runs a 10-second iperf3 upload once a week to ep0's tunnel-only server:
|
||||||
|
Tuesday 10:00–15:59 local from a per-host slot, only after 10 quiet minutes of the physical uplink (< 2 Mbit/s each
|
||||||
|
minute), under the one-heavy-op gate, best of 3, never in the night chain; „server busy" → later the same day. The
|
||||||
|
controller reads it (`GET /host/line-test`) every 15 minutes; the watch judges against the higher of test and floor;
|
||||||
|
the system page says „N Mbit/s (heti sebességméréssel)". demo-hp measured 90.7 Mbit/s (`audits/line-test-2026-10-24/`).
|
||||||
|
2. **Trigger, per app** (its compose project's containers summed; exact counters from the Docker Engine API, once a
|
||||||
|
minute): upload ≥ max(5 Mbit/s, 25 % of the line) for 15 minutes; CPU ≥ max(150 %, 50 % × cores) of one core for 30
|
||||||
|
minutes. Infrastructure (the controller — its own push is the line measure —, traefik, cloudflared, Samba) is never
|
||||||
|
judged; the box's own work on an app (a backup run, an update, a deploy) is never judged.
|
||||||
|
3. **The explanation.** qBittorrent (`up_info_data`) and Transmission (`uploadedBytes`): explained when the container
|
||||||
|
sent ≤ 1.25 × what the app counted + 1 Mbit/s. Plex: any viewer explains upload; any transcode explains CPU. An app
|
||||||
|
that does not answer gives no number (a hacked app may refuse its own API). No explainer: upload alarms only at ≥ 10 ×
|
||||||
|
its own p95 for 60 minutes after ~20 h of history; CPU at ≥ max(200 %, 75 % × cores) for 60 minutes.
|
||||||
|
4. **Who is told** (the R-388 rule, §8.1): **`app_unexplained_upload` / `app_unexplained_cpu`** (warning) and their
|
||||||
|
`_cleared` (info, recorded) are **operator-only** (hub `operatorOnlyEvents`), one per episode, per-app cooldown. The
|
||||||
|
household is told only about the qBittorrent limit, because they can change it (R-1246 asks the operator).
|
||||||
|
5. **qBittorrent's default upload limit:** 50 % of the TESTED line (D41 = 50 %, operator 2026-10-24; never of the
|
||||||
|
push floor), every 15 minutes, re-set when the line moves > 20 %; never over a limit the household set (one there
|
||||||
|
first, or any change from the value qBittorrent stored for the box — it keeps KiB/s). No test → none set. **A box
|
||||||
|
that seeds all day (R-1250, brief 2026-10-24; committed agent `29bcf44`+`1e087bb`, controller `046b808`, not yet
|
||||||
|
released):** the 10 quiet minutes never come, so when the agent reports `waiting_for_quiet` (due, in the window,
|
||||||
|
tunnel up, the full 10-minute window watched and busy) and qBittorrent or Transmission runs, the controller writes
|
||||||
|
`upwatch.line_test_hold` (the app's limit exactly as stored) FIRST, holds the app at 1 KiB/s (Transmission 1 kB/s),
|
||||||
|
calls `POST /host/line-test/run` (5 s settle, 10 s quiet check < 2 Mbit/s, three runs 5 s apart — about 55 s), then
|
||||||
|
puts the old limit back, the household's own included; a change made during the test stays; a killed controller's
|
||||||
|
start-up check puts it back; the 50 % rule waits while a record exists. Measured on demo-hp with a real seed (10
|
||||||
|
Mbit/s to DooPlex's qBittorrent): refused `not-quiet` without the hold; with it the uplink fell to 0.02–0.16 Mbit/s,
|
||||||
|
94.2 Mbit/s measured, the app held 56.8 s and back to 1 249 280 B/s exactly (`audits/dooplex-alarms-2026-10-24/`).
|
||||||
|
**[DESIGN — R-1250 drain wait, 2026-10-27; controller `710ceac`, released v0.346.0]:** between the hold and the call
|
||||||
|
the controller waits until every held app REPORTS its own upload ≤ 16 KiB/s on two readings 2 s apart (at most 90 s;
|
||||||
|
an unreadable speed waits 20 s) — qBittorrent 5.2.4 kept sending at full speed ~14 s after its limit fell, inside the
|
||||||
|
agent's 5–15 s quiet read, and four tries in an hour were refused; with the wait the test ran (94.5 Mbit/s, the limit
|
||||||
|
back exactly; `audits/linetest-2026-10-27/`; `09` §3 decision 269).
|
||||||
|
**[DESIGN — R-1261, 2026-10-25 night; controller `a79a292`, unreleased]:** EVERY running seeder is held (qBittorrent
|
||||||
|
and Transmission — one left seeding kept the test refused `not-quiet` every time); all records first
|
||||||
|
(`upwatch.line_test_holds`), then the apps; a record whose app was removed while held is dropped (it used to stop the
|
||||||
|
50 % rule for good); qBittorrent's hold names its slot (`up_limit` / `alt_up_limit` through `app/preferences` —
|
||||||
|
`transfer/uploadLimit` follows the CURRENT speed mode, measured on 5.2.4), so the household's switch to the
|
||||||
|
alternative speed during the test never leaves 1 KiB/s behind. **[DESIGN — R-1292, controller `2619191`, unreleased]:**
|
||||||
|
the 50 % rule owns the NORMAL slot only (`up_limit` through `app/preferences`); the household's alternative limit is
|
||||||
|
never read or written by it, and a mode switch never reads as „the household changed it". **[FACT]** the night's rig (BYO VM, agent 0.165.0,
|
||||||
|
rc.1, two seeders to 9202's qBittorrent): with a mode switch mid-test both limits came back, the test refused
|
||||||
|
(the switch let qBittorrent upload); without, both held (qBittorrent in its alternative slot), 94.8 Mbit/s measured,
|
||||||
|
both limits back exactly (`audits/night-2026-10-25/rig/hold-watch*.txt`).
|
||||||
|
While the box owns it, qBittorrent's page says: „A qBittorrent
|
||||||
|
feltöltése legfeljebb N Mbit/s, hogy a családi internet gyors maradjon. A qBittorrentben átállíthatod."
|
||||||
|
6. **Limits, stated:** *(added 2026-10-24, review round 2: ANY viewer explains any amount of upload, so a hacked app can send freely while one real playback runs — `upwatch/watch.go:275,299`; R-1262)*; an attacker who knows the check can make the app's API lie — a per-container ceiling (`tc` from
|
||||||
|
a helper in the app's network namespace, measured to work in an unprivileged guest; spike §5) would cap the flood
|
||||||
|
whatever the API says; not built. The box cannot tell LAN from internet bytes per app; a host-network container
|
||||||
|
(Samba) has no counter; the push floor is a lower bound, the weekly test is close to the line.
|
||||||
|
7. **Release:** released 2026-10-24 on the operator's yes (controller v0.342.0, hub v0.169.0; R-1244) — corrected 2026-10-24.
|
||||||
|
|
||||||
|
## 10. A held app the operator can see [DESIGN — built 2026-10-26 (commit only): controller `5bd03ad`, hub `7e57e824`; D53 = operator-only, `09` §3 decision 265; R-1291]
|
||||||
|
|
||||||
|
**Why.** A held app is stopped on purpose and stays stopped until a person acts — and today nobody is told after the
|
||||||
|
first minute. demo-hp's Kimai sat held for 11 days (2026-10-14 → 10-26) and was found by accident (R-1286). The
|
||||||
|
controller sends one event when the crash-loop guard stops an app (`app_stopped_unhealthy`); no other hold sends
|
||||||
|
anything, and the hub has no held-app state at all. On a household box the same silence would hide a failed restore.
|
||||||
|
|
||||||
|
**The holds that exist** (controller `settings.RestoreHold.Reason`, the sentences of `holdWhy`, R-1286):
|
||||||
|
|
||||||
|
| Reason | What put it there | Who may clear it |
|
||||||
|
|---|---|---|
|
||||||
|
| `restore_failed` | a restore whose rollback also failed | **a person only** (R-379: support, `--clear-restore-hold`) |
|
||||||
|
| `restore_mixed` | an off-site restore whose database replay failed | a person (the same) |
|
||||||
|
| `update_failed` | a failed update | the household, by restoring the app from its backup |
|
||||||
|
| `unhealthy_stop` | the crash-loop guard | the household's Start (one more try, `09` §3 decision 28) |
|
||||||
|
| `putback_pending` | a restore waiting for its pre-restore copy | nobody — it clears itself when the drive comes back |
|
||||||
|
|
||||||
|
**What the controller reports.** The legacy report is frozen (hub rule), so the event channel carries it:
|
||||||
|
`app_held` when a hold is written (stack, reason, since — the hold's own `At`) and `app_hold_cleared` when it goes.
|
||||||
|
Because an event can be lost, the controller re-sends `app_held` for every hold still in place **once a day**; the hub
|
||||||
|
treats it as idempotent. Both types enter `allowedEventTypes` AND `customerMessages` (hu + en) in the same hub commit
|
||||||
|
(the coupling rule); `app_held` is operator-only in v1.
|
||||||
|
|
||||||
|
**Where the hub shows it.** The host page (and the customer page) get a „Held apps" list: app, reason in one line
|
||||||
|
(the `holdWhy` sentence), since, and who may clear it. Built from the open `app_held` events per customer and stack.
|
||||||
|
|
||||||
|
**When it mails.** One operator mail per hold after a grace that follows the reason: `putback_pending` 6 hours (it
|
||||||
|
should have cleared itself — if not, something is stuck); `restore_failed` / `restore_mixed` 1 day (only a person can
|
||||||
|
act); `update_failed` and `unhealthy_stop` 3 days (the household can act first). Then one reminder a week while it
|
||||||
|
lasts, through the dispatcher's existing per-type cooldown (§6.2). A cleared hold sends nothing.
|
||||||
|
|
||||||
|
**Who clears it.** Unchanged: R-379 holds only a person; the household's Start or restore for its own two. **The hub
|
||||||
|
never clears a hold** — it only shows and reminds.
|
||||||
|
|
||||||
|
**Cost.** One controller release (the two events + the daily re-send, ~½ day with tests) and one hub release (allow-list,
|
||||||
|
the i18n keys, the list on two pages, a reminder checker; ~1 day). No agent change.
|
||||||
|
|
||||||
|
**Ruled (D53, operator 2026-10-26, `09` §3 decision 265):** v1 is operator-only. Household mails for the two holds it can
|
||||||
|
clear itself may follow once the operator list has run.
|
||||||
|
|
||||||
|
**As built (2026-10-26, commit only — releases ride an attended run).** The wire reason is the controller's own constant:
|
||||||
|
`putback_pending` (one word), and the empty legacy reason is sent as `restore_failed`. The controller's reporter is ONE
|
||||||
|
observer of the hold store, ticked every minute (`internal/holdreport`), never a send inside a hold writer; the set it
|
||||||
|
last reported is kept in settings, so a hold cleared while the controller was down is still reported cleared. The hub
|
||||||
|
mirrors the events in `held_apps`, sends `app_held_reminder` (warning, operator-only, per-app cooldown, the subject
|
||||||
|
names the app) once after the grace and then weekly, records the reminder before it sends it, and does not remind about
|
||||||
|
a row the box stopped repeating for 50 h (the list still shows it, marked). `alerting.held_app_grace_override` replaces
|
||||||
|
every grace — a test configuration only. Live proof: `audits/d53-held-apps-2026-10-26/` (a crash-looped throwaway app on
|
||||||
|
9202 against a test hub with a 3-minute grace: listed on both pages, one operator mail, Start cleared it, the row went,
|
||||||
|
every household leg `skipped operator_only`).
|
||||||
|
|
||||||
|
*[DESIGN] R-1298 (2026-10-26 night, controller main `b156e45`): an install's hold (`unhealthy_stop`, `update_failed`, `putback_pending`) of an app that is NOT installed (no `app.yaml`) is not reported — it would remind weekly about an app that does not exist; a restore hold is always reported (R-379, a person clears it).*
|
||||||
|
|
||||||
|
## 11. The security watch — three operator-only signals [DESIGN — built 2026-10-26: felhom.eu `1f62d629`, `6ee23a3b`; R-1303]
|
||||||
|
|
||||||
|
The nightly CI job `security-watch` (03:40 UTC, `scripts/security_watch.py`) reads where each shipped program publishes
|
||||||
|
its security notices (sources and their limits: `audits/security-watch-2026-10-26/SPIKE.md`) and mails **admin@felhom.eu
|
||||||
|
only** — no household, no hub event:
|
||||||
|
|
||||||
|
| Signal | When | Once? |
|
||||||
|
|---|---|---|
|
||||||
|
| **ADVISORY** — „[Felhom security] <programs, box first> — N new advisories" | a notice affects the version we pin | once per (program, notice); a reminder every 7 days while it stays open; a vendor-blog notice never repeats |
|
||||||
|
| **BEHIND** — „[Felhom security] weekly: …" | Monday (or the last summary is 8+ days old) | one summary; it also lists GAP, FLOATING and tonight's NOT CHECKED |
|
||||||
|
| **NOT CHECKED** — „[Felhom security] the watch could not check …" | the same source failed two nights in a row | once per streak; a good night resets it |
|
||||||
|
|
||||||
|
The memory is `documentation/security-watch/state.json`, committed by the job itself; a missing memory re-seeds AND mails
|
||||||
|
that it did. A job that does not finish mails „[felhom CI] security watch FAILED". The mail starts the security-fix lane
|
||||||
|
(capability map B23).
|
||||||
|
|
||||||
|
**[DESIGN — R-1327/R-1328, built 2026-10-27 night, commit only] A fourth source, „runtime", and a third group,
|
||||||
|
„DooPlex".** HTTP2Shell (CVE-2026-111497) sat in the Node.js and Python Brotli INSIDE app images, where no app notice
|
||||||
|
mentions it. The watch now reads each pin's image (`scripts/image_runtime.py`: the registry config, then only where
|
||||||
|
needed the layers — Alpine/Debian package databases, Node's `node_version.h`, Python `.dist-info` — never a pull) and
|
||||||
|
matches it against `scripts/security_watch_runtimes.json`; a hit is an ADVISORY with source „runtime", a read that
|
||||||
|
fails is NOT CHECKED. Results are cached by the tag's digest in `state.json` (`runtime_cache`) and the nightly lookup is
|
||||||
|
a HEAD request (Docker Hub does not count it against the anonymous pull limit — the first run's GETs hit HTTP 429).
|
||||||
|
**The cache fills itself in CI (2026-10-28, R-1327; replaces the 2026-10-27 night's DooPlex filling run).** Images with
|
||||||
|
zstd layers (Home Assistant) are read with the `zstandard` package, version 0.25.0, as a wheel pinned by the sha256 PyPI
|
||||||
|
publishes and checked every time it is opened; the workflows fetch it (`FELHOM_WHEEL_FETCH=1`) into the runner's cache.
|
||||||
|
Not the `zstd` program in the CI runner image: that image is built in the homelab repo, and changing it is a DooPlex
|
||||||
|
change. A missing or wrong wheel makes the zstd layers NOT CHECKED, said in the log's `RUNTIME` line — never a pass. A
|
||||||
|
new image is read on the night its tag first appears; the log's `RUNTIME read=… (… s) from-cache=…` line says how much
|
||||||
|
was read and how long it took. The DooPlex group (`documentation/security-watch/dooplex-workloads.txt`,
|
||||||
|
from homelab-manifests) is read the same way and mailed in its own block; Felhom changes nothing on DooPlex.
|
||||||
|
|
||||||
|
**Docker Hub's tag API is asked gently (2026-10-28, R-1334).** The 2026-10-28 night run was refused 15 times (HTTP 429)
|
||||||
|
when it sent 211 tag-list requests to `hub.docker.com` in about 2 minutes from six workers; a 3.5-minute dry run that
|
||||||
|
morning sent 224 and was refused none. So: the requests to Docker Hub are spaced (one per second across the workers);
|
||||||
|
a 429 waits (its `Retry-After`, else 10/30/60 s) and tries again, and the image is NOT CHECKED only after the fourth
|
||||||
|
try; a tag list is kept for 24 hours per registry path in the CI runner's cache (`~/.cache/felhom-watch-tags.json`, not
|
||||||
|
committed), so two names of one path (`lscr.io/linuxserver/x`, `linuxserver/x`) and a same-day re-run fetch nothing.
|
||||||
|
The log's `TAGS fetched=… from-cache=… retried-429=…` line counts it. No Docker Hub login is used. Two CI runners take turns, each with its own
|
||||||
|
cache, so the cache helps a same-day re-run on one runner, not the nightly; the spacing is what fixes the night (two
|
||||||
|
dry runs 2026-10-28: 0 refused). The image reader (`image_runtime.py`) also waits out a 429 and tries again (3 tries):
|
||||||
|
ghcr.io refused two blob reads when two dry runs read the same 14 new images 15 minutes apart.
|
||||||
|
|
||||||
|
**[DESIGN — 2026-11-02, R-1334] A pin the watch can never read is a stated GAP, not a failed source; a read timeout is tried
|
||||||
|
again.** The memory showed `registry:docker.io` and `runtime` down for a week, and the logs said otherwise: 0 Docker Hub 429
|
||||||
|
on five real nights, on both runners. A streak is per SOURCE, so seven pins that can never be read (six of our own images in
|
||||||
|
the private registry; `minio/minio`, which Docker Hub answers with 401) held both streaks open, and a real outage of either
|
||||||
|
source could not have mailed. Now those seven are GAP, with their reason, in every weekly mail. Only the stated answer counts:
|
||||||
|
`STATED_UNREADABLE` names the pin AND its one answer, so an unlisted 401 (a Docker Hub outage) or the same pin timing out stays
|
||||||
|
NOT CHECKED. A transport failure on any source, and a ghcr.io 429, is tried again (a budget of 30 a night, so an outage cannot
|
||||||
|
pass the job's limit). The log prints `FAILED-SOURCE …` for every failed source of every pin — the table shows one class per
|
||||||
|
pin, so an ADVISORY pin hid its failure — and `FAILED-SOURCES n`. Evidence: `audits/security-2026-11-02/part1/DIAGNOSIS.md`.
|
||||||
|
Tests W12, W15.
|
||||||
|
|
||||||
|
**[DESIGN — 2026-11-02, R-1310] What counts as a notice.** A release that names GHSA ids gives one entry per id (the lowest
|
||||||
|
fixing release, never twice); a release with a security line and no id gives one `release:` entry; a line where „security" is
|
||||||
|
only a word (`WORD_ONLY`: „security announcements", „no longer receives security fixes") gives none. A notice naming only
|
||||||
|
another program in the repo (`PACKAGE_NOT_OURS`: Gitea's runner, Home Assistant's phone apps, Uptime Kuma's rootless image) is
|
||||||
|
not listed; a name not in the table is listed — the safe direction. A `YY.MM.YYYYMMDD` tag is not „newer". Test W16.
|
||||||
|
|
||||||
@@ -65,6 +65,13 @@ build is added to both in the same commit.
|
|||||||
|
|
||||||
### 1.1 The button
|
### 1.1 The button
|
||||||
|
|
||||||
|
> *Dated note 2026-10-24 (review round 2): §1 is the 2026-09 MEASUREMENT before the arc. `UpdateStack` no longer exists
|
||||||
|
> (deleted in v0.237.0, controller `REUSE.md`); the button calls `StartGuardedUpdate` — backup precondition, safety
|
||||||
|
> dump, pin journal, undo copy, health wait, undo (`internal/api/router.go:794`; `internal/stacks/update.go`). The line
|
||||||
|
> numbers below are the measurement's; e.g. `copyTemplates` is now `sync.go:436`, `copyIfChanged` `:700`, the `15m`
|
||||||
|
> default `config.go:399`, `RestartStack` `manager.go:1454` (it now runs `up -d --force-recreate` and errors when no
|
||||||
|
> container restarted, R-1026).*
|
||||||
|
|
||||||
`Manager.UpdateStack` (`felhom-controller/controller/internal/stacks/manager.go:1199`) is two compose
|
`Manager.UpdateStack` (`felhom-controller/controller/internal/stacks/manager.go:1199`) is two compose
|
||||||
commands and nothing else:
|
commands and nothing else:
|
||||||
|
|
||||||
@@ -267,7 +274,8 @@ builds them (`audits/update-rulings-2026-09-23/`).
|
|||||||
next night. **Why:** the update then leans on the freshest copy the box ever has, and there is no
|
next night. **Why:** the update then leans on the freshest copy the box ever has, and there is no
|
||||||
second window for anyone to set or misread. **Replaces:** §3b Q1's recommendation of a fixed
|
second window for anyone to set or misread. **Replaces:** §3b Q1's recommendation of a fixed
|
||||||
02:30–05:00. *For the builder:* `stacks.update_window` (`config.go` L195, default `03:00-05:00`)
|
02:30–05:00. *For the builder:* `stacks.update_window` (`config.go` L195, default `03:00-05:00`)
|
||||||
exists and **no Go code reads it** — it appears only in `config.go` (field + default),
|
exists and **no Go code reads it** *(dated note 2026-10-24: the field is now removed — no controller source names
|
||||||
|
it; `controller/README.md` „`stacks.update_window` is removed")* — it appears only in `config.go` (field + default),
|
||||||
`setup/handlers.go` L482 (written into a new box's `controller.yaml`) and
|
`setup/handlers.go` L482 (written into a new box's `controller.yaml`) and
|
||||||
`configs/controller.yaml.example`. Remove it or fold it into the chain; do not build a second
|
`configs/controller.yaml.example`. Remove it or fold it into the chain; do not build a second
|
||||||
window.
|
window.
|
||||||
@@ -278,17 +286,27 @@ builds them (`audits/update-rulings-2026-09-23/`).
|
|||||||
13. **The test decides, not the tag** (Q3). **REPLACES decision 3's second half, "never across a
|
13. **The test decides, not the tag** (Q3). **REPLACES decision 3's second half, "never across a
|
||||||
major".** The box applies by itself every step the catalog holds, because **the catalog holds only
|
major".** The box applies by itself every step the catalog holds, because **the catalog holds only
|
||||||
tested steps**; the size of the version jump does not matter, the test does. The box does not
|
tested steps**; the size of the version jump does not matter, the test does. The box does not
|
||||||
parse tags to decide. The tag rule (`stacks.CompareImageRefs`) moves to the **catalog gate** as a
|
parse tags to decide. ~~The tag rule (`stacks.CompareImageRefs`) moves to the **catalog gate** as a
|
||||||
push-time safety net: an image move with no test record is refused there. Two exceptions, both
|
push-time safety net:~~ **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** The tag rule (`stacks.CompareImageRefs`) did not move: it
|
||||||
|
stays in the controller (`internal/stacks/updateorder.go`), where it only tells whether an installed app is
|
||||||
|
AHEAD of the catalog; it never decides an update. The catalog gate is the push-time safety net, and it asks for a
|
||||||
|
proven test instead (§6.4 part 4): an image move with no test record is refused there. Two exceptions, both
|
||||||
**marks the test sets on the step**: *files may change* → automatic only when a fresh copy holds
|
**marks the test sets on the step**: *files may change* → automatic only when a fresh copy holds
|
||||||
the files, else a person (this is Q2's refinement); *needs a person* → the tester writes why.
|
the files, else a person (this is Q2's refinement); *needs a person* → the tester writes why.
|
||||||
**Why:** decision 3 was a proxy. "Within a major" was standing in for "known to work", and the
|
**Why:** decision 3 was a proxy. "Within a major" was standing in for "known to work", and the
|
||||||
update night measured the proxy wrong in both directions — minor moves that held
|
update night measured the proxy wrong in both directions — minor moves that held
|
||||||
(adventurelog, outline) and a tag shape it cannot read at all (§3b Q3). The test record is the
|
(adventurelog, outline) and a tag shape it cannot read at all (§3b Q3). The test record is the
|
||||||
thing the proxy was guessing at.
|
thing the proxy was guessing at.
|
||||||
|
**Dated note 2026-10-23 (decision 255, D39 = B):** a second step kind now exists — the `security_fix` step, proven
|
||||||
|
on the box with the bench optional, for an urgent upstream fix with a public notice. Decision 13's rule is
|
||||||
|
unchanged for every other move: a step a box takes was still tested, on at least the box venue.
|
||||||
|
|
||||||
14. **The update ladder** (Q3). A box more than one step behind **climbs one tested step at a time, in
|
14. **The update ladder** (Q3). A box more than one step behind **climbs one tested step at a time, in
|
||||||
order, and never jumps**. Each step is the full guarded update of §6.1. **Why:** a step was tested
|
order, and ~~never jumps~~ the night leg never jumps**. Each step is the full guarded update of §6.1.
|
||||||
|
**[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** An app older than the ladder (its installed version matches no entry's `from`) is never
|
||||||
|
pressed by the night leg (`internal/stacks/unattended.go`). A PERSON's press on such an app takes the catalog's
|
||||||
|
current definition, logged by name (`internal/stacks/ladder.go`, `TestLadder_UnknownInstalledJumpsAndSaysSo`); the
|
||||||
|
guarded update and its undo (decision 15) still run around it. **Why:** a step was tested
|
||||||
from the version before it, not from three versions before it; R-40's multi-major jump is what
|
from the version before it, not from three versions before it; R-40's multi-major jump is what
|
||||||
this prevents. This is the "stepping" §6.1 already assigned to slice 6.
|
this prevents. This is the "stepping" §6.1 already assigned to slice 6.
|
||||||
|
|
||||||
@@ -352,6 +370,17 @@ builds them (`audits/update-rulings-2026-09-23/`).
|
|||||||
both options. **Why (b):** decision 13's mark exists for "does not fit the memory" (RomM, R-635), and
|
both options. **Why (b):** decision 13's mark exists for "does not fit the memory" (RomM, R-635), and
|
||||||
only the app's own memory decides that. The cgroup peak stays in every record (`memory_cgroup_peak_pct`
|
only the app's own memory decides that. The cgroup peak stays in every record (`memory_cgroup_peak_pct`
|
||||||
in the ladder entry). R-652.
|
in the ladder entry). R-652.
|
||||||
|
**Refined 2026-10-28 night (R-1338, the night brief's „fix the measurement — the gate reads the right figure"):** the
|
||||||
|
mark reads the LIVE anon — `anon` without the pages the app already gave back. An allocator that frees with
|
||||||
|
MADV_FREE leaves the pages mapped (`anon` keeps counting them) on the INACTIVE FILE list, and the kernel takes them
|
||||||
|
back first. Measured: Wanderer's Meilisearch read `anon` 83–96 % of its 384 MiB limit on the bench AND on 9202 (with
|
||||||
|
or without swap, with or without `MEILI_MAX_INDEXING_MEMORY`), 0 kills, while `file` was ~1 MB and `inactive_file`
|
||||||
|
~290 MB; it used ~100 MB. Figure (one home: catalog `scripts/memfigure.py`): the larger of `anon − max(0,
|
||||||
|
active_file + inactive_file − file)` and `active_anon + inactive_anon − shmem`, capped at `anon` — it errs high.
|
||||||
|
Wanderer v0.20.0 → v0.21.2 re-run on the bench: `proven`, no marks (search 83.3 % anon, **27.7 % live**); 9202:
|
||||||
|
95.2 % anon, **25.1 % live**. The evening's „17.7 % with the cap" was a different container (the one after the
|
||||||
|
swap, started over existing data), not the cap's effect. The catalog change (and Wanderer's entry without
|
||||||
|
`memory_tight`) is prepared, not pushed — the morning list. `audits/night-2026-10-28/r1338-meili/`.
|
||||||
|
|
||||||
23. **The push-time test-record gate asks the registry — for moved refs only** — *decided by CC
|
23. **The push-time test-record gate asks the registry — for moved refs only** — *decided by CC
|
||||||
unattended 2026-09-23 night — operator may reverse.* *One sentence:* may a `--fast` (hook) gate use
|
unattended 2026-09-23 night — operator may reverse.* *One sentence:* may a `--fast` (hook) gate use
|
||||||
@@ -421,7 +450,8 @@ R-636's louder repeated alarm.
|
|||||||
(a) broke the second. Shipped as controller **v0.269.1** — a second release in the session, instead of
|
(a) broke the second. Shipped as controller **v0.269.1** — a second release in the session, instead of
|
||||||
shipping (a) to the fleet with the floor.
|
shipping (a) to the fleet with the floor.
|
||||||
*Note 2026-09-30 (R-740) — the cost line above is not what the code does.* „(or the automatic leg)": the
|
*Note 2026-09-30 (R-740) — the cost line above is not what the code does.* „(or the automatic leg)": the
|
||||||
automatic leg never presses a digest-only change (`unattended.go:433–435`, skip `no_test_record`). And the
|
automatic leg never presses a digest-only change (`unattended.go:433–435` then, `:449-451` now — line corrected
|
||||||
|
2026-10-24 — skip `no_test_record`). And the
|
||||||
catalog never writes a re-test of a floating tag at a new digest (the writer refuses a step that moves no
|
catalog never writes a re-test of a floating tag at a new digest (the writer refuses a step that moves no
|
||||||
image), so a same-name upstream fix reaches no box by either route today. Measured, with the options, in
|
image), so a same-name upstream fix reaches no box by either route today. Measured, with the options, in
|
||||||
R-740; the decision is the operator's (STATUS 2026-09-30). This decision's ruling — the sync never moves
|
R-740; the decision is the operator's (STATUS 2026-09-30). This decision's ruling — the sync never moves
|
||||||
@@ -976,8 +1006,11 @@ its length, and both fixes cost something the household would notice — operato
|
|||||||
installs the vouched golden and updates itself to the floor (not yet proven end to end — R-1052). **The golden
|
installs the vouched golden and updates itself to the floor (not yet proven end to end — R-1052). **The golden
|
||||||
stays weekly:** a CHANGED golden selection is still refused when it is older than the fleet's newest controller
|
stays weekly:** a CHANGED golden selection is still refused when it is older than the fleet's newest controller
|
||||||
(`TestVouchGoldenChangedToOlderIsStillRefused`).
|
(`TestVouchGoldenChangedToOlderIsStillRefused`).
|
||||||
5. The push gate judges the golden by its AGE (older than 7 days while the floor is above it), not by the newest
|
5. The push gate judges the golden by its AGE, not by the newest CHANGELOG heading
|
||||||
CHANGELOG heading (`scripts/golden_currency_gate.py`).
|
(`scripts/golden_currency_gate.py`). **[DESIGN — corrected 2026-10-25, D45 = all picks, R-1266]** ~~(older than 7 days while the floor is above
|
||||||
|
it)~~ The gate runs offline and cannot read the floor, so „a release is ahead of the newest bake” stands in for
|
||||||
|
„the floor is above it”: it convicts when a release is ahead AND the newest bake is more than 7 days old; ahead
|
||||||
|
and at most 7 days old is an advisory. The floor follows a release within hours, so the gate is the stricter one.
|
||||||
251. **Scripts use scoped hub tokens, not the operator password; CC's floor step uses its own.** *Operator ruling
|
251. **Scripts use scoped hub tokens, not the operator password; CC's floor step uses its own.** *Operator ruling
|
||||||
2026-10-15 11:38.* Tokens with groups (read/write per group), managed on the Configuration tab, shown once, stored
|
2026-10-15 11:38.* Tokens with groups (read/write per group), managed on the Configuration tab, shown once, stored
|
||||||
hashed, every use logged by name (`05` §17, hub v0.152.0). Never by any token: a secret shown or minted, a delete or
|
hashed, every use logged by name (`05` §17, hub v0.152.0). Never by any token: a secret shown or minted, a delete or
|
||||||
@@ -1031,6 +1064,284 @@ its length, and both fixes cost something the household would notice — operato
|
|||||||
Built in controller v0.314.0 (`internal/stacks/restore_gate.go`); pinned by `restore_gate_test.go` and
|
Built in controller v0.314.0 (`internal/stacks/restore_gate.go`); pinned by `restore_gate_test.go` and
|
||||||
`r1057_unit_gate_test.go` (ten mutations each seen failing). Proof: `audits/gate-restore-2026-10-15/9202/`.
|
`r1057_unit_gate_test.go` (ten mutations each seen failing). Proof: `audits/gate-restore-2026-10-15/9202/`.
|
||||||
|
|
||||||
|
### 2026-10-23 (~12:00) — D39 = B: the security-fix step (operator ruling by starting the run; R-1239)
|
||||||
|
|
||||||
|
255. **An urgent upstream security fix reaches the boxes at night as a `security_fix` ladder step: the box walk is
|
||||||
|
required, the bench is optional, and a public notice is the authority.** *Operator ruling 2026-10-23 ~12:00 (D39 =
|
||||||
|
B, R-1239), taken by starting the D39 brief; its design is Claude.ai's recommendation, taken with it.* Why: on
|
||||||
|
2026-10-23 the only route for the Plex2Shell fix was an operator word, `git push --no-verify` past the move gate,
|
||||||
|
and a hand press on each box — and the night leg never takes a step with no test record (R-1238). That does not
|
||||||
|
scale past the first customers and needs the operator awake.
|
||||||
|
1. **The entry** (`scripts/ladder.py` `check_security`, the one reader): `"verdict": "security_fix"` with `security`
|
||||||
|
`{ids: [CVE-…|GHSA-…], advisory: "https://…", fixed_in}`, `box_evidence` (required), `seed` `read_back`|`none`,
|
||||||
|
`memory_peak_pct` from the BOX soak with `memory_basis: "box_soak"`; `evidence` = the bench record or null — a
|
||||||
|
bench given must be `proven` (`bench_verdict`, `harness_version` ≥ 2 ride with it). Never `backfilled`,
|
||||||
|
`digest_from` (a same-tag fix stays decision 52's lane), `engine_conversion`, nor a database engine MAJOR (any
|
||||||
|
engine — MariaDB included; the judgement is `check-engine-major.py`'s own `engine_of`/`major_of`, imported).
|
||||||
|
2. **No operator word per incident.** The session that writes the step reports it to the operator after the push.
|
||||||
|
3. **The gates:** the move gate accepts a `security_fix` move exactly as a proven one (from/to per service, digests
|
||||||
|
the registry still serves); `check-test-record.py` WARNs on every push while a template's head is a security-fix
|
||||||
|
step with no bench test („<app>: security-fix step <to> has no bench test (since <tested_at>)"). Seven decoys
|
||||||
|
(`test_gate_decoys.py`), each seen failing with the rules switched off.
|
||||||
|
4. **The box venue:** `box_walk.py --security` on 9202 through the product's Update; verdict `proven` only if the
|
||||||
|
update ended `done` with no undo and no hold, the app runs, the four version observables are TO, the seed (if
|
||||||
|
any) reads back, and a soak (default 600 s, cgroup `anon` against the limit, OOM kills and restarts counted)
|
||||||
|
sees no kill and no restart. A negative control (a TO image that exits at once) comes back `failed`.
|
||||||
|
5. **The writer:** `upgrade-test.py --write-ladder --security …`, refusing a box walk not proven, a bench given and
|
||||||
|
not proven, a template not at the walk's FROM, a digest that moved since the walk, an engine major, a missing
|
||||||
|
field. The compose move, `catalog_since` and the superseded step's own files are the shared `apply_entry`.
|
||||||
|
6. **The controller (v0.341.0):** the leg takes `proven` OR `security_fix`; every other guard runs unchanged, in
|
||||||
|
the same order; it logs `[update-leg] <app>: security fix <ids> taken (<from> → <to>)` and the hub report's
|
||||||
|
step carries the ids. A person's press and the badge are unchanged. **A controller ≤ 0.340.0 skips the step at
|
||||||
|
night (`not_proven`) and a hand press takes it** — proven on 9202, below.
|
||||||
|
7. **The trade, stated.** A security-fix step skipped the bench. For an app with a box fixture the data is still
|
||||||
|
read back after the update; for an app without one (Plex today) the step is checked only by „it starts, it is
|
||||||
|
healthy, it runs the new version, and a 10-minute soak sees no kill or restart". `files_may_change` is a bench
|
||||||
|
measurement: with no bench the step carries `false`. That is accepted because the leg's own guard is not the only
|
||||||
|
net — the update takes a per-app backup first, the undo puts the pre-update data back on a failed health check
|
||||||
|
(decision 15), and a security fix that rewrites files without breaking the health check is the case the bench
|
||||||
|
would catch and this lane accepts the risk of. A session that knows a fix migrates files runs the bench.
|
||||||
|
**Proof (9202, drill catalog, 2026-10-23):** `audits/security-lane-2026-10-23/` — the negative control (TO
|
||||||
|
`busybox:1.36.1`) came back `failed` (the box undid it after 335 s) and the writer refused it; Plex 1.43.4.10903 →
|
||||||
|
1.43.5.11029 `proven` (done in 36.9 s, the four observables at TO, digest `sha256:01bdf0ad…5c5f`, soak 600 s: anon
|
||||||
|
peak 10.7 % of 2 GiB, 0 kills, 0 restarts, `seed: none`); the writer wrote the drill step (CVE-2026-96652, -96654,
|
||||||
|
-96655, -96656; Plex's own notice as CISA KEV cites it, added 2026-10-20; fixed in 1.43.3.10861); on 0.340.0 the
|
||||||
|
night leg logged `plex: skipped — not_proven` and a hand press took the step; on 0.341.0-rc.1 the night leg logged
|
||||||
|
`plex: security fix CVE-2026-96652,CVE-2026-96654,CVE-2026-96655,CVE-2026-96656 taken` and ended `done` in 30.1 s
|
||||||
|
after the update's second-drive copy and undo copy. Released as controller 0.341.0, global floor 11:24 UTC.
|
||||||
|
**Batch use, 2026-10-27 (the lane's first live use, `audits/security-app-updates-2026-10-27/`):** the security
|
||||||
|
watch (R-1303) flagged 38 pins; a hand triage put 15 apps in the lane. One negative control first (TO
|
||||||
|
`busybox:1.36.1` → `failed`, undone), then walks on 9202 three at a time, each written to the drill, then **a STOP
|
||||||
|
per batch of about five** before the live catalog: batch 1 `06e603e` (rallly, romm, code-server, outline, vikunja),
|
||||||
|
batch 2 `7b0071e` (n8n, privatebin, docmost, grafana), batch 3 `65a6b11` (homebox, mealie, audiobookshelf,
|
||||||
|
bentopdf, uptime-kuma) — 14 steps, 12 with the seed read back after the update, 2 `seed: none` (code-server: no
|
||||||
|
script route; bentopdf: no data). Two notices with no CVE/GHSA id (wishlist's critical account takeover,
|
||||||
|
bookstack's two leaks) could not use the lane and went as NORMAL steps (bench 9401 + box) in the same batches.
|
||||||
|
**What the batch use taught:** (a) a walk can be `inconclusive` because the FIXTURE is stale, not the step — four
|
||||||
|
were (Rallly 4.15's code-only sign-in, Jellyfin 10.11's wizard, Homebox 0.26's entities API, Mealie's generated
|
||||||
|
admin password); each was fixed and the walk run again; (b) three walks at a time on 9202 (25 GiB) kept every soak
|
||||||
|
clean; (c) a one-step-per-app lane stops below a fix that needs a NEW main version — those stay big steps with a
|
||||||
|
row each (R-1311, R-1313–R-1322).
|
||||||
|
|
||||||
|
### 2026-10-26 (night brief) — D55: demo-hp's one clean-up window gets a cap of 260
|
||||||
|
|
||||||
|
268. **demo-hp's one-shot clean-up window is granted with a raised cap of 260, not 220 (D55 = 260).** *Operator,
|
||||||
|
2026-10-26 night.* Options: A) 260 — cost: one window may remove up to 260 old copies on a test box (the box still
|
||||||
|
removes only what its normal keep rule selects, and the fake-copy guard still runs); B) 220 as first ruled — cost:
|
||||||
|
the plan grows ~14 a night (209 on 2026-10-26, ~223 at the next run), so it is refused again. Why A: the cap must sit
|
||||||
|
above the honest plan on the night it is used. CC's token lacks `offsite:write`, so the operator made a one-purpose
|
||||||
|
token `cc-d55-window` (Off-site write only); CC used it once, 2026-10-26 19:10:31Z (`max:260` stored, read back).
|
||||||
|
**The window ran** 2026-10-27 03:18:15–03:18:41Z (window 17): 223 of 223 planned removed, 373 → 150, closed by the
|
||||||
|
box, no alarm (hub row + the box's own log agree). The operator revokes the token.
|
||||||
|
|
||||||
|
### 2026-10-27 (evening, operator 21:33) — D60: CC never deletes from the secret store; D56–D59 = CC's picks
|
||||||
|
|
||||||
|
271. **CC never deletes from the `sec` store, with or without the operator's yes (D60).** *Operator, 2026-10-27 21:33.*
|
||||||
|
`sec rm` is the operator's command, run in his own terminal. Why: one purpose of the store is that CC cannot delete
|
||||||
|
by mistake (the old credentials file was wiped once). Made mechanical the same evening: cc-secrets 0.4.0 (`092e282`,
|
||||||
|
hardened `27fd806`) refuses `sec rm` in every form and any removal or overwrite of the store file or the age key
|
||||||
|
(Bash and Write/Edit guards), proven in a new CC session. The 46 names removed earlier that day by CC (on the
|
||||||
|
operator's yes, before this ruling) can come back by `git revert` of their `rm NAME` commits — the operator's act.
|
||||||
|
|
||||||
|
272. **D56 = keep SparkyFitness offered** (R-785) while its two flaws have no fix — both need a family account or a leaked
|
||||||
|
photo link; the fix rides its big step. *Operator, 2026-10-27 21:33 (CC's pick).*
|
||||||
|
|
||||||
|
273. **D57 = keep Gitea's repository import on** (R-1315) — sign-up is closed; judged again in its big step. *Operator,
|
||||||
|
2026-10-27 21:33 (CC's pick).*
|
||||||
|
|
||||||
|
274. **D58 = Termix waits for its big step** (R-1311); the forged-address header is NOT stripped now — stripping it lets
|
||||||
|
a stranger lock the family out (R-775). *Operator, 2026-10-27 21:33 (CC's pick).*
|
||||||
|
|
||||||
|
275. **D59 = hide Wanderer from new installs until its big step is done** (R-1313) — any server on the internet can
|
||||||
|
overwrite its records with no login; no box runs it; an installed copy is not touched. *Operator, 2026-10-27 21:33
|
||||||
|
(CC's pick).*
|
||||||
|
|
||||||
|
276. **A Home Assistant monthly release (2026.9 → 2026.10) is the SAME main version for the ladder, so a security fix
|
||||||
|
may ride the lane.** *Decided by CC unattended 2026-10-27 — operator may reverse.* Options: A) the lane — the one
|
||||||
|
comparator (`stacks.CompareImageRefs`, Q3) parses „2026" as the major, a notice id exists (CVE-2026-111497, Python
|
||||||
|
Brotli 1.2.0 → 1.2.1 inside the image); cost: a monthly HA release carries feature changes into the night lane;
|
||||||
|
B) a normal step — cost: a second rule beside the comparator for one app's version shape (Q3 warns that is how a
|
||||||
|
major gets automated by accident). Why A: the brief („a normal step if it counts as a main version in our ladder
|
||||||
|
rules, else the lane") and the one-comparator rule; the box walk + soak is the guard either way.
|
||||||
|
|
||||||
|
### 2026-10-28 (evening, attended) — D57 judged again in Gitea's big step: imports from three hosts only
|
||||||
|
|
||||||
|
280. **Gitea's repository import is limited to github.com, codeberg.org and gitlab.com** (`[migrations] EGRESS_MODE=strict`,
|
||||||
|
`ALLOWED_HOST_LIST`, Gitea 28's own egress rules) — this replaces D57's „keep import on, unrestricted" (decision 273).
|
||||||
|
Measured on 9202 with 28.2.0: the default (`lax`) already refuses home-network and Docker addresses (422); the
|
||||||
|
allow-list also refuses every other public host (gitea.com 422) and lets GitHub import (201). Options: the allow-list
|
||||||
|
(cost: an import from a self-hosted forge or Bitbucket is refused) or 28's default (cost: the box fetches any public
|
||||||
|
address a family member names). *Operator, 2026-10-28, at the Gitea STOP; CC's pick.*
|
||||||
|
|
||||||
|
### 2026-10-28 (afternoon, attended) — the four big steps; D59 ends; Wanderer's inbound federation stays off
|
||||||
|
|
||||||
|
279. **D59 ends: Wanderer is offered again** after its big step v0.20.0 → v0.21.2 went live (catalog `1825247`), and
|
||||||
|
**its inbound federation stays OFF** — the template sets no `POCKETBASE_PROXY_SECRET`, so v0.21 rejects every
|
||||||
|
inbound ActivityPub delivery (fail closed, upstream's docs). Options: off (a family instance takes nothing from
|
||||||
|
strangers' servers; the route GHSA-fprh abused stays shut) or on (a generated shared secret; other servers deliver
|
||||||
|
follows, comments, updates). *Operator, 2026-10-28, at the Wanderer STOP; CC's pick both times.*
|
||||||
|
|
||||||
|
**[DESIGN] A pin move needs its step — also when no box runs the app (2026-10-28, stated here because the brief asked
|
||||||
|
for „the rule for a pin change when no box runs the old version" and this document did not say it).** The rule lives in
|
||||||
|
the catalog: `app-catalog-felhom.eu/CLAUDE.md` („An `image:` move needs its TEST RECORD") and its gates —
|
||||||
|
`check-test-record.py` (the newest ladder entry IS the compose) and `check-test-record-move.py` (a moved ref must add a
|
||||||
|
proven entry). There is no exception for an app no box runs: the writer (`upgrade-test.py --write-ladder`) refuses
|
||||||
|
unless the bench AND the box walk say `proven`, so every big step is a normal ladder step. A big step's UNDO is proven
|
||||||
|
on 9202 as well (the 2026-10-27 Jellyfin method, generalised in `audits/dockerhub-bigsteps-2026-10-28/undo_proof2.py`):
|
||||||
|
TO with failing checks must end `undone` and the seed must read back on FROM. **The broken probe must fail on any app:**
|
||||||
|
a single-page app answers 200 on every path, so a „no such path" probe passed and Termix's update ended `done`; the
|
||||||
|
tool now probes port 1. `test-record-move` also refuses a `memory_tight` step whose range does not raise the limit —
|
||||||
|
Wanderer's Meilisearch went 256 → 384 MB and was measured again (it sizes itself to its limit: 88.8 %).
|
||||||
|
|
||||||
|
### 2026-10-29 (morning, attended) — D62: an unused restore copy stays 30 days, and the household may delete it; D63: prove or revert
|
||||||
|
|
||||||
|
281. **D62 = an unused restore copy is kept 30 days, and the household can delete it earlier** with one button
|
||||||
|
(„Törlés most") in the kept-copies list on the restore page (place A), after a warning that names the app and the
|
||||||
|
restore's date and the app's name typed. A copy a hold or a pending put-back needs is never deleted — not by the
|
||||||
|
sweep, not by the button (it is off, with the reason in one line); a delete while a backup or restore runs is refused.
|
||||||
|
One log line and one operator event (`restore_copy_deleted`: who, app, date, size) per delete. Replaces the night's
|
||||||
|
14 days (decided by CC unattended). *Operator, 2026-10-29 08:17.* `07` §6.3.
|
||||||
|
282. **D63 = R-1256 H4 and H5's failed-stop branch are released only with a live proof; else they are reverted** (their
|
||||||
|
tests stay, skipped with the reason) before the controller tag. The proof uses a fault switch that exists only in a
|
||||||
|
`-tags faultinject` test build — never in the release image (`TestD63_TheReleaseBuildHasNoFaultSwitch`). *Operator,
|
||||||
|
2026-10-29 08:17.* `07` §6.3.
|
||||||
|
|
||||||
|
### 2026-10-29 (15:50, operator) — D65 = A: the hub may reboot a guest
|
||||||
|
|
||||||
|
284. **D65 = A — `reboot_guest` joins the AGENT's closed operator-action list, with no signature.** The operator reboots
|
||||||
|
one guest from the hub's host page (`pct reboot` of THAT guest, the existing guest-reboot path — never the host),
|
||||||
|
for a stuck Docker or a stuck network inside the box that „Restart controller" cannot reach. A reboot destroys
|
||||||
|
nothing (`03`, „healing … non-destructive by construction"). Guards on the agent: no backup of any tier, restore,
|
||||||
|
update, controller swap or other heavy job (the controller's busy file included); not a locked, stopped or parked
|
||||||
|
guest; at most one per hour per guest; the restart brake counts it. A press the agent did not pick up in 30 minutes
|
||||||
|
expires. The household's dashboard says for a day that Felhom support restarted the box. `04` §6.1, `03`.
|
||||||
|
*Operator, 2026-10-29 15:50.*
|
||||||
|
|
||||||
|
### 2026-10-29 (12:10, operator) — D64 = B: the hub may clear any hold
|
||||||
|
|
||||||
|
283. **D64 = B — the hub's „Clear hold" works for every hold reason, with no operator signature.** The operator accepted
|
||||||
|
that a compromised hub could clear a data-protecting hold (an app could start on half-restored data); the confirm
|
||||||
|
step shows the hold's reason and, for a restore or put-back hold, that warning before the press. R-379 now reads
|
||||||
|
„a person — on the box, or the operator on the hub". Built as the controller's `clear_hold` operator action
|
||||||
|
(`04` §6.1), cleared by the running controller (R-1343). *Operator, 2026-10-29 12:10.*
|
||||||
|
|
||||||
|
### 2026-10-28 (day, attended) — RomM kept; the runtime cache fills itself in CI
|
||||||
|
|
||||||
|
277. **D61 = keep RomM offered** (R-1330) while no RomM release carries Node ≥ 24.22.1; the first one that does is taken
|
||||||
|
as a lane step (CVE-2026-111497). No Node process runs in RomM (gunicorn, rq and nginx — `ps` on demo-hp 9201) and the
|
||||||
|
setup gate is in front; the watch flags it every night until then. *Operator, 2026-10-28 (starting the day brief),
|
||||||
|
CC's pick.*
|
||||||
|
|
||||||
|
278. **The watch reads zstd layers in CI with a hash-pinned `zstandard` wheel, not with `zstd` in the CI runner image.**
|
||||||
|
*CC, 2026-10-28, the brief's choice („say which and why").* Options: A) the wheel (0.25.0, PyPI's sha256 per
|
||||||
|
Python/libc, checked on every open, fetched into the runner's cache) — cost: a download from PyPI when the cache is
|
||||||
|
empty, and a pin to renew when the runner's Python moves; B) `zstd` in `felhom-act-runner` — cost: a homelab-repo
|
||||||
|
change, an image rebuild and a k3s rollout on DooPlex, which the brief forbids, and every CI job depends on that
|
||||||
|
image. Why A: no DooPlex change, and a wrong or missing wheel is NOT CHECKED out loud, never a pass (`08` §11).
|
||||||
|
|
||||||
|
**The lane batch of 2026-10-28 (R-1332), how it ran — the method, not a new rule.** Twenty apps carried a bad runtime
|
||||||
|
(the watch's first full read). For each pin the registry's tags in the pinned tag's own shape were listed, and the newest
|
||||||
|
one in the same main version and the newest overall were READ FROM THE IMAGE (`image_runtime.py`), never from release
|
||||||
|
notes (`audits/runtime-fixes-2026-10-28/TABLE.md`). Three states: a fixed release in the same main version → a
|
||||||
|
security-fix step; a fix only in a new main version → the app's big-step row carries it; no fixed release → a row with
|
||||||
|
the exposure. The main version is the tag's first number, and for a `0.x` app the minor (Homebox 0.26 → 0.27 on
|
||||||
|
2026-10-27; Karakeep 0.33 → 0.34 today). Batches of about five, each walked on 9202 (`box_walk.py --security`, soak
|
||||||
|
600 s, the seeded data read back), the writer run on the live catalog, `catalog_gates.py --fast`, and a STOP before each
|
||||||
|
push. None of the eight apps has a bench fixture, so no bench ran (§6.4, the lane's stated trade). Two tool gaps found
|
||||||
|
on the way were fixed in the session: the drill mover moved two services that share one image only half (Dawarich), and
|
||||||
|
a drill commit made outside a walk (the revert to FROM) never reached the box's mirror before the install.
|
||||||
|
|
||||||
|
### 2026-10-27 (attended release block) — the line-test hold is fixed the same day; R-1218's live proof is waived
|
||||||
|
|
||||||
|
269. **The line-test hold waits until the held app has really slowed down — fixed in the controller the same day.**
|
||||||
|
*Operator, 2026-10-27 12:30 CET.* Options: A) fix in the controller now, prove it on an rc on demo-hp's 9201 while the
|
||||||
|
seed runs, ride today's 0.346.0 — cost: ~1.5 h, the release later in the afternoon; B) file it and fix later — cost:
|
||||||
|
a box that seeds all day keeps getting no weekly test. Why A: measured four refusals in an hour (qBittorrent sends
|
||||||
|
~14 s after its limit falls; the agent reads 5–15 s after the call). Built `710ceac`; live on `0.346.0-rc.1`: 94.5
|
||||||
|
Mbit/s, the limit back exactly (`audits/linetest-2026-10-27/`).
|
||||||
|
|
||||||
|
270. **R-1218 is released without its live proof (rule 2 waived for this row).** *Operator, 2026-10-27 15:15 CET.*
|
||||||
|
Options: A) keep both halves, waive the live proof — cost: released key code proven by red tests only; a box that
|
||||||
|
lost its key file entirely now gets a 409 on its ceremony instead of a silent demotion; B) revert both halves — cost:
|
||||||
|
a hub 0.173.1 and a second deploy, and the demotion bug stays. Why A: no product path makes the box's saved off-site
|
||||||
|
target „off" on a hub box (the hub page's switch did not reach Tester 1's target; the household's clear is refused on
|
||||||
|
a hub tier), so no live run can tell the old code from the new; the change only widens a safety step and adds a
|
||||||
|
refusal. R-1312 asks what the hub switch does on the box.
|
||||||
|
|
||||||
|
### 2026-10-26 (evening brief) — D53: held apps reach the operator; D54: demo-hp's stuck clean-up; R-1297 (B): a second CI runner
|
||||||
|
|
||||||
|
265. **A held app reaches the OPERATOR first, never the household (D53 = operator-only reminders first).** *Operator,
|
||||||
|
2026-10-26 evening (starting the brief is the yes).* Options: A) operator only — cost: a household whose app is held
|
||||||
|
still learns only from the app's page; B) the operator and, for the two holds a household can clear, the household
|
||||||
|
too — cost: two more customer mails to write and translate, about a stop they may not understand. Why A: the
|
||||||
|
operator can act on every reason; household mails can follow once the operator list has run. Built from `08` §10.
|
||||||
|
266. **demo-hp's off-site store gets ONE clean-up window with a raised cap of 220, through the hub's existing one-shot
|
||||||
|
grant; the cap rule itself is unchanged (D54 = prune once).** *Operator, 2026-10-26 evening.* Options: A) a one-shot
|
||||||
|
grant — cost: one window prunes 209 snapshots at once on a test box (the plan was read honest, R-1294); B) change the
|
||||||
|
cap rule — cost: hub and controller code, and a weaker guard against a mass deletion on every box. Why A: the guard
|
||||||
|
stays as strong as it is; the grant expires after one use.
|
||||||
|
267. **CI gets a second runner on DooPlex; both runners have 2 CPUs (R-1297 B).** *Done by the operator, 2026-10-26.*
|
||||||
|
Options: A) a second runner — cost: DooPlex CPU while two jobs run; B) only trim the suites — cost: every repo's job
|
||||||
|
still queues behind felhom.eu's run. Why both (the operator's „both"): the trim makes one run short, the runner stops
|
||||||
|
the queue. The trim (A) is CC's and keeps every check (path filters, a tag run and a nightly run).
|
||||||
|
|
||||||
|
### 2026-10-26 (attended release run) — D51: Kimai's hold; D52: the missed weekly golden
|
||||||
|
|
||||||
|
263. **demo-hp's Kimai, held by the crash-loop guard since the 2026-10-14 power drill, is started again, not removed
|
||||||
|
(D51 = Start).** *Operator, 2026-10-26 07:57.* Options: A) Start — cost: if it crash-loops again it is held again;
|
||||||
|
B) remove it through the product — cost: the test app and its data are gone. Why A: the household's Start is the
|
||||||
|
designed way to lift this kind of hold (decision 28), and it is a test box. Result: hold lifted, healthy (R-1286).
|
||||||
|
264. **The missed Sunday bake is made up the same day, on the new controller, and the waiver covers that one day only
|
||||||
|
(D52 = bake today on v0.344.0).** *Operator, 2026-10-26.* Options: A) renew the waiver to the next Sunday — cost: a
|
||||||
|
week of new installs on a golden 13 releases behind; B) bake today — cost: about an hour of the release day. Why B:
|
||||||
|
the gate goes green by a fresh golden, not by a waiver. The weekly Sunday bake (`runbooks/nightly-rotation.md`)
|
||||||
|
stays the rule; every Sunday-night brief names it.
|
||||||
|
|
||||||
|
### 2026-10-25 (evening, attended) — D50: which agent packages the registry keeps
|
||||||
|
|
||||||
|
262. **The registry keeps the newest 5 released agent versions, plus the hub's MinAgent floor, plus the version the hub's
|
||||||
|
day-0 manifest names; CI asserts exactly these (D50 = A).** *Operator ruling 2026-10-25 evening.* Options: A) this
|
||||||
|
— cost: a box on a version outside the set cannot fetch its own binary again (none is, on the day); B) the newest 10,
|
||||||
|
never pruned by hand — cost: today's four are gone, so CI stays red until ten new versions exist. Why A: it is what
|
||||||
|
the operator prunes to, and the two pinned versions are the only old ones anything still asks for. Built: agent
|
||||||
|
`scripts/retention-policy.json` (`generic_versions_kept` 5, `pinned`) + `check-published-versions.py`; the pins are
|
||||||
|
copies of hub state, updated by the session that records a vouch or a new MinAgent (R-1290).
|
||||||
|
|
||||||
|
### 2026-10-25 (attended release run) — D49: a restore that does not start keeps both copies; D44–D46: the review picks
|
||||||
|
|
||||||
|
258. **A restore whose app does not start keeps the restored data AND the old copy, tells the operator, and never puts the
|
||||||
|
old data back by itself (D49 = A).** *Operator ruling 2026-10-25, given by starting the attended release run.* Options:
|
||||||
|
A) keep and wait — cost: an app that never starts needs a person; B) pour the old data back after the wait — cost: a
|
||||||
|
slow but correct first start (a database migration) is thrown away. Why A: the slow first start is the common case,
|
||||||
|
the old copy is kept, a late start turns the record into a success. Built: `07` §8.2e and fix A (the verdict needs a
|
||||||
|
stable start; the copy goes only after 3 more minutes of it) — controller 0.343.0.
|
||||||
|
259. **The 21 confirm lines and four code changes of the second review round are accepted as picked (D44 = all picks).**
|
||||||
|
*Operator, 2026-10-25.* The notes carry a normal `[DESIGN — confirmed …]`; the code changes became R-1280–R-1283
|
||||||
|
(R-1265 closed).
|
||||||
|
260. **Where a design document and the code disagree on the seven named points, the text follows the code (D45 = all
|
||||||
|
picks).** *Operator, 2026-10-25.* Each corrected place is tagged `[DESIGN — corrected 2026-10-25, D45 = all picks,
|
||||||
|
R-1266]`, the old words struck through, not deleted (R-1266 closed).
|
||||||
|
261. **Homepage is behind the family gate, like Glance (D46 = yes).** *Operator, 2026-10-25.* Why: it has no login of its
|
||||||
|
own and showed a stranger the box's CPU, memory and disk. Tried on the drill first; catalog `ebbf1f4` (R-1272 closed).
|
||||||
|
|
||||||
|
### 2026-10-25 (08:38) — D47: nothing changes an existing backup; D48: a restore downloads first (operator rulings by starting the run; R-1269)
|
||||||
|
|
||||||
|
256. **An existing recovery unit is never changed in part (D47).** A unit's definition (compose, pins, `app.yaml`) and
|
||||||
|
its data are written only together, by a backup of the same install. A new install never adopts, rewrites or
|
||||||
|
deletes a unit that an earlier install left; that unit is set aside whole and stays restorable. *Operator ruling
|
||||||
|
2026-10-25 08:38, taken by starting the R-1269 brief.* Why: on 2026-10-24 the catalog sweep measured three times
|
||||||
|
that a reinstall's own five-minute refresh wrote the new install's database password into the unit „remove — keep
|
||||||
|
backups" had kept; a restore then could not open the old data (R-1269). Built: every install has an identity
|
||||||
|
(`install_id`), the unit records it, and every writer asks first — `07` §8.2a „An earlier install's unit is set
|
||||||
|
aside" (controller, unreleased; rides v0.343.0).
|
||||||
|
257. **Every restore path resolves and pulls every image its definition names before the first change to the app; a
|
||||||
|
failed pull is a refusal with the app untouched (D48).** *Operator ruling 2026-10-25 08:38, taken by starting the
|
||||||
|
brief.* Why: the restore stopped the app and moved its data first and pulled only inside the database-only start;
|
||||||
|
a version that could not be downloaded then ended in the D22 put-back, a full stop-and-pour-back for nothing.
|
||||||
|
Built: `07` §6.6 „The limit — the image is not in the backup" (controller, unreleased; rides v0.343.0).
|
||||||
|
**Confirmed, no change:** `07` §6.6 — a restore brings the app back at its data's version, then the guarded update
|
||||||
|
climbs the ladder; R-698 (a deleted version cannot come back) and R-933 (our own image mirror) are unchanged.
|
||||||
|
|
||||||
### 2026-10-15 (07:48–08:20) — bake first; Plex through its own key; HSTS one day (recorded before the media build)
|
### 2026-10-15 (07:48–08:20) — bake first; Plex through its own key; HSTS one day (recorded before the media build)
|
||||||
|
|
||||||
247. ~~**Bake first.** A controller version is RELEASED only when its golden is baked.~~ **REVERSED by decision 250,
|
247. ~~**Bake first.** A controller version is RELEASED only when its golden is baked.~~ **REVERSED by decision 250,
|
||||||
@@ -1932,7 +2243,7 @@ moves a stack dir.
|
|||||||
| writer | pin source |
|
| writer | pin source |
|
||||||
|---|---|
|
|---|---|
|
||||||
| the deploy path (`runComposeDeploy`) | the template just deployed from |
|
| the deploy path (`runComposeDeploy`) | the template just deployed from |
|
||||||
| **`UpdateStack`** | the catalog's current template, written **BEFORE** the pull |
|
| **`UpdateStack`** | the catalog's current template, written **BEFORE** the pull — *as built: `UpdateStack` was deleted in v0.237.0; the guarded update writes the pin (`advancePinTo`, the `pinning` phase before `pulling`), and two more paths move a pin BACK — `pinBack` and the undo's `restoreDefinition` (`internal/stacks/update.go`, `undo.go`; noted 2026-10-24)* |
|
||||||
| the restore (`stackAdapter.RecreateStackDefinitionFromUnit`) | the recovery unit's captured compose — **this closes R-441** |
|
| the restore (`stackAdapter.RecreateStackDefinitionFromUnit`) | the recovery unit's captured compose — **this closes R-441** |
|
||||||
| `Manager.AdoptPins` | the observation, once, and only when complete AND matching |
|
| `Manager.AdoptPins` | the observation, once, and only when complete AND matching |
|
||||||
|
|
||||||
@@ -1998,7 +2309,7 @@ earlier feature is the failure mode to look for whenever a file changes meaning.
|
|||||||
| **3** | **The compose file becomes DERIVED** — the pin in `app.yaml` wins; the syncer renders instead of copying. | **SHIPPED, controller v0.235.0 (2026-09-06)** — operator ruling §3.4 |
|
| **3** | **The compose file becomes DERIVED** — the pin in `app.yaml` wins; the syncer renders instead of copying. | **SHIPPED, controller v0.235.0 (2026-09-06)** — operator ruling §3.4 |
|
||||||
| **4** | **A guarded update** — verified-backup precondition, abort-on-failure, and the truth at the moment of action rather than 5m16s later (R-443). | **SHIPPED + PROVEN LIVE, controller v0.237.0 (job) + v0.238.0 (page) + v0.238.1 (2026-09-13); any backup tier since v0.239.0 (§3 decision 8)** — §6.1 |
|
| **4** | **A guarded update** — verified-backup precondition, abort-on-failure, and the truth at the moment of action rather than 5m16s later (R-443). | **SHIPPED + PROVEN LIVE, controller v0.237.0 (job) + v0.238.0 (page) + v0.238.1 (2026-09-13); any backup tier since v0.239.0 (§3 decision 8)** — §6.1 |
|
||||||
| **5** | **An upgrade test that runs again** — a harness that upgrades a real app with real data in it and asks the app for the data back. | **SHIPPED, `app-catalog/scripts/upgrade-test.py` (2026-09-06)** — 7 edges, 3 apps; see §4.1 and §10 |
|
| **5** | **An upgrade test that runs again** — a harness that upgrades a real app with real data in it and asks the app for the data back. | **SHIPPED, `app-catalog/scripts/upgrade-test.py` (2026-09-06)** — 7 edges, 3 apps; see §4.1 and §10 |
|
||||||
| **6** | **Automatic updates** — ~~within a major, a human across one~~ **the catalog's tested steps, climbed one at a time, as a leg of the backup chain, undone by the box on failure** (§3 decisions 11–17); **an engine change gets its own edge.** | OPEN — R-450; rulings 2026-09-23; build order §6.4 |
|
| **6** | **Automatic updates** — ~~within a major, a human across one~~ **the catalog's tested steps, climbed one at a time, as a leg of the backup chain, undone by the box on failure** (§3 decisions 11–17); **an engine change gets its own edge.** | ~~OPEN — R-450~~ **BUILT** — the leg in `internal/stacks/unattended.go`, wired into the off-site job; R-450 CLOSED 2026-10-05 (capability map B11) *(noted 2026-10-24, review round 2)* |
|
||||||
| **7** | **A fleet sweep pipeline** — the operator can see, and move, how far behind every box is. | OPEN — R-451; ruled 2026-09-23 (decision 18), built later |
|
| **7** | **A fleet sweep pipeline** — the operator can see, and move, how far behind every box is. | OPEN — R-451; ruled 2026-09-23 (decision 18), built later |
|
||||||
|
|
||||||
### 6.1 Slice 4 as SHIPPED (controller v0.237.0 + v0.238.0, 2026-09-13)
|
### 6.1 Slice 4 as SHIPPED (controller v0.237.0 + v0.238.0, 2026-09-13)
|
||||||
@@ -2013,12 +2324,13 @@ closed by construction: nothing reports an update complete on the compose exit c
|
|||||||
| # | phase | what happens | on failure |
|
| # | phase | what happens | on failure |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| 0 | refusals (409, before the intent is recorded) | held (R-439), busy (backup/restore/app-data op/quiesce), migration, already updating, deploying, memory (the deploy's `memoryVerdict`, releasing the app's own request), disk (**fixed 2 GB floor** — image size unknown without a registry), **no copy on ANY tier and no backup can be taken now** (since v0.239.0; before it, no restorable Tier-2 copy) | nothing moves, nothing is recorded |
|
| 0 | refusals (409, before the intent is recorded) | held (R-439), busy (backup/restore/app-data op/quiesce), migration, already updating, deploying, memory (the deploy's `memoryVerdict`, releasing the app's own request), disk (**fixed 2 GB floor** — image size unknown without a registry), **no copy on ANY tier and no backup can be taken now** (since v0.239.0; before it, no restorable Tier-2 copy) | nothing moves, nothing is recorded |
|
||||||
| 1 | `checking` | walks Tier 2 → Tier 1 → Tier 3 for the first copy younger than `update.backup_max_age` (v0.239.0) | nothing moves |
|
| 1 | `checking` | walks Tier 2 → Tier 1 → Tier 3 for the first copy younger than `update.backup_max_age` (v0.239.0) — *an app whose data lives in bind-mounted files walks second drive → OFF-SITE → own unit (decision 9, R-479, v0.241.0; `internal/backup/update_guard.go` `updateTierOrderBindData`; noted 2026-10-24)* | nothing moves |
|
||||||
| 2 | `backing-up` — only when no tier holds a fresh copy | `RunAppBackupNow`: this app's DB dump → volume dump → unit capture (marked proven current) → Tier-2 copy, whose failure is a WARN since v0.239.0 | refused with the backup's own error; nothing moves |
|
| 2 | `backing-up` — only when no tier holds a fresh copy | `RunAppBackupNow`: this app's DB dump → volume dump → unit capture (marked proven current) → Tier-2 copy, whose failure is a WARN since v0.239.0 | refused with the backup's own error; nothing moves |
|
||||||
| 3 | `safety-dump` | `WriteUpdateSafetyDump` (R-361's undo copy) — **before the pin moves** | refused; nothing moves |
|
| 3 | `safety-dump` | `WriteUpdateSafetyDump` (R-361's undo copy) — **before the pin moves** | refused; nothing moves |
|
||||||
| 4 | `pinning` | the previous definition is copied aside and journaled, then the pin advances | pin put back |
|
| 4 | `pinning` | the previous definition is copied aside and journaled, then the pin advances | pin put back |
|
||||||
| 5 | `pulling` | `compose pull` | **pin and definition PUT BACK** — nothing ran (Scenario E) |
|
| 5 | `pulling` | `compose pull` | **pin and definition PUT BACK** — nothing ran (Scenario E) |
|
||||||
| 5a | `copying` **(v0.263.0)** | `compose stop`, then each NAMED volume `cp -a` into `<volume>.pre-update-<stamp>` by a helper that writes a finished-marker last (bind folders never) | copies removed, **pin put back, the old version started again** — nothing new ran |
|
| 5a | `copying` **(v0.263.0)** | `compose stop`, then each NAMED volume `cp -a` into `<volume>.pre-update-<stamp>` by a helper that writes a finished-marker last (bind folders never) | copies removed, **pin put back, the old version started again** — nothing new ran |
|
||||||
|
| 5b | `converting` **(v0.273.0, row added 2026-10-24)** | only on a step whose ladder entry carries `engine_conversion`: the box converts the PostgreSQL major (dump from the old engine, load into the new, check) — `internal/stacks/pgconvert.go` | the undo puts the old engine's data back |
|
||||||
| 6 | `starting` | `compose up -d --remove-orphans` | **UNDO** (below), HOLD only if the undo fails |
|
| 6 | `starting` | `compose up -d --remove-orphans` | **UNDO** (below), HOLD only if the undo fails |
|
||||||
| 7 | `verifying` | the `.felhom.yml` health check through the existing probe **when it resolves to a container**, else 60 s of every container running and none restarting; bounded by `update.health_timeout` | **UNDO** (below), HOLD only if the undo fails. Before v0.263.0: stop + HOLD, the pin stays (Scenario F) |
|
| 7 | `verifying` | the `.felhom.yml` health check through the existing probe **when it resolves to a container**, else 60 s of every container running and none restarting; bounded by `update.health_timeout` | **UNDO** (below), HOLD only if the undo fails. Before v0.263.0: stop + HOLD, the pin stays (Scenario F) |
|
||||||
| 7a | `undoing` **(v0.263.0)** | every copy validated (marker) BEFORE anything is poured back; volumes emptied and refilled; definition, pin and the pinned version's `.felhom.yml` record put back from the job's own copies; `up`; health with the OLD version's probe | HOLD, the sentence prefixed *„A frissítés nem sikerült, és az automatikus visszaállítás sem."* + the data state (`untouched` / `half` / `not_started`); copies kept |
|
| 7a | `undoing` **(v0.263.0)** | every copy validated (marker) BEFORE anything is poured back; volumes emptied and refilled; definition, pin and the pinned version's `.felhom.yml` record put back from the job's own copies; `up`; health with the OLD version's probe | HOLD, the sentence prefixed *„A frissítés nem sikerült, és az automatikus visszaállítás sem."* + the data state (`untouched` / `half` / `not_started`); copies kept |
|
||||||
@@ -2067,7 +2379,7 @@ the harness has proven it, is slice 6's.
|
|||||||
> copy taken before the pin moved**, so the old version meets pre-migration data. Spiked by hand on
|
> copy taken before the pin moved**, so the old version meets pre-migration data. Spiked by hand on
|
||||||
> 9202 the same day — §6.1a.
|
> 9202 the same day — §6.1a.
|
||||||
|
|
||||||
**Not gated here:** a multi-major jump (R-40). It fails health and is held honestly; stepping is slice 6.
|
**Not gated here:** a multi-major jump (R-40). It fails health and is held honestly; stepping is slice 6. *(Noted 2026-10-24: stepping is built — the ladder, one tested step per press, decision 14.)*
|
||||||
|
|
||||||
### 6.1a The undo (decision 15) — SPIKED BY HAND, then BUILT: controller v0.263.2 (2026-09-23)
|
### 6.1a The undo (decision 15) — SPIKED BY HAND, then BUILT: controller v0.263.2 (2026-09-23)
|
||||||
|
|
||||||
@@ -2081,7 +2393,8 @@ only the live box could show, fixed the same day:** the undo's probe was never a
|
|||||||
current probe held the app `unhealthy` (v0.263.1), and the "old" `.felhom.yml` taken at update time
|
current probe held the app `unhealthy` (v0.263.1), and the "old" `.felhom.yml` taken at update time
|
||||||
was already the new one, because `.felhom.yml` flows in on every catalog sync (v0.263.2: the pinned
|
was already the new one, because `.felhom.yml` flows in on every catalog sync (v0.263.2: the pinned
|
||||||
version's file is now recorded in `applied-meta/` whenever a version is pinned). **Residual (R-646):**
|
version's file is now recorded in `applied-meta/` whenever a version is pinned). **Residual (R-646):**
|
||||||
an app pinned before v0.263.2 has no such record until its next pin. **Two more, found by the chaos hour of
|
an app pinned before v0.263.2 has no such record until its next pin. *(Closed since: `BackfillAppliedMeta` records it at
|
||||||
|
start for every app CURRENT with the catalog, and names the rest — `internal/stacks/undo.go:646-652`; noted 2026-10-24.)* **Two more, found by the chaos hour of
|
||||||
2026-09-23 night (`audits/DRILL-night-2026-09-23.md` Part D):** the undo finds an app's volumes by their compose
|
2026-09-23 night (`audits/DRILL-night-2026-09-23.md` Part D):** the undo finds an app's volumes by their compose
|
||||||
label, and a restore recreates them WITHOUT it — so after any restore the undo copies nothing (R-658, P1); and a
|
label, and a restore recreates them WITHOUT it — so after any restore the undo copies nothing (R-658, P1); and a
|
||||||
held file-leg app is pointed at a restore that refuses a database-only copy, with no route left on a box without
|
held file-leg app is pointed at a restore that refuses a database-only copy, with no route left on a box without
|
||||||
@@ -2221,7 +2534,7 @@ breaks.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### 6.2 Slice 6, as it will be built (OPEN — R-450; ruled 2026-09-23, decisions 11–17)
|
### 6.2 Slice 6, as it will be built (~~OPEN — R-450~~ BUILT, R-450 closed 2026-10-05 — noted 2026-10-24; ruled 2026-09-23, decisions 11–17)
|
||||||
|
|
||||||
**The shape the rulings fix. The build order and costs are §6.4.** Rewritten 2026-09-23; the
|
**The shape the rulings fix. The build order and costs are §6.4.** Rewritten 2026-09-23; the
|
||||||
earlier "as it would be built" draft keyed on the window of Q1's recommendation and on
|
earlier "as it would be built" draft keyed on the window of Q1's recommendation and on
|
||||||
@@ -2292,7 +2605,7 @@ retried. A working caller in this exact shape exists as evidence, not product:
|
|||||||
**The per-box switch key is `app_update.unattended`**, default **true** (decision 12).
|
**The per-box switch key is `app_update.unattended`**, default **true** (decision 12).
|
||||||
|
|
||||||
⚠ **NOT `auto_update` — that name is TAKEN, and by the very thing this must not collide with.**
|
⚠ **NOT `auto_update` — that name is TAKEN, and by the very thing this must not collide with.**
|
||||||
`self_update.auto_update` / `self_update.auto_update_time` (`config/config.go` L280-281, default
|
`self_update.auto_update` / `self_update.auto_update_time` (`config/config.go` L280-281 then, L278-279 now; default
|
||||||
**04:30** at L422) are the CONTROLLER's own update. Two settings with that name, one meaning the
|
**04:30** at L422) are the CONTROLLER's own update. Two settings with that name, one meaning the
|
||||||
controller and one meaning apps, is the kind of collision that is only discovered by an operator who
|
controller and one meaning apps, is the kind of collision that is only discovered by an operator who
|
||||||
turned off the wrong one. The app-scoped key is `app_update.*`. **And `stacks.update_window` is
|
turned off the wrong one. The app-scoped key is `app_update.*`. **And `stacks.update_window` is
|
||||||
@@ -2322,12 +2635,12 @@ what the part can do to a household's data if it is wrong, not how likely that i
|
|||||||
|
|
||||||
| # | part | rulings / rows | cost | depends on | risk to customer data |
|
| # | part | rulings / rows | cost | depends on | risk to customer data |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
| **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold`: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. |
|
| **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold` *(as SHIPPED — noted 2026-10-24: the copy is the FOLDER copy decision 19 chose: every named volume `cp -a` into a sibling volume with a finished-marker, put back on failure; no database load — `internal/stacks/undo.go:26-36`; the text that follows is the design before the bake-off)*: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. |
|
||||||
| **2** | **SHIPPED — controller v0.264.0 + hub v0.120.0, proven live on 9202 and 9201 2026-09-23** (`audits/undo-fleet-2026-09-23/`). **The update sentences in the household's language** (R-606) and a mail when an automatic update is undone or held: events `app_update_undone` (warning) and `app_update_held` (error), one per app per outcome, on by default, mailed in the household's language with the app named in the subject; per-app cooldown on both legs. Leftovers: R-647. | R-606, 15 | **1** | — | none |
|
| **2** | **SHIPPED — controller v0.264.0 + hub v0.120.0, proven live on 9202 and 9201 2026-09-23** (`audits/undo-fleet-2026-09-23/`). **The update sentences in the household's language** (R-606) and a mail when an automatic update is undone or held: events `app_update_undone` (warning) and `app_update_held` (error), one per app per outcome, on by default, mailed in the household's language with the app named in the subject; per-app cooldown on both legs. Leftovers: R-647. | R-606, 15 | **1** | — | none |
|
||||||
| **3** | **SHIPPED — controller v0.264.0, proven live on 9202 2026-09-23.** **A disabled notifier says so** (R-620), so the mail of part 2 can be measured on a scratch box at all. | R-620 | **0.5** | — | none |
|
| **3** | **SHIPPED — controller v0.264.0, proven live on 9202 2026-09-23.** **A disabled notifier says so** (R-620), so the mail of part 2 can be measured on a scratch box at all. | R-620 | **0.5** | — | none |
|
||||||
| **4** | **SHIPPED — catalog `6db08a5`, night 2026-09-23** (`audits/DRILL-night-2026-09-23.md`): `update_ladder:` in `.felhom.yml`, one JSON entry per line (spiked on controller v0.266.0 and v0.267.0 first — the controller ignores the key); `check-test-record.py` (static, CI) + `check-test-record-move.py` (history + the registry for moved refs only, decision 23); the only writer `upgrade-test.py --write-ladder`; the 21 moves of 2026-09-22 backfilled from their records (21 proven). Not built: `steps/<to>.yml` (part 5 needs it once an app has two steps); `CompareImageRefs` did NOT move to the gate — the gate asks for a proven test instead, which is decision 13's own test. **The test record + the catalog gate + the memory check.** The harness writes the ladder entry (below) from its verdict record, including the memory watch's peak and marks; the gate refuses an image move with no entry, an entry with a `failed` verdict, or one with no memory watch; `CompareImageRefs`' rule moves here as the push-time safety net. **Backfill:** one entry per current pin — the 21 proven moves from their records, every other pin `needs_person: "never tested"`, which is honest and keeps them manual. A version move re-checks `mem_limit` against the watch's peak (the RomM follow-up: gate, not checklist, because the watch now produces the number). | 13, R-635 follow-up | **2.5** | the memory watch (shipped 2026-09-23) | none on a box — catalog-side only |
|
| **4** | **SHIPPED — catalog `6db08a5`, night 2026-09-23** (`audits/DRILL-night-2026-09-23.md`): `update_ladder:` in `.felhom.yml`, one JSON entry per line (spiked on controller v0.266.0 and v0.267.0 first — the controller ignores the key); `check-test-record.py` (static, CI) + `check-test-record-move.py` (history + the registry for moved refs only, decision 23); the only writer `upgrade-test.py --write-ladder`; the 21 moves of 2026-09-22 backfilled from their records (21 proven). Not built: `steps/<to>.yml` (part 5 needs it once an app has two steps); `CompareImageRefs` did NOT move to the gate — the gate asks for a proven test instead, which is decision 13's own test. **The test record + the catalog gate + the memory check.** The harness writes the ladder entry (below) from its verdict record, including the memory watch's peak and marks; the gate refuses an image move with no entry, an entry with a `failed` verdict, or one with no memory watch; `CompareImageRefs`' rule moves here as the push-time safety net. **Backfill:** one entry per current pin — the 21 proven moves from their records, every other pin `needs_person: "never tested"`, which is honest and keeps them manual. A version move re-checks `mem_limit` against the watch's peak (the RomM follow-up: gate, not checklist, because the watch now produces the number). | 13, R-635 follow-up | **2.5** | the memory watch (shipped 2026-09-23) | none on a box — catalog-side only |
|
||||||
| **5** | **SHIPPED — controller v0.268.0 (`206b035`) + catalog `5ed599c`, proven live on 9202 2026-09-24** (`audits/ladder-2026-09-24/partD/`): romm 5.3.0/11.4 → 5.3.1/11.4 (its app step, from `steps/90dd9d68258286ef.yml`) → 5.3.1/11.8 (the engine step; `mariadb-upgrade` ran) in two presses, the seeded account read back after each, the page's „Hátralévő frissítési lépések" 2 → 1 → none. Step files: `templates/<app>/steps/<StepKey(to)>.yml` (sha256 of `to` as canonical JSON, 16 hex; `stacks.StepKey` = `ladder.step_key`), refused absent or wrong by `check-test-record.py` rule 4, written by `--write-ladder` when a step is superseded, 8 backfilled from the NEWEST commit naming each step's images (romm's from `f4eb94f`, not the OOM-looping `15f9ebf`). The box reads them from its `--depth 1` clone (the whole tree is there — measured). One press = one step; a missing step file refuses before anything moves; an installed version matching no entry jumps, logged by name (measured live: vikunja 2.5.0). Limitation: a step has no `.felhom.yml` of its own (R-664). **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/<to>.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape |
|
| **5** | **SHIPPED — controller v0.268.0 (`206b035`) + catalog `5ed599c`, proven live on 9202 2026-09-24** (`audits/ladder-2026-09-24/partD/`): romm 5.3.0/11.4 → 5.3.1/11.4 (its app step, from `steps/90dd9d68258286ef.yml`) → 5.3.1/11.8 (the engine step; `mariadb-upgrade` ran) in two presses, the seeded account read back after each, the page's „Hátralévő frissítési lépések" 2 → 1 → none. Step files: `templates/<app>/steps/<StepKey(to)>.yml` (sha256 of `to` as canonical JSON, 16 hex; `stacks.StepKey` = `ladder.step_key`), refused absent or wrong by `check-test-record.py` rule 4, written by `--write-ladder` when a step is superseded, 8 backfilled from the NEWEST commit naming each step's images (romm's from `f4eb94f`, not the OOM-looping `15f9ebf`). The box reads them from its `--depth 1` clone (the whole tree is there — measured). One press = one step; a missing step file refuses before anything moves; an installed version matching no entry jumps, logged by name (measured live: vikunja 2.5.0). Limitation: a step has no `.felhom.yml` of its own (R-664). *(Closed since: a step's own `steps/<key>.felhom.yml` is read when present — `ladder.go:127-132`, v0.269.0; noted 2026-10-24.)* **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/<to>.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape |
|
||||||
| **6** | **BOTH HALVES SHIPPED. BOX HALF — controller v0.269.0 + v0.269.1, proven live on 9202 2026-09-24** (`audits/night-2026-09-24/B/`): the compose that runs pins `tag@sha256` from the ladder entry that tested those refs; pins and records stay digest-free; the badge reads behind for a newer TESTED digest; **an installed app keeps its digest until a guarded Update moves it** (v0.269.0 let the sync move it — found live, fixed in v0.269.1, `CarryDigests`); Compose accepts the form for all 25 ladder apps. **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. **The box half (compare, render `name:tag@sha256`) is not built.** **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low |
|
| **6** | **BOTH HALVES SHIPPED. BOX HALF — controller v0.269.0 + v0.269.1, proven live on 9202 2026-09-24** (`audits/night-2026-09-24/B/`): the compose that runs pins `tag@sha256` from the ladder entry that tested those refs; pins and records stay digest-free; the badge reads behind for a newer TESTED digest; **an installed app keeps its digest until a guarded Update moves it** (v0.269.0 let the sync move it — found live, fixed in v0.269.1, `CarryDigests`); Compose accepts the form for all 25 ladder apps. **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. ~~**The box half (compare, render `name:tag@sha256`) is not built.**~~ *Built — the box renders the tested digest, carries it on sync, and the badge reads the tested digest (`internal/stacks/digest.go:15-24`, `updateorder.go`; noted 2026-10-24).* **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low |
|
||||||
| **7** | **SHIPPED — controller v0.271.0 (`9cf13a3`), proven on 9202 over four simulated nights 2026-09-24/25 and watched through the demo boxes' first real automatic night (`audits/DRILL-night-2026-09-25.md`).** Decisions 31–33 taken unattended. Gaps a scratch box cannot close: R-687; no resume after a restart: R-686. *(Before:)* **SPIKED 2026-09-24 (night, Part C) — the measured build brief is §6.4.2 below.** **The update leg in the chain + the automatic caller + the switch.** A leg that starts when the off-site leg has ENDED (on every exit path, skips included), one app at a time, one step per press, `app_update.unattended` default ON (decision 12), `stacks.update_window` removed, reads `UpdateRefusal.Reason`, remembers a failed step so it never re-presses it, and the full-system backup's gate waits for it until W+5h (decision 20). | 11, 12, 14, 15, 20 | **3** | 1, 2, 5 | medium — the only part that acts with nobody watching; everything above is what makes it safe |
|
| **7** | **SHIPPED — controller v0.271.0 (`9cf13a3`), proven on 9202 over four simulated nights 2026-09-24/25 and watched through the demo boxes' first real automatic night (`audits/DRILL-night-2026-09-25.md`).** Decisions 31–33 taken unattended. Gaps a scratch box cannot close: R-687; no resume after a restart: R-686. *(Before:)* **SPIKED 2026-09-24 (night, Part C) — the measured build brief is §6.4.2 below.** **The update leg in the chain + the automatic caller + the switch.** A leg that starts when the off-site leg has ENDED (on every exit path, skips included), one app at a time, one step per press, `app_update.unattended` default ON (decision 12), `stacks.update_window` removed, reads `UpdateRefusal.Reason`, remembers a failed step so it never re-presses it, and the full-system backup's gate waits for it until W+5h (decision 20). | 11, 12, 14, 15, 20 | **3** | 1, 2, 5 | medium — the only part that acts with nobody watching; everything above is what makes it safe |
|
||||||
| **8** | **SHIPPED — controller v0.265.0 + hub v0.121.0, proven live on 9202 2026-09-23** (`audits/cleanup-2026-09-23/`; a kernel `oom_kill` counter, not the sticky flag — `08` §6.2). **R-636** — the same OOM key re-firing escalates instead of staying one `warning` for six hours. | R-636 | **1** | — | none |
|
| **8** | **SHIPPED — controller v0.265.0 + hub v0.121.0, proven live on 9202 2026-09-23** (`audits/cleanup-2026-09-23/`; a kernel `oom_kill` counter, not the sticky flag — `08` §6.2). **R-636** — the same OOM key re-firing escalates instead of staying one `warning` for six hours. | R-636 | **1** | — | none |
|
||||||
| **9** | **SHIPPED — controller v0.265.0, proven live on 9202 2026-09-23** (badge „Megállítva — visszaállítás szükséges" / "Stopped — restore needed", no Update button, 409 `held` unchanged). **R-625** — a held app stops offering an Update it will refuse. With the undo, holds become rarer; the lie on the page does not go away by itself. | R-625 | **0.5** | 1 | none |
|
| **9** | **SHIPPED — controller v0.265.0, proven live on 9202 2026-09-23** (badge „Megállítva — visszaállítás szükséges" / "Stopped — restore needed", no Update button, 409 `held` unchanged). **R-625** — a held app stops offering an Update it will refuse. With the undo, holds become rarer; the lie on the page does not go away by itself. | R-625 | **0.5** | 1 | none |
|
||||||
@@ -2410,7 +2723,7 @@ During the db-dump leg every press was refused `busy` (transient, retried next p
|
|||||||
update's own finish should refresh them.
|
update's own finish should refresh them.
|
||||||
- **An Update pressed on an app already at the head runs the whole guarded update** — backup, pull,
|
- **An Update pressed on an app already at the head runs the whole guarded update** — backup, pull,
|
||||||
restart, `done` — with no refusal (R-679: navidrome restarted four times for nothing). The leg never
|
restart, `done` — with no refusal (R-679: navidrome restarted four times for nothing). The leg never
|
||||||
presses an app whose pin equals the head; the preflight should refuse with reason `current`.
|
presses an app whose pin equals the head; the preflight should refuse with reason `current` (as built the reason key is `already_current`, `update.go:432` — noted 2026-10-24).
|
||||||
- **The product does not remember a failed step.** After vikunja's undo the badge still offers the
|
- **The product does not remember a failed step.** After vikunja's undo the badge still offers the
|
||||||
same step; only the caller's memory kept it from a re-press. Decision 15 needs it on the box: the
|
same step; only the caller's memory kept it from a re-press. Decision 15 needs it on the box: the
|
||||||
failed `to` is recorded per app and the leg skips it until the catalog's ladder changes.
|
failed `to` is recorded per app and the leg skips it until the catalog's ladder changes.
|
||||||
@@ -2492,6 +2805,24 @@ and E are the ones that unblock a decision; leg A is the one that takes the time
|
|||||||
harness on DooPlex for the image-side ones.
|
harness on DooPlex for the image-side ones.
|
||||||
|
|
||||||
|
|
||||||
|
### 6.4.x Decisions in code no section records [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
|
||||||
|
|
||||||
|
Read at `file:line` on controller `main` (`321df1f`) by the main session:
|
||||||
|
|
||||||
|
- **`build.sh --push` also refuses a DIRTY tree, a HEAD that is not `origin/main`, and a directory that is not a git
|
||||||
|
checkout** — `controller/build.sh:144-166`. §0 names only the tag-exists and unreachable-registry refusals.
|
||||||
|
- **The leg retries a passing refusal ONCE in the same night** (`busy`, `updating`, `deploying`, `migrating`,
|
||||||
|
`self_updating`), after `RetryWait` — not „the next pass/night" (`internal/stacks/unattended.go:65-66`, the pass loop).
|
||||||
|
- **The leg skips an app the household stopped** (`stopped_by_household`, R-721): an update would start it
|
||||||
|
(`unattended.go:416-418`).
|
||||||
|
- **A manual run of the chain** (R-705, the debug action) gives the leg a deadline of its start + the leg's normal
|
||||||
|
length (`backupwindow.UpdateLegLengthMin`), not W+5h (`unattended.go:232-238`).
|
||||||
|
- **Two preflight refusals are plain skips to the leg:** `already_current` and `engine_no_test`
|
||||||
|
(`internal/stacks/update.go:432,454`).
|
||||||
|
- **The render table (§5.4) has one more row:** when the catalog template's images cannot be read, the syncer leaves
|
||||||
|
the app's compose file alone for that cycle (`internal/sync/sync.go:622-627`); and the box's R-971 allow-list runs
|
||||||
|
before ANY compose write (`cmd/controller/main.go:421`, `SetAdmitFn`).
|
||||||
|
|
||||||
## 6.5 The drill catalog and the image store — the standing method for update drills
|
## 6.5 The drill catalog and the image store — the standing method for update drills
|
||||||
|
|
||||||
**Why this section exists.** On 2026-09-21 an afternoon session put a deliberately broken image into
|
**Why this section exists.** On 2026-09-21 an afternoon session put a deliberately broken image into
|
||||||
@@ -2580,7 +2911,9 @@ customer box has credentials for it. The store listens on the guest's loopback o
|
|||||||
### 7.1 The record (slice 1)
|
### 7.1 The record (slice 1)
|
||||||
|
|
||||||
`Manager.recordInstalledImages` (`felhom-controller/controller/internal/stacks/installed.go`) runs
|
`Manager.recordInstalledImages` (`felhom-controller/controller/internal/stacks/installed.go`) runs
|
||||||
after a successful compose up from `StartStack`, `RestartStack`, `UpdateStack` and `runComposeDeploy`,
|
after a successful compose up from `StartStack`, `RestartStack`, `UpdateStack` and `runComposeDeploy`
|
||||||
|
*(as built — noted 2026-10-24: `UpdateStack` is gone; the callers are `StartStack`, `RestartStack`, the deploy path, the
|
||||||
|
end of the guarded update and the end of the undo — `manager.go`, `deploy.go`, `update.go`, `undo.go`)*,
|
||||||
and writes `app.yaml`:
|
and writes `app.yaml`:
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
@@ -2656,7 +2989,7 @@ Version strings stay in the logs, the API and the hub.
|
|||||||
The 8th, immich's own ghcr build, is UNMEASURED — ghcr exposes no anonymous last-modified. So on
|
The 8th, immich's own ghcr build, is UNMEASURED — ghcr exposes no anonymous last-modified. So on
|
||||||
demo-hp today four apps read „Naprakész" over a database engine image that has demonstrably moved.
|
demo-hp today four apps read „Naprakész" over a database engine image that has demonstrably moved.
|
||||||
Digest-level comparison needs the catalog to record the digest at push time — **R-446**, put to the
|
Digest-level comparison needs the catalog to record the digest at push time — **R-446**, put to the
|
||||||
operator as §3b **Q6**, recommended YES.
|
operator as §3b **Q6**, recommended YES. *(Built since for every ref with a ladder entry — `updateorder.go`, `digest.go`; the limitation remains only for a ref no entry records; noted 2026-10-24.)*
|
||||||
2. **~~Nothing enforces `catalog_since`.~~ Enforced by the pre-push hook since 2026-09-13 (R-452, `app-catalog-felhom.eu/scripts/check-catalog-since.py`); CI's shallow clone still skips it out loud.** A commit that moves an `image:` line and forgets the date
|
2. **~~Nothing enforces `catalog_since`.~~ Enforced by the pre-push hook since 2026-09-13 (R-452, `app-catalog-felhom.eu/scripts/check-catalog-since.py`); CI's shallow clone still skips it out loud.** A commit that moves an `image:` line and forgets the date
|
||||||
under-reports how far behind a box is. The gates runner fetches at `--depth 1` and has no parent
|
under-reports how far behind a box is. The gates runner fetches at `--depth 1` and has no parent
|
||||||
commit to diff against, so the gate needs a deeper fetch — **R-452**.
|
commit to diff against, so the gate needs a deeper fetch — **R-452**.
|
||||||
@@ -2681,13 +3014,18 @@ Version strings stay in the logs, the API and the hub.
|
|||||||
refuses without a restorable, proven Tier-2 copy, backs up first when that copy is stale, takes a
|
refuses without a restorable, proven Tier-2 copy, backs up first when that copy is stale, takes a
|
||||||
safety dump, and holds an app that does not come up. **What stays true:** it still has no automatic
|
safety dump, and holds an app that does not come up. **What stays true:** it still has no automatic
|
||||||
rollback (deliberately, §6.1) and can still attempt a multi-major jump the app will refuse (R-40) —
|
rollback (deliberately, §6.1) and can still attempt a multi-major jump the app will refuse (R-40) —
|
||||||
that now ends HELD rather than crash-looping behind a green button.
|
that now ends HELD rather than crash-looping behind a green button. *(Dated note 2026-10-24: a copy on ANY tier now
|
||||||
|
satisfies the precondition (v0.239.0), the box undoes a failed update itself (decision 15, v0.263.2), and the
|
||||||
|
ladder climbs one tested step per press (decision 14) — `internal/backup/update_guard.go`, `internal/stacks/undo.go`,
|
||||||
|
`ladder.go`.)*
|
||||||
7. ~~**An engine major can be applied without its datadir upgrade, and nothing notices.**~~ **CLOSED
|
7. ~~**An engine major can be applied without its datadir upgrade, and nothing notices.**~~ **CLOSED
|
||||||
2026-09-13 for MariaDB (R-459):** every `mariadb:` sidecar carries `MARIADB_AUTO_UPGRADE=1`, and the
|
2026-09-13 for MariaDB (R-459):** every `mariadb:` sidecar carries `MARIADB_AUTO_UPGRADE=1`, and the
|
||||||
harness shows the conversion running on the E3/E3b edges (§3 decision 5). **What stays true:** the
|
harness shows the conversion running on the E3/E3b edges (§3 decision 5). **What stays true:** the
|
||||||
PostgreSQL half (R-463) has no equivalent — the image performs no `pg_upgrade` — and the
|
PostgreSQL half (R-463) has no equivalent — the image performs no `pg_upgrade` — and the
|
||||||
engine-major rule (§3 precaution 3, R-469) is what keeps both engines inside their major until
|
engine-major rule (§3 precaution 3, R-469) is what keeps both engines inside their major until
|
||||||
Slice 4 gives the Update button a backup.
|
Slice 4 gives the Update button a backup. *(Dated note 2026-10-24: MariaDB is lifted — its major is its own edge; a
|
||||||
|
PostgreSQL major may pass with a two-venue proof and an `engine_conversion` mark, and the box converts it
|
||||||
|
(v0.273.0) — `app-catalog-felhom.eu/scripts/check-engine-major.py`.)*
|
||||||
8. ~~**Only three of 53 apps have ever had an upgrade measured.**~~ **WIDENED 2026-09-21 to 21
|
8. ~~**Only three of 53 apps have ever had an upgrade measured.**~~ **WIDENED 2026-09-21 to 21
|
||||||
EDGES ACROSS 19 APPS** (`audits/DRILL-update-night-2026-09-21.md`), on scratch guest 9202
|
EDGES ACROSS 19 APPS** (`audits/DRILL-update-night-2026-09-21.md`), on scratch guest 9202
|
||||||
through the product's own guarded Update, against a PRIVATE DRILL CATALOG so the live catalog
|
through the product's own guarded Update, against a PRIVATE DRILL CATALOG so the live catalog
|
||||||
|
|||||||
@@ -128,7 +128,9 @@ the hub's e-mails follow.**
|
|||||||
`expired` pinned to Hungarian so the language cannot become the oracle the text refuses to be.
|
`expired` pinned to Hungarian so the language cannot become the oracle the text refuses to be.
|
||||||
Full design: `05-hub-architecture.md` §15. **R-555 closed** — the `language` allowlist entry is out
|
Full design: `05-hub-architecture.md` §15. **R-555 closed** — the `language` allowlist entry is out
|
||||||
of `wire_contract_gate.py` and the gate now checks the field for real.
|
of `wire_contract_gate.py` and the gate now checks the field for real.
|
||||||
- **[DESIGN] Slice 3 Part A as planned — now built; the box half (Part B) is the remaining piece:** the hub stores a per-customer language, renders it into
|
- **[DESIGN] Slice 3 Part A as planned — now built; the box half (Part B) is the remaining piece** *(Part B is built
|
||||||
|
too — the controller reads `customer.language` as its start language since v0.256.0, `internal/config/config.go:149-154`;
|
||||||
|
corrected 2026-10-24)*: the hub stores a per-customer language, renders it into
|
||||||
`controller.yaml` next to `customer.id/name/domain/email` (`hub/internal/configgen/configgen.go`),
|
`controller.yaml` next to `customer.id/name/domain/email` (`hub/internal/configgen/configgen.go`),
|
||||||
and the box uses it **only while the household has never chosen** (`settings.json` empty). The
|
and the box uses it **only while the household has never chosen** (`settings.json` empty). The
|
||||||
household's own choice always wins; the hub's e-mails follow the language the box **reports**, which
|
household's own choice always wins; the hub's e-mails follow the language the box **reports**, which
|
||||||
@@ -184,7 +186,8 @@ bundles. They now judge the page **as rendered in each language** (`controller/s
|
|||||||
- **hu** — the product speaks in „te". The converted copy carries formal („ön") forms (R-516). They are
|
- **hu** — the product speaks in „te". The converted copy carries formal („ön") forms (R-516). They are
|
||||||
not fixed by a localisation release (§1), so the gate counts them against `HU_FORMAL_CEILING` as a
|
not fixed by a localisation release (§1), so the gate counts them against `HU_FORMAL_CEILING` as a
|
||||||
ratchet: a new one convicts, fixing one lowers the ceiling. The ceiling followed the conversion:
|
ratchet: a new one convicts, fixing one lowers the ceiling. The ceiling followed the conversion:
|
||||||
6 (spike) → 10 (release A) → 12 (release B) → **16** (release C). **[FACT] Measured limit:** the stem
|
6 (spike) → 10 (release A) → 12 (release B) → **16** (release C) → … → **0** (R-516 item 12, the last eleven in the
|
||||||
|
te-form — `felhom-controller/controller/scripts/i18n_missing_gate.py:41`; corrected 2026-10-24). **[FACT] Measured limit:** the stem
|
||||||
list is short — on the release C pages it counts 4 formal forms, while a wider list („írja be",
|
list is short — on the release C pages it counts 4 formal forms, while a wider list („írja be",
|
||||||
„adja meg", „válassza", „biztosan eltávolítja", „engedélyezze", …) finds at least 22 keys (R-516).
|
„adja meg", „válassza", „biztosan eltávolítja", „engedélyezze", …) finds at least 22 keys (R-516).
|
||||||
- **en** — second person, plain: no „please", no „kindly".
|
- **en** — second person, plain: no „please", no „kindly".
|
||||||
@@ -201,7 +204,8 @@ vissza") — R-564. The hub copy gate and the catalog have no language concept
|
|||||||
- **[DESIGN] The hub's operator pages** — the operator reads them; they stay English (decision 1).
|
- **[DESIGN] The hub's operator pages** — the operator reads them; they stay English (decision 1).
|
||||||
- **[DESIGN] The first-boot wizard** (`controller/internal/setup/`, 8 pages, 95 strings) — out of scope
|
- **[DESIGN] The first-boot wizard** (`controller/internal/setup/`, 8 pages, 95 strings) — out of scope
|
||||||
and obsolete (decision 4, 2026-09-17; `02-controller-module-map.md` L56). A household still reaches
|
and obsolete (decision 4, 2026-09-17; `02-controller-module-map.md` L56). A household still reaches
|
||||||
it when bootstrap ingestion leaves `customer.id` empty (inventory §2.7). Its deletion is **R-554**.
|
it when bootstrap ingestion leaves `customer.id` empty (inventory §2.7). Its deletion is **R-554**. *(Deleted since:
|
||||||
|
`internal/setup/` is gone (v0.308.0); `bootstrap.WaitForConfig` shows a waiting page instead — corrected 2026-10-24.)*
|
||||||
- **[FACT]** App containers' own UIs (Uptime Kuma, PrivateBin…) — not Felhom's copy (R-516 items 5-6).
|
- **[FACT]** App containers' own UIs (Uptime Kuma, PrivateBin…) — not Felhom's copy (R-516 items 5-6).
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -226,7 +230,7 @@ i18n:
|
|||||||
```
|
```
|
||||||
|
|
||||||
Why: one app's copy stays in one file a reviewer sees whole; the controller's `yaml.Unmarshal` ignores
|
Why: one app's copy stays in one file a reviewer sees whole; the controller's `yaml.Unmarshal` ignores
|
||||||
unknown keys (`internal/stacks/metadata.go` L316 — **verified: this repo constructs no `yaml.Decoder`
|
unknown keys (`internal/stacks/metadata.go` L316 then, L475 now — **verified: this repo constructs no `yaml.Decoder`
|
||||||
at all, so there is no `KnownFields` anywhere to turn strict**), so older controllers are unaffected;
|
at all, so there is no `KnownFields` anywhere to turn strict**), so older controllers are unaffected;
|
||||||
a missing English field falls back field by field.
|
a missing English field falls back field by field.
|
||||||
|
|
||||||
@@ -234,7 +238,7 @@ a missing English field falls back field by field.
|
|||||||
reached only through `LocalizeStacks` / `LocalizeStackPtr` / `Stack.MetaFor`.
|
reached only through `LocalizeStacks` / `LocalizeStackPtr` / `Stack.MetaFor`.
|
||||||
|
|
||||||
- **`For("hu")` is the parsed struct with `I18n` cleared and nothing else**, deep-compared against all
|
- **`For("hu")` is the parsed struct with `I18n` cleared and nothing else**, deep-compared against all
|
||||||
53 real catalog files (`TestMetaForHuIsIdentity`, fixtures in `internal/stacks/testdata/catalog/`).
|
53 real catalog files (the identity fixtures are still those 53; the catalog has 65 apps — noted 2026-10-24) (`TestMetaForHuIsIdentity`, fixtures in `internal/stacks/testdata/catalog/`).
|
||||||
§1 therefore holds by construction for catalog copy, and `LocalizeStacks` returns the INPUT SLICE
|
§1 therefore holds by construction for catalog copy, and `LocalizeStacks` returns the INPUT SLICE
|
||||||
for Hungarian rather than a copy, so nothing on the Hungarian path is even touched.
|
for Hungarian rather than a copy, so nothing on the Hungarian path is even touched.
|
||||||
- **Fallback is field by field, and blank counts as absent** (ruling 3). A half-translated app is a
|
- **Fallback is field by field, and blank counts as absent** (ruling 3). A half-translated app is a
|
||||||
@@ -307,10 +311,10 @@ for the English page test (R-565), arriving again in a different repo. So
|
|||||||
`app-catalog-felhom.eu/scripts/check-copy-i18n.py` folds and matches word-bounded stems, with a
|
`app-catalog-felhom.eu/scripts/check-copy-i18n.py` folds and matches word-bounded stems, with a
|
||||||
positive and two negative controls printed on every run.
|
positive and two negative controls printed on every run.
|
||||||
|
|
||||||
**[FACT] The gate**, fifth row of `catalog_gates.py`, static, in the pre-push hook:
|
**[FACT] The gate**, fifth row of `catalog_gates.py` (the sixth since — corrected 2026-10-24), static, in the pre-push hook:
|
||||||
|
|
||||||
1. **FREEZE** — every Hungarian copy string equals `scripts/copy_freeze/hu.json`, captured in its own
|
1. **FREEZE** — every Hungarian copy string equals `scripts/copy_freeze/hu.json`, captured in its own
|
||||||
commit before any translation. Runs on all 53 apps whatever scope is named. A new app must be
|
commit before any translation. Runs on every app in the freeze file (65 on 2026-10-24) whatever scope is named. A new app must be
|
||||||
admitted with `--add-app NAME --reason "…"`.
|
admitted with `--add-app NAME --reason "…"`.
|
||||||
2. **STRUCTURE** — the `en` block may carry copy fields and nothing else; every key-matched entry must
|
2. **STRUCTURE** — the `en` block may carry copy fields and nothing else; every key-matched entry must
|
||||||
have a Hungarian twin, or it is INERT on the box and the translator never learns.
|
have a Hungarian twin, or it is INERT on the box and the translator never learns.
|
||||||
|
|||||||
@@ -11,7 +11,7 @@
|
|||||||
|
|
||||||
> | | |
|
> | | |
|
||||||
> |---|---|
|
> |---|---|
|
||||||
> | **Status** | **NOT RATIFIED — a PROPOSAL with operator rulings, corrected by the 2026-10-04 spike (§7.1, C1–C12); §8 step 2 BUILT 2026-10-04 (§8.1).** Ratification is Viktor's review, not an editor's. |
|
> | **Status** | **NOT RATIFIED — a PROPOSAL with operator rulings, corrected by the 2026-10-04 spike (§7.1, C1–C12); §8 step 2 BUILT 2026-10-04 (§8.1); steps 2–6 are BUILT by §8's own list (corrected 2026-10-24 — the ratification is still the operator's).** Ratification is Viktor's review, not an editor's. |
|
||||||
> | **Written** | 2026-10-04, by the reviewer (project Claude), before any spike. |
|
> | **Written** | 2026-10-04, by the reviewer (project Claude), before any spike. |
|
||||||
> | **Verified against** | felhom.eu `d07a1a9` · felhom-controller `99a1497` (v0.290.0) · felhom-agent `d766666` (v0.138.0) · hub v0.128.0 |
|
> | **Verified against** | felhom.eu `d07a1a9` · felhom-controller `99a1497` (v0.290.0) · felhom-agent `d766666` (v0.138.0) · hub v0.128.0 |
|
||||||
> | **Freshness** | **CURRENT** as of 2026-10-04. The spike `TASK-backup-close-and-os-updates-spike-2026-10-04` adds measurements here as `[FACT]` and corrects every claim it disproves. Mark this file STALE when it falls behind what the product does. |
|
> | **Freshness** | **CURRENT** as of 2026-10-04. The spike `TASK-backup-close-and-os-updates-spike-2026-10-04` adds measurements here as `[FACT]` and corrects every claim it disproves. Mark this file STALE when it falls behind what the product does. |
|
||||||
@@ -76,10 +76,22 @@ boxes installed. Nobody keeps a hand-written list.
|
|||||||
- **A BYO host** (the owner brought their own Proxmox). `[FACT]` The installer leaves its repositories
|
- **A BYO host** (the owner brought their own Proxmox). `[FACT]` The installer leaves its repositories
|
||||||
alone: *"apt repo alignment skipped (byo — the owner manages repos)"* (`scripts/felhom-host-install.sh`,
|
alone: *"apt repo alignment skipped (byo — the owner manages repos)"* (`scripts/felhom-host-install.sh`,
|
||||||
`align_apt_repos`). `[PROPOSAL]` On a BYO box we update the guest and Docker only, never the host.
|
`align_apt_repos`). `[PROPOSAL]` On a BYO box we update the guest and Docker only, never the host.
|
||||||
|
**[DESIGN, operator ruling D43 = A, 2026-10-24 — committed agent `c8d3c17`, unreleased]** The host's root-key-only sshd
|
||||||
|
drop-in (D4, R-982) is an APPLIANCE-only file of the config bundle: written only when the root-owned install record
|
||||||
|
says `appliance`; on a `byo` record nothing is written and a copy the bundle wrote earlier (this path, our first line)
|
||||||
|
is removed; a box whose record proves nothing is left as it is (R-1257). **[FACT] 2026-10-25 (BYO test host, agent
|
||||||
|
0.164.0 → 0.165.0):** the removal came one bundle LATE — the bundle that brings the new wrapper is planned by the OLD
|
||||||
|
one (R-1285). **[DESIGN — R-1285, agent `abcc1e6`, unreleased]:** right after every bundle install the agent asks the
|
||||||
|
wrapper now on disk for a policy pass (`mode: "bundle-policy"`), which removes our appliance-only files from a BYO
|
||||||
|
record (no signature: it only removes a file we wrote; a failing `sshd -t` puts it back).
|
||||||
|
**[DESIGN — R-1285 follow-up, agent `b1bc63d`, unreleased]:** in the normal delivery the bundle job runs in the OLD
|
||||||
|
process (measured on demo-hp 2026-10-26: the bundle 09:03:07–09:03:09, the restart into 0.166.0 at 09:03:10), so the
|
||||||
|
new agent also runs the pass once right after its self-update commits (`selfupdate.Manager.SetOnCommitted`). Only a
|
||||||
|
box coming from 0.164.0 can show it: 0.165.0's wrapper already applies D43 at plan time.
|
||||||
- **The Proxmox MAJOR upgrade** (PVE 9 → 10). It is a later step of its own, drilled first (R-808 item 5).
|
- **The Proxmox MAJOR upgrade** (PVE 9 → 10). It is a later step of its own, drilled first (R-808 item 5).
|
||||||
- **`cloudflared` and the other infrastructure images** (traefik, filebrowser). **[FACT] (C8)** They are containers in
|
- **`cloudflared` and the other infrastructure images** (traefik, filebrowser). **[FACT] (C8)** They are containers in
|
||||||
the guest, pinned in the controller (`internal/infra/infra.go:26`: `cloudflare/cloudflared:2026.6.0`, since
|
the guest, pinned in the controller (`internal/infra/infra.go:26`: `cloudflare/cloudflared:2026.6.0`, since
|
||||||
2026-06-11) and baked into the golden; upstream was `2026.9.3` on 2026-10-04. They move only by a controller release
|
2026-06-11 — *`2026.9.3` from 2026-10-24; `2026.10.3` since controller v0.345.1, 2026-10-26 — Cloudflare's eMudhra notice, R-1300*) and baked into the golden; upstream was `2026.9.3` on 2026-10-04. They move only by a controller release
|
||||||
— the app-image question (`09`), not an OS package. The gap is R-838.
|
— the app-image question (`09`), not an OS package. The gap is R-838.
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -101,9 +113,10 @@ gets the Docker engine that was current when its golden was baked, and keeps it.
|
|||||||
|
|
||||||
~~**[FACT] The agent may not run `apt` today, with one exception.** Its sudoers allowlist
|
~~**[FACT] The agent may not run `apt` today, with one exception.** Its sudoers allowlist
|
||||||
(`felhom-agent/configs/felhom-agent.sudoers`) holds one apt line: `apt-get install -y -q dnsmasq`.~~
|
(`felhom-agent/configs/felhom-agent.sudoers`) holds one apt line: `apt-get install -y -q dnsmasq`.~~
|
||||||
**[FACT] (C1) The agent may run `apt-get install` for two named packages:** `felhom-agent.sudoers:58`
|
**[FACT] (C1) The agent may run `apt-get install` for two named packages:** `felhom-agent.sudoers:68`
|
||||||
(`apt-get install -y -q dnsmasq`, used by `internal/lanresolver/lanresolver.go:107`) and `:194`
|
(`apt-get install -y -q dnsmasq`, used by `internal/lanresolver/lanresolver.go:133`) and `:228`
|
||||||
(`apt-get install -y -q wireguard-tools`, used by `internal/wgtunnel/manager.go:628`). Nothing else.
|
(`apt-get install -y -q wireguard-tools`, used by `internal/wgtunnel/manager.go:659`). Nothing else. *(Line numbers
|
||||||
|
corrected 2026-10-24.)*
|
||||||
`apt` as root runs package scripts as root, so a broad `apt` grant is a full root grant. §5.4
|
`apt` as root runs package scripts as root, so a broad `apt` grant is a full root grant. §5.4
|
||||||
proposes how to avoid that.
|
proposes how to avoid that.
|
||||||
|
|
||||||
@@ -250,7 +263,7 @@ approval of that day stays in force (decided by CC unattended — operator may r
|
|||||||
versions. A broken-into hub can choose an older version or no version. It cannot make a box install a
|
versions. A broken-into hub can choose an older version or no version. It cannot make a box install a
|
||||||
package that the publisher did not sign.
|
package that the publisher did not sign.
|
||||||
|
|
||||||
### 5.4.1 The root wrapper's interface — DRAFT (2026-10-04, design only, nothing installed) `[PROPOSAL]`
|
### 5.4.1 The root wrapper's interface — DRAFT (2026-10-04, design only, nothing installed) `[PROPOSAL]` — *BUILT and installed since (§8.1); the agent's sudoers holds only the `--plan` line, no `--repair-only` (`felhom-agent/configs/felhom-agent.sudoers:337`); noted 2026-10-24*
|
||||||
|
|
||||||
`felhom-os-apply` — root-owned (`0755 root:root`), installed by the installer beside `felhom-selfupdate-guarded`;
|
`felhom-os-apply` — root-owned (`0755 root:root`), installed by the installer beside `felhom-selfupdate-guarded`;
|
||||||
the agent's sudoers gets exactly `/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json` and
|
the agent's sudoers gets exactly `/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json` and
|
||||||
@@ -315,7 +328,7 @@ PID 1 or `lxc-start`, or a kernel was installed.
|
|||||||
Decision 96. Before it, the signed `agent_update` replaced only the binary; wrappers, units and sudoers lines reached
|
Decision 96. Before it, the signed `agent_update` replaced only the binary; wrappers, units and sudoers lines reached
|
||||||
an installed box by reinstall or by hand.
|
an installed box by reinstall or by hand.
|
||||||
|
|
||||||
- **One source of truth.** `BUNDLE_FILES` in `felhom-os-apply` is the ONE table of root-owned paths (22: sudoers ×2, the
|
- **One source of truth.** `BUNDLE_FILES` in `felhom-os-apply` is the ONE table of root-owned paths (22 then, **32** counted 2026-10-24 — plus the vzdump-partial-clean, thin-check, internal-place and priv-apply wrappers, the guest hook, the shared-parent unit, the GRUB one-shot pair and the sshd root-key-only drop-in; sudoers ×2, the
|
||||||
five wrappers, the crash guard and its units + config, the agent and rollback units, the start-limit drop-in, the mgmt
|
five wrappers, the crash guard and its units + config, the agent and rollback units, the start-limit drop-in, the mgmt
|
||||||
watchdog and its tmpfiles/units, the OOB belt's four files). `scripts/build-config-bundle.py` builds the bundle from it
|
watchdog and its tmpfiles/units, the OOB belt's four files). `scripts/build-config-bundle.py` builds the bundle from it
|
||||||
reproducibly; `release-agent.sh` publishes it beside the binary; the hub vouches its sha with the agent (exact-name
|
reproducibly; `release-agent.sh` publishes it beside the binary; the hub vouches its sha with the agent (exact-name
|
||||||
@@ -393,6 +406,9 @@ must never overlap a backup, a restore-test or a self-update.~~
|
|||||||
- **Operator:** a hub event for each update and each failure; a fleet view showing each box's OS release,
|
- **Operator:** a hub event for each update and each failure; a fleet view showing each box's OS release,
|
||||||
how far behind it is, and whether it needs a reboot. A box that is more than N days behind the newest
|
how far behind it is, and whether it needs a reboot. A box that is more than N days behind the newest
|
||||||
approved release raises an alarm (`08`).
|
approved release raises an alarm (`08`).
|
||||||
|
- **[FACT, added 2026-10-24 — hub v0.169.0] The System page's „Upload line" cell** shows the best of the last SUCCESSFUL
|
||||||
|
weekly line test (R-1247), amber when it is older than 15 days or was never measured after an attempt — judged on
|
||||||
|
`last_ok_at`, never on the attempt time (`hub/internal/web/system.go` `lineStaleAfter`, `lineTestCell`).
|
||||||
- **[FACT, hub v0.132.0] The System page** (`/system`, R-852, decision 89): one row per box — ring and switch with
|
- **[FACT, hub v0.132.0] The System page** (`/system`, R-852, decision 89): one row per box — ring and switch with
|
||||||
buttons, the tunnel, host Proxmox / running and next-boot kernel / Debian / release / pending / not covered / held /
|
buttons, the tunnel, host Proxmox / running and next-boot kernel / Debian / release / pending / not covered / held /
|
||||||
reboot needed / `kernel.panic` / oops / crash restarts / the guard, guest Debian / release / pending / restart needed,
|
reboot needed / `kernel.panic` / oops / crash restarts / the guard, guest Debian / release / pending / restart needed,
|
||||||
@@ -480,7 +496,9 @@ Decision 88 (R-851): *"Yes, but maybe not indefinitely."* Evidence `audits/os-do
|
|||||||
**What it updates:** the host's Proxmox USERSPACE packages (pve-manager, qemu-server, pve-container, lxc-pve,
|
**What it updates:** the host's Proxmox USERSPACE packages (pve-manager, qemu-server, pve-container, lxc-pve,
|
||||||
libpve-*, proxmox-widget-toolkit, the ceph client libraries …) — the 77–78 packages behind on the demo boxes on
|
libpve-*, proxmox-widget-toolkit, the ceph client libraries …) — the 77–78 packages behind on the demo boxes on
|
||||||
2026-10-07. **What it never touches:** a kernel, boot, firmware or microcode name (`HOST_SLOW_RE`: `proxmox-kernel*`,
|
2026-10-07. **What it never touches:** a kernel, boot, firmware or microcode name (`HOST_SLOW_RE`: `proxmox-kernel*`,
|
||||||
`proxmox-default-kernel`, `pve-kernel*`, `pve-firmware`, `shim*`, `grub*`, `systemd-boot`, `*-microcode`, `efibootmgr`)
|
`proxmox-default-kernel`, `pve-kernel*`, `pve-firmware`, `shim*`, `grub*`, `systemd-boot`, `*-microcode`, `efibootmgr`,
|
||||||
|
and — read 2026-10-24 in `felhom-os-apply` — `linux-(image|headers|kbuild|modules|base)*`, `firmware-*`,
|
||||||
|
`proxmox-secure-boot-support`)
|
||||||
— that is R-836's lane (decision 164). No reboot.
|
— that is R-836's lane (decision 164). No reboot.
|
||||||
|
|
||||||
- **Wrapper** (`felhom-agent/configs/felhom-os-apply`): layer `pve`, lane `slow` only. Origin `Proxmox Debian
|
- **Wrapper** (`felhom-agent/configs/felhom-os-apply`): layer `pve`, lane `slow` only. Origin `Proxmox Debian
|
||||||
|
|||||||
@@ -20,8 +20,10 @@ workspace-root `CLAUDE.md` § „Artifact taxonomy". **[FACT]** The spec shape i
|
|||||||
class). Validation is project Claude's unless CC is asked (`CLAUDE.md`, same section).
|
class). Validation is project Claude's unless CC is asked (`CLAUDE.md`, same section).
|
||||||
|
|
||||||
**A session with no spec** (a `/goal`, a night brief, „work the register") inherits the discipline a
|
**A session with no spec** (a `/goal`, a night brief, „work the register") inherits the discipline a
|
||||||
spec used to carry from `.claude/rules/unprompted-work.md` — **[FACT]** five byte-identical copies:
|
spec used to carry from `.claude/rules/unprompted-work.md` — **[FACT]** five copies:
|
||||||
the workspace root and each of the four repos (the file's own header says „change all five or none").
|
the workspace root and each of the four repos (the file's own header says „change all five or none"). *(Measured
|
||||||
|
2026-10-24: they are NOT byte-identical — the felhom.eu and felhom-controller copies carry one more paragraph, „A
|
||||||
|
version number is pushed once" (added 2026-10-15); the review round 2 session aligned the other three, see its report.)*
|
||||||
|
|
||||||
## 2. Where an instruction lives
|
## 2. Where an instruction lives
|
||||||
|
|
||||||
@@ -29,7 +31,7 @@ the workspace root and each of the four repos (the file's own header says „cha
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Cross-repo facts and standing rules (production host, gates, secrets, CHANGELOG/REPORT) | workspace-root `CLAUDE.md` | every session in the workspace |
|
| Cross-repo facts and standing rules (production host, gates, secrets, CHANGELOG/REPORT) | workspace-root `CLAUDE.md` | every session in the workspace |
|
||||||
| One repo's stable orientation | `<repo>/CLAUDE.md` | when a file in that repo is touched |
|
| One repo's stable orientation | `<repo>/CLAUDE.md` | when a file in that repo is touched |
|
||||||
| Path-scoped rules (a subsystem's traps) | `<repo>/.claude/rules/*.md` | when a matching path is touched |
|
| Path-scoped rules (a subsystem's traps) | `<repo>/.claude/rules/*.md` | when a matching path is touched — a file with `unconditional: true` (e.g. `live-probes.md`) or without `paths:` (the `unprompted-work.md` copies) loads every session (corrected 2026-10-24) |
|
||||||
| A procedure with steps (build, test, diagnose, write for the operator) | a skill, `felhom.eu/skills/<name>/SKILL.md` | when its trigger matches, or by name |
|
| A procedure with steps (build, test, diagnose, write for the operator) | a skill, `felhom.eu/skills/<name>/SKILL.md` | when its trigger matches, or by name |
|
||||||
| Current state, decisions, open work | `CONTEXT.md`, `STATUS.md`, `documentation/backlog/OPEN-ITEMS.md` | read on demand — never copied into an instruction file |
|
| Current state, decisions, open work | `CONTEXT.md`, `STATUS.md`, `documentation/backlog/OPEN-ITEMS.md` | read on demand — never copied into an instruction file |
|
||||||
| Durable cross-session facts about the operator and the machines | the memory folder (`.claude-memory/`, index `MEMORY.md`) | the index each session; files on recall |
|
| Durable cross-session facts about the operator and the machines | the memory folder (`.claude-memory/`, index `MEMORY.md`) | the index each session; files on recall |
|
||||||
@@ -38,8 +40,9 @@ Which rung a piece of material belongs on is the doc-authoring skill's §3 („W
|
|||||||
material sits") and §6 („One rule, one home"). This page does not repeat them.
|
material sits") and §6 („One rule, one home"). This page does not repeat them.
|
||||||
|
|
||||||
**[FACT] Skills** are written in `felhom.eu/skills/`, installed by
|
**[FACT] Skills** are written in `felhom.eu/skills/`, installed by
|
||||||
`python3 felhom.eu/scripts/install_skills.py` as symlinks into `~/.claude/skills/` (so a repo edit is
|
`python3 felhom.eu/scripts/install_skills.py` as symlinks into `~/.claude/skills/` (so an edit to an INSTALLED skill is
|
||||||
live at once), and checked by `python3 felhom.eu/scripts/check_skills.py` (shape, the 150-line limit
|
live at once; a NEW skill folder is live only after `install_skills.py` runs again — measured 2026-10-24:
|
||||||
|
`skills/felhom-web-design` has no link yet), and checked by `python3 felhom.eu/scripts/check_skills.py` (shape, the 150-line limit
|
||||||
and its one grandfathered entry, R-394). The list is the folder; `skills/SOURCES.md` names where each
|
and its one grandfathered entry, R-394). The list is the folder; `skills/SOURCES.md` names where each
|
||||||
skill's material came from.
|
skill's material came from.
|
||||||
|
|
||||||
@@ -69,4 +72,7 @@ secret rule or a review step, or remove a rule — that is the operator's. The f
|
|||||||
that uses a value by NAME and never prints it, and Claude Code hooks in `~/.claude/settings.json` — so every CC
|
that uses a value by NAME and never prints it, and Claude Code hooks in `~/.claude/settings.json` — so every CC
|
||||||
session on DooPlex — that refuse the known leak shapes, mask stored values in tool output, and a nightly sweep of old
|
session on DooPlex — that refuse the known leak shapes, mask stored values in tool output, and a nightly sweep of old
|
||||||
session records that mails NAMES only. Built 2026-10-17 (R-1118); evidence `audits/cc-creds-build-2026-10-17/`.
|
session records that mails NAMES only. Built 2026-10-17 (R-1118); evidence `audits/cc-creds-build-2026-10-17/`.
|
||||||
|
**D60 (operator 2026-10-27 21:33, `09` §3 decision 271): CC never deletes from the store.** `sec rm` is the operator's
|
||||||
|
command; the hooks refuse it in every form, and any removal or overwrite of the store file or the age key (Bash and a
|
||||||
|
`pre-write` Write/Edit hook, cc-secrets ≥ 0.4.0).
|
||||||
`runbooks/secrets.md` stays the home of the CLUSTER Secrets.
|
`runbooks/secrets.md` stays the home of the CLUSTER Secrets.
|
||||||
@@ -36,6 +36,7 @@ decision (R-925).
|
|||||||
| g | a compromised Gitea / registry / build | push to the catalog, the images and the installer tag |
|
| g | a compromised Gitea / registry / build | push to the catalog, the images and the installer tag |
|
||||||
| h | a thief with the box | the disks |
|
| h | a thief with the box | the disks |
|
||||||
| i | Cloudflare | the plaintext of tunnel traffic — **accepted** (R-904) |
|
| i | Cloudflare | the plaintext of tunnel traffic — **accepted** (R-904) |
|
||||||
|
| j | **added 2026-10-23:** a hacked homelab app pod on DooPlex (happened: Plex2Shell, HM-040) | the pod network with free egress: every cluster Service, the node's ports (NFS, kubelet, k3s API), the LAN masqueraded as DooPlex; no service-account token |
|
||||||
|
|
||||||
## 4. Entry points
|
## 4. Entry points
|
||||||
|
|
||||||
@@ -46,8 +47,8 @@ scripts are held with the review's working notes.
|
|||||||
| component | routes | no credential | listeners |
|
| component | routes | no credential | listeners |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| controller | 231 (+8 in setup mode) | 21, of which 7 change state, each behind its own password, code or token | port 8080 on the guest's Docker networks (not published); mail relay 2525/2465/2526 |
|
| controller | 231 (+8 in setup mode) | 21, of which 7 change state, each behind its own password, code or token | port 8080 on the guest's Docker networks (not published); mail relay 2525/2465/2526 |
|
||||||
| hub | 121 (1 health, 53 API, 67 web — v0.152.0 added the two token routes; both route tables are now pinned to the source by tests, R-952) | appliance register (rate-limited, by design), re-enroll (only while the operator has armed recovery mode), login, health | one ingress, path `/`, geo filter (HU, IE, LAN) |
|
| hub | 121 (1 health, 53 API, 67 web — v0.152.0 added the two token routes; both route tables are now pinned to the source by tests, R-952) — *the API table has 54 entries since `/box-config/` (v0.161.0) and `/preferences` (v0.165.0); counted from `hub/internal/api/routes.go`, 2026-10-24* | appliance register (rate-limited, by design), re-enroll (only while the operator has armed recovery mode), login, health, and the public self-bind page `/bind/<token>` (a capability link, added to this list 2026-10-24) | one ingress, path `/`, geo filter (HU, IE, LAN) |
|
||||||
| agent | 38 local-API routes (`POST /host/poweroff` added 2026-10-13, R-998) | 0 | local API on the island bridge; WireGuard; poke (tunnel only); felhom-sshd (tunnel only, measured) |
|
| agent | 41 local-API routes on `main` (counted 2026-10-24 in `felhom-agent/internal/localapi/server.go`; `POST /host/poweroff` added 2026-10-13, R-998; `POST /host/line-test/run` added 2026-10-24, R-1250, unreleased) | 0 | local API on the island bridge; WireGuard; poke (tunnel only); felhom-sshd (tunnel only, measured) |
|
||||||
|
|
||||||
**Outbound fetches that bring code or config onto a box:** the catalog (git, `main`), app images, the four infra
|
**Outbound fetches that bring code or config onto a box:** the catalog (git, `main`), app images, the four infra
|
||||||
images, the controller image (by tag), `controller.yaml` from the hub, the agent binary and config bundle (signed),
|
images, the controller image (by tag), `controller.yaml` from the hub, the agent binary and config bundle (signed),
|
||||||
@@ -57,11 +58,11 @@ OS packages (exact versions, apt-signed), the installer script (by tag, R-980).
|
|||||||
|
|
||||||
| boundary | verdict | evidence |
|
| boundary | verdict | evidence |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| end-user ↔ apps | holds, with gaps | traefik routing and header stripping hold; homepage has the Docker socket and no gate (R-981) |
|
| end-user ↔ apps | holds, with gaps | traefik routing and header stripping hold; homepage has the Docker socket and no gate (R-981). **Re-check (2026-10-24, review round 2):** the socket is gone from the homepage template (catalog `templates/homepage/docker-compose.yml:21`) and the box's allow-list is empty, so a socket bind is refused (`felhom-controller/controller/internal/privpolicy/allowlist.json`) |
|
||||||
| **added 2026-10-14 (R-1033, P1):** a visitor ↔ Cloudflare over plain http | holds on boxes with controller ≥ 0.312.0 | before: every public name answered `http://` (login pages too), and Sonarr/Radarr/Prowlarr redirected https visitors to an absolute `http://` login; a password could cross an untrusted network unencrypted (attacker: anyone on the visitor's network). Now traefik sends every `http://` request to https (301) before an app sees it, HSTS one day; boxes below 0.312.0 (Tester 2 at 0.292.0) keep the gap until the floor delivers it. `audits/https-only-2026-10-14/` |
|
| **added 2026-10-14 (R-1033, P1):** a visitor ↔ Cloudflare over plain http | holds on boxes with controller ≥ 0.312.0 | before: every public name answered `http://` (login pages too), and Sonarr/Radarr/Prowlarr redirected https visitors to an absolute `http://` login; a password could cross an untrusted network unencrypted (attacker: anyone on the visitor's network). Now traefik sends every `http://` request to https (301) before an app sees it, HSTS one day; boxes below 0.312.0 (Tester 2 at 0.292.0) keep the gap until the floor delivers it. `audits/https-only-2026-10-14/` |
|
||||||
| **added 2026-10-15 (R-1045):** a stranger ↔ a fresh Plex | holds on controller ≥ 0.313.0; on older ones only while the household does not press „Kész" before claiming | an unclaimed Plex answers everything to a private address with no token, and through the tunnel every request comes from one. The setup gate (catalog `9a23ee5`) closes the public path; 0.313.0 opens it only when Plex itself says claimed. A status probe meant to cover older controllers OPENED an unclaimed Plex while it started (`audits/media-build-2026-10-15/G7/`) and was removed |
|
| **added 2026-10-15 (R-1045):** a stranger ↔ a fresh Plex | holds on controller ≥ 0.313.0; on older ones only while the household does not press „Kész" before claiming | an unclaimed Plex answers everything to a private address with no token, and through the tunnel every request comes from one. The setup gate (catalog `9a23ee5`) closes the public path; 0.313.0 opens it only when Plex itself says claimed. A status probe meant to cover older controllers OPENED an unclaimed Plex while it started (`audits/media-build-2026-10-15/G7/`) and was removed |
|
||||||
| customer ↔ controller UI | partly | bcrypt and session hold; small gaps (R-983); family sessions survive a reset (R-976) |
|
| customer ↔ controller UI | partly | bcrypt and session hold; small gaps (R-983); family sessions survive a reset (R-976). **Re-check (2026-10-24, review round 2):** both fixed — a password change or a claim/reset ends every dashboard, family and gate session (controller v0.324.0, `internal/web/auth.go:352-366`) |
|
||||||
| controller ↔ agent | partly | every route bound to the caller's guest; a compromised controller can still flush restore points (R-973), void the recovery code (R-978), and — since agent v0.155.0 — switch the whole machine off once per power-on, holding the hub's silence alarms for up to 14 days (R-998; mitigations and residual risk: `14-host-power-off.md` §5) |
|
| controller ↔ agent | partly | every route bound to the caller's guest; a compromised controller can still flush restore points (R-973), void the recovery code (R-978) *(re-check 2026-10-24: both narrowed — a cadence cap on `POST /backup` since agent v0.160.0, `internal/localapi/backup_cap.go`; a new escrow package is uploaded only at the claim since agent v0.160.0, `escrow_ceremony.go`)*, and — since agent v0.155.0 — switch the whole machine off once per power-on, holding the hub's silence alarms for up to 14 days (R-998; mitigations and residual risk: `14-host-power-off.md` §5) |
|
||||||
| **added (R-1007):** controller → an app's own job queue | holds — nothing new | the `after_unclean_boot` commands run inside the app's container through the docker socket the controller already holds, and read the app's database with a read-only session: no key into the household's account, nothing a compromised controller could not already do (exec in any app, read its database). The household's lost-file names stay on the box (`app.yaml`, the app page); the operator event carries counts only, and a failing command's output is never logged (security review of controller `8d862e4`, fixed in `3b85a0c`) |
|
| **added (R-1007):** controller → an app's own job queue | holds — nothing new | the `after_unclean_boot` commands run inside the app's container through the docker socket the controller already holds, and read the app's database with a read-only session: no key into the household's account, nothing a compromised controller could not already do (exec in any app, read its database). The household's lost-file names stay on the box (`app.yaml`, the app page); the operator event carries counts only, and a failing command's output is never logged (security review of controller `8d862e4`, fixed in `3b85a0c`) |
|
||||||
| **added:** agent ↔ root on the host | holds by the sudoers | every root verb is one regex-anchored line or a root wrapper that re-checks its input (`03` §3); agent v0.156.1 adds one delete: `felhom-vzdump-partial-clean` removes only a DEAD half-written vzdump of the named guest on a `dir` storage (no vzdump on the host, nothing holding it open) — a compromised agent gains the deletion of a leftover that is not a backup |
|
| **added:** agent ↔ root on the host | holds by the sudoers | every root verb is one regex-anchored line or a root wrapper that re-checks its input (`03` §3); agent v0.156.1 adds one delete: `felhom-vzdump-partial-clean` removes only a DEAD half-written vzdump of the named guest on a `dir` storage (no vzdump on the host, nothing holding it open) — a compromised agent gains the deletion of a leftover that is not a backup |
|
||||||
| agent ↔ hub | holds for destructive ops | signed jobs verified in the documented order; unsigned spec has no size bounds (R-984) |
|
| agent ↔ hub | holds for destructive ops | signed jobs verified in the documented order; unsigned spec has no size bounds (R-984) |
|
||||||
@@ -69,12 +70,27 @@ OS packages (exact versions, apt-signed), the installer script (by tag, R-980).
|
|||||||
| controller ↔ hub | **did not hold until 2026-10-12** | the global key was public (R-950, rotated) and three routes needed no key (R-951, fixed); per-box keys are bound to their box (R-955) |
|
| controller ↔ hub | **did not hold until 2026-10-12** | the global key was public (R-950, rotated) and three routes needed no key (R-951, fixed); per-box keys are bound to their box (R-955) |
|
||||||
| box ↔ PBS | holds as written | ciphertext only; the zero-knowledge wording is stronger than a live box allows (R-985) |
|
| box ↔ PBS | holds as written | ciphertext only; the zero-knowledge wording is stronger than a live box allows (R-985) |
|
||||||
| guest ↔ Proxmox host | holds | the guest has no Proxmox credentials (measured) |
|
| guest ↔ Proxmox host | holds | the guest has no Proxmox credentials (measured) |
|
||||||
| hub ↔ Cloudflare API | holds for the zone check | tokens are stored unsealed in the hub database (R-975); R-137 open |
|
| hub ↔ Cloudflare API | holds for the zone check | tokens are stored unsealed in the hub database (R-975); R-137 open. **Re-check (2026-10-24, review round 2):** the tokens are sealed (`enc:v1:`) since hub 0.168.0, deployed 2026-10-24 (`hub/internal/store/r975_config_token_seal.go`) |
|
||||||
| **added:** app ↔ app (shared Docker daemon) | partly | separate containers and networks; any docker-socket app is root in the guest (R-981) |
|
| **added:** app ↔ app (shared Docker daemon) | partly | separate containers and networks; any docker-socket app is root in the guest (R-981) |
|
||||||
| **added:** app ↔ controller | partly | apps share a network with the controller; dashboard routes still need a session |
|
| **added:** app ↔ controller | partly | apps share a network with the controller; dashboard routes still need a session |
|
||||||
| **added:** LAN ↔ box | partly | root password SSH on the host from the LAN (R-982) |
|
| **added:** LAN ↔ box | partly | root password SSH on the host from the LAN (R-982). **Re-check (2026-10-24, review round 2):** the config bundle's key-only drop-in shipped in agent v0.163.0 (`felhom-agent/configs/felhom-os-apply:230`) |
|
||||||
| **added:** hub ↔ box (inbound) | **does not hold** | `01` §4 says the box exposes no control endpoint; a hub-key route can replace the box's settings (R-970) |
|
| **added:** hub ↔ box (inbound) | **does not hold** | `01` §4 says the box exposes no control endpoint; a hub-key route can replace the box's settings (R-970). **Re-check (2026-10-24, review round 2):** the config routes are removed and no route takes the hub key — pinned by `felhom-controller/controller/cmd/controller/r970_no_hub_key_door_test.go` (`01` §4 says the same) |
|
||||||
| **added:** catalog ↔ every box | **design gap** | a catalog push reaches every box with no check on the box (R-971) |
|
| **added:** catalog ↔ every box | **design gap** | a catalog push reaches every box with no check on the box (R-971). **Re-check (2026-10-24, review round 2):** the box checks every rendered compose against its allow-list before any write since controller v0.328.0 (`internal/privpolicy/privpolicy.go`) |
|
||||||
|
| **added 2026-10-23 (R-1241):** a hacked app pod on DooPlex ↔ Felhom's secrets | **does not hold**; not used in the Plex2Shell window | two no-login doors: the Longhorn NFS export (the hub's unencrypted nightly copies with the unsealed tokens of R-975; by a published file-handle escape the whole sda1 disk, the operator signing keys included — not exercised) and the Longhorn API (backup a volume to any target). Plex2Shell, 2026-10-17 → 10-23: **NOT REACHED** — 0 new NFS connections in the window, no off-schedule backup or volume; the hub, the k8s API and the kubelet refuse a caller with no credential (measured). The export was narrowed to the pod network on 2026-10-23; the pod side stays open (`audits/plex2shell-felhom-impact-2026-10-23/`). **Felhom half, 2026-10-23 night:** the last plain tokens sealed in hub 0.168.0 (deployed 2026-10-24 — corrected 2026-10-24; `audits/hub-token-seal-2026-10-23/`); the two plain Longhorn copies of the hub volume already in the folder are R-1243's decision. A household box's version of this attacker (a hacked app sending or mining) is what the upload watch (R-1244, released 2026-10-24 in controller v0.342.0 + hub v0.169.0 — corrected 2026-10-24) detects |
|
||||||
|
|
||||||
|
**[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1281]:** the file manager's one
|
||||||
|
sign-in (R-1100) — FileBrowser trusts its proxy header blindly, and every app container can reach `filebrowser:80`
|
||||||
|
directly; the only guard is that the header NAME is a per-box secret, which traefik's forwardAuth strips from every
|
||||||
|
client request and sets only for a browser holding the dashboard's files pass
|
||||||
|
(`felhom-controller/controller/internal/infra/filebrowser_sso.go:15-22`, `internal/web/files_sso.go`). The name file's
|
||||||
|
creation race was fixed by review round 2 (controller `b01bb4b`). The pick: FileBrowser joins only the network traefik uses, so no app
|
||||||
|
container reaches it directly. **Built 2026-10-25 night (controller `7ea942a`, unreleased):** a network of its own for
|
||||||
|
the route, `felhom-files` — traefik is on it, the CHECKED FileBrowser joins it by id after R-1126's default-login check,
|
||||||
|
and FileBrowser leaves traefik-public; OnlyOffice joins it while — and only while — its FileBrowser link is on (it
|
||||||
|
fetches and saves documents at `http://filebrowser:80`), the one stated exception. **[FACT] measured on the night's rig:**
|
||||||
|
before, from the qBittorrent and Transmission containers the forged header as the household user got the drive's
|
||||||
|
listing (200); after, no answer (`000`); the household's „Fájlok ↗" walk still gets 200 through traefik
|
||||||
|
(`audits/night-2026-10-25/rig/r1281-*.txt`).
|
||||||
|
|
||||||
## 6. What we accept (do not re-report)
|
## 6. What we accept (do not re-report)
|
||||||
|
|
||||||
|
|||||||
@@ -22,7 +22,7 @@ came back (BIOS „last state", inferred), its power button did (dashboard in 42
|
|||||||
| 2 | controller `POST /api/server/poweroff` (`internal/web/poweroff_handlers.go`) | dashboard login (`RequireAuth` + CSRF), POST only, the word again, the controller's own backup/restore/app-update check, the `host_poweroff` feature gate (MinAgent 0.155.0) |
|
| 2 | controller `POST /api/server/poweroff` (`internal/web/poweroff_handlers.go`) | dashboard login (`RequireAuth` + CSRF), POST only, the word again, the controller's own backup/restore/app-update check, the `host_poweroff` feature gate (MinAgent 0.155.0) |
|
||||||
| 3 | agent `POST /host/poweroff` (`internal/localapi/hostpoweroff.go`) | `withGuest` token scope; the guards (§3); the record (`internal/poweroff`) |
|
| 3 | agent `POST /host/poweroff` (`internal/localapi/hostpoweroff.go`) | `withGuest` token scope; the guards (§3); the record (`internal/poweroff`) |
|
||||||
| 4 | agent → hub | a host report carrying `deliberate_off.requested_at`, waited for until the hub ACCEPTED it (≤ 60 s, `Loop.SetReportedHook`) |
|
| 4 | agent → hub | a host report carrying `deliberate_off.requested_at`, waited for until the hub ACCEPTED it (≤ 60 s, `Loop.SetReportedHook`) |
|
||||||
| 5 | agent | `pct shutdown <vmid> --timeout 120`, then `systemctl poweroff` (two exact sudoers vectors, `FELHOM_POWEROFF`) |
|
| 5 | agent | `pct shutdown <vmid> --timeout 120`, then `systemctl poweroff` (two exact sudoers vectors: `pct ^shutdown [0-9]+ --timeout [0-9]+$` in `FELHOM_GUESTHOOK`, and `systemctl poweroff` in `FELHOM_POWEROFF` — `felhom-agent/configs/felhom-agent.sudoers:81-90`; corrected 2026-10-24) |
|
||||||
| 6 | hub (`store/deliberateoff.go`) | `hosts.deliberate_off_at`; silence alarms held 14 days (§4) |
|
| 6 | hub (`store/deliberateoff.go`) | `hosts.deliberate_off_at`; silence alarms held 14 days (§4) |
|
||||||
| 7 | the household | presses the machine's power button; the first report of the new boot has no `deliberate_off` → the hub clears it |
|
| 7 | the household | presses the machine's power button; the first report of the new boot has no `deliberate_off` → the hub clears it |
|
||||||
|
|
||||||
@@ -33,11 +33,11 @@ came back (BIOS „last state", inferred), its power button did (dashboard in 42
|
|||||||
| first 10 min after boot | `too_soon_after_boot` | `TooSoonAfterBoot` |
|
| first 10 min after boot | `too_soon_after_boot` | `TooSoonAfterBoot` |
|
||||||
| one accepted request per boot (file keyed by `/proc/stat` btime) | `already_requested_this_boot` | `SecondRequestSameBootRefused`, `poweroff/state_test.go` |
|
| one accepted request per boot (file keyed by `/proc/stat` btime) | `already_requested_this_boot` | `SecondRequestSameBootRefused`, `poweroff/state_test.go` |
|
||||||
| any other guest or VM running (`/proc`: `lxc-start -n N`, `kvm -id N`) | `other_guests_running` | `RefusedWhileOtherGuestsRun`, `RunningGuestsFromProc` |
|
| any other guest or VM running (`/proc`: `lxc-start -n N`, `kvm -id N`) | `other_guests_running` | `RefusedWhileOtherGuestsRun`, `RunningGuestsFromProc` |
|
||||||
| heavy-operation gate held (backup, restore test, OS step, sweep, trim) | `backup_running` / `restore_test_running` / `os_update_running` / `host_busy` | `RefusedDuringBackup` |
|
| heavy-operation gate held (backup, restore test, OS step, sweep, trim, and since agent v0.164.0 the weekly upload line test — corrected 2026-10-24) | `backup_running` / `restore_test_running` / `os_update_running` / `host_busy` | `RefusedDuringBackup` |
|
||||||
| backup job, controller swap, drive format, NAS verify, escrow ceremony, Proxmox vzdump | `backup_running` / `controller_update_running` / `drive_operation_running` / `escrow_ceremony_running` | `RefusedDuringVzdump`, `FailsClosedWithoutTaskReader` |
|
| backup job, controller swap, drive format, NAS verify, escrow ceremony, Proxmox vzdump | `backup_running` / `controller_update_running` / `drive_operation_running` / `escrow_ceremony_running` | `RefusedDuringVzdump`, `FailsClosedWithoutTaskReader` |
|
||||||
|
| after acceptance: the heavy gate is TAKEN and kept; before each step the jobs and other guests are read again — a job waits ≤ 15 min, another guest cancels at once, a job still running cancels | (log) | `JobStartedAfterAcceptance`, `OtherGuestStartedAfterAcceptance` |
|
||||||
|
|
||||||
**„A Proxmox vzdump runs” (agent v0.156.1, R-1008):** a vzdump task for the guest in Proxmox's active list whose worker process still lives — Proxmox's own rule (the UPID's pid + pstart against `/proc/<pid>/stat`). After a power cut the dead task stays listed for ~30–60 s; it no longer counts. Doubt (an unparsable task, an unreadable `/proc`) still refuses. The same answer serves the start-up stale-lock pass, the guest-power watchdog and the controller supervisor (`stalelock_inflight_test.go`).
|
**„A Proxmox vzdump runs” (agent v0.156.1, R-1008):** a vzdump task for the guest in Proxmox's active list whose worker process still lives — Proxmox's own rule (the UPID's pid + pstart against `/proc/<pid>/stat`). After a power cut the dead task stays listed for ~30–60 s; it no longer counts. Doubt (an unparsable task, an unreadable `/proc`) still refuses. The same answer serves the start-up stale-lock pass, the guest-power watchdog and the controller supervisor (`stalelock_inflight_test.go`).
|
||||||
| after acceptance: the heavy gate is TAKEN and kept; before each step the jobs and other guests are read again — a job waits ≤ 15 min, another guest cancels at once, a job still running cancels | (log) | `JobStartedAfterAcceptance`, `OtherGuestStartedAfterAcceptance` |
|
|
||||||
|
|
||||||
**[DESIGN]** A cancelled power-off leaves the box on (gate released) — better a box that did not switch off than a
|
**[DESIGN]** A cancelled power-off leaves the box on (gate released) — better a box that did not switch off than a
|
||||||
backup, update or format cut in the middle. **[DESIGN]** A failed guest shutdown does not stop the power-off: Proxmox's
|
backup, update or format cut in the middle. **[DESIGN]** A failed guest shutdown does not stop the power-off: Proxmox's
|
||||||
@@ -86,5 +86,15 @@ alternative (an out-of-band confirmation) needs a second channel the box does no
|
|||||||
|
|
||||||
## 6. Not done here
|
## 6. Not done here
|
||||||
|
|
||||||
- The website FAQ line („to turn it off: Beállítások → Rendszer → A gép kikapcsolása") — a later website task (R-1000).
|
- ~~The website FAQ line („to turn it off: Beállítások → Rendszer → A gép kikapcsolása") — a later website task (R-1000).~~ Published — `website/gyik.html:128,603` (corrected 2026-10-24).
|
||||||
- The BIOS „after power loss" value of the N100 was not read (no monitor); the household text promises only the button.
|
- The BIOS „after power loss" value of the N100 was not read (no monitor); the household text promises only the button.
|
||||||
|
|
||||||
|
## Decisions in code no section records [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
|
||||||
|
|
||||||
|
- **A re-enroll while the household power-off hold is active counts as „gone"** (a reinstall over the old host), not
|
||||||
|
„live" (R-1154; `hub/internal/api/reenroll.go:67-68`).
|
||||||
|
- **Setting and clearing the deliberate-off time each record an info event** — `host_deliberate_off`,
|
||||||
|
`host_deliberate_off_ended` — on the customer's event list; neither is mailed (`hub/internal/api/deliberate_off.go:12-13`).
|
||||||
|
- **The confirmation word is accepted in three forms:** „KIKAPCSOLÁS", the unaccented „KIKAPCSOLAS" (a foreign keyboard)
|
||||||
|
and „TURN OFF" (`felhom-controller/controller/internal/web/poweroff_handlers.go:28-29`).
|
||||||
|
|
||||||
@@ -165,7 +165,14 @@ directory) with that tag. Facts the link code needs:
|
|||||||
**[FACT]** An unlisted writable `${…_PATH}` bind is MANDATORY (every tier, off-site included); a plain host path is in
|
**[FACT]** An unlisted writable `${…_PATH}` bind is MANDATORY (every tier, off-site included); a plain host path is in
|
||||||
no capture set (`stacks/classify_binds.go` `classifyRoot`). Measured over the live templates: Transmission's unlisted
|
no capture set (`stacks/classify_binds.go` `classifyRoot`). Measured over the live templates: Transmission's unlisted
|
||||||
`watch` bind goes off-site; the download folders are `optional`, so a second drive gets a nightly copy of every
|
`watch` bind goes off-site; the download folders are `optional`, so a second drive gets a nightly copy of every
|
||||||
download (R-1019). **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
|
download (R-1019). *(Fixed since: catalog `c50adfc` marks the download and watch folders `excluded` —
|
||||||
|
`app-catalog-felhom.eu/templates/transmission/.felhom.yml:46-51`; noted 2026-10-24.)* **[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1283]:** the same commit also
|
||||||
|
marks the download folders of NZBGet, SABnzbd and MeTube `excluded` (`templates/{nzbget,sabnzbd,metube}/.felhom.yml`).
|
||||||
|
NZBGet and SABnzbd stay excluded (their files can be fetched again). The pick for MeTube: `optional`, so the
|
||||||
|
second-drive copy carries it and the off-site copy does not — a MeTube video can be gone from its source site.
|
||||||
|
**Built in the DRILL catalog 2026-10-25 night (`app-catalog-drill` `6733764`, not live):** `class: optional`, and the
|
||||||
|
download-binds gate allows `optional` only for an app named with its ruling (`OPTIONAL_BY_RULING`); mandatory and
|
||||||
|
unlisted stay refused. The live push is the operator's (the Monday list). **[DESIGN]** The per-drive media bind is declared `excluded` for every media app — the family's
|
||||||
media is no single app's data; whether the library gets its own household-level backup is a separate decision.
|
media is no single app's data; whether the library gets its own household-level backup is a separate decision.
|
||||||
|
|
||||||
## 8. Removal and export assume one drive per app
|
## 8. Removal and export assume one drive per app
|
||||||
@@ -192,6 +199,13 @@ household's existing files under `media/torrents` stay where they are.
|
|||||||
Evidence: `audits/media-drive-events-2026-10-14/` (K1–K7). „Idempotent" = a second run changes nothing (K2: run 2 left every entry, folder and library as it
|
Evidence: `audits/media-drive-events-2026-10-14/` (K1–K7). „Idempotent" = a second run changes nothing (K2: run 2 left every entry, folder and library as it
|
||||||
was). „Household" = a step only the household can take.
|
was). „Household" = a step only the household can take.
|
||||||
|
|
||||||
|
> *Dated note 2026-10-24 (review round 2): this table is the 2026-10-14 MEASUREMENT („today" = then). As built (§11):
|
||||||
|
> on a drive event the controller only REWRITES the binds; the media apps restart when the household presses
|
||||||
|
> „Újraindítás most" or „Később" (03:30) — the file browser alone recreates itself (`internal/stacks/media_library.go`,
|
||||||
|
> `internal/web/media_events.go`); a safe removal keeps the drive REGISTERED and its return re-binds with no household
|
||||||
|
> step (`SetEjected`, `media_events.go`); and since controller `b01bb4b` (unreleased) neither restart recreates an app
|
||||||
|
> the household stopped (R-721).*
|
||||||
|
|
||||||
| event → app | action | measured call | idempotent | household | evidence |
|
| event → app | action | measured call | idempotent | household | evidence |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
| **added** — every media app | the compose gains the drive's bind → `up -d` recreates it (the compose CHANGED, so today's restart works) | controller compose `up -d` | yes | no | K5-1: 1–10 s per app |
|
| **added** — every media app | the compose gains the drive's bind → `up -d` recreates it (the compose CHANGED, so today's restart works) | controller compose `up -d` | yes | no | K5-1: 1–10 s per app |
|
||||||
@@ -258,3 +272,14 @@ save path when its client is saved before the wiring; **a Plex status probe (`/l
|
|||||||
unclaimed Plex because Plex answers 401 for a moment while it starts — never gate on a status a starting app also
|
unclaimed Plex because Plex answers 401 for a moment while it starts — never gate on a status a starting app also
|
||||||
returns** (G7f).
|
returns** (G7f).
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
**[DESIGN — the operator picked a code change 2026-10-25 (D44); row R-1282]:** the „Később"
|
||||||
|
03:30 media restart is an IN-MEMORY timer — a controller restart forgets it and only the notice stays
|
||||||
|
(`internal/web/media_events.go` `handleMediaRestart`, `time.AfterFunc`). The pick: the controller stores the due
|
||||||
|
time and runs the restart after a restart of its own. **Built 2026-10-25 night (controller `b5863a9`, unreleased):**
|
||||||
|
`settings.media_restart_later_at`; at start `ResumeMediaLater` re-arms it, runs it at once when it passed while the
|
||||||
|
controller was down, and clears it once it ran.
|
||||||
|
|
||||||
|
**[DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]:** the Sonarr/Radarr API key is READ from
|
||||||
|
`/config/config.xml`; the „set at install" option §6 calls cleaner was not built (`internal/mediawire/mediawire.go:336-345`).
|
||||||
@@ -15,8 +15,8 @@
|
|||||||
| Place | Found by | Identified by | Mounted | What lives there | The backup page says |
|
| Place | Found by | Identified by | Mounted | What lives there | The backup page says |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
| Host root / PVE `local` | PVE storage `local` [FACT] A/internal/storage/observe.go:424; never a candidate [FACT] A/internal/storage/role.go:53-79 | `path:` / `store:local` [FACT] A/internal/storage/durableid.go:54-67; role system | host `/`, `/var/lib/vz` | the **local whole-guest copies** (vzdump, retention 3) [FACT] E/scripts/felhom-host-install.sh:730-734; `07` §6.1 | `backup.target.degraded` „…ugyanazon a lemezen van, mint a rendszer…" [FACT] C/internal/web/backup_target_offer.go:78-100 |
|
| Host root / PVE `local` | PVE storage `local` [FACT] A/internal/storage/observe.go:424; never a candidate [FACT] A/internal/storage/role.go:53-79 | `path:` / `store:local` [FACT] A/internal/storage/durableid.go:54-67; role system | host `/`, `/var/lib/vz` | the **local whole-guest copies** (vzdump, retention 3) [FACT] E/scripts/felhom-host-install.sh:730-734; `07` §6.1 | `backup.target.degraded` „…ugyanazon a lemezen van, mint a rendszer…" [FACT] C/internal/web/backup_target_offer.go:78-100 |
|
||||||
| Guest `mp0` `/var/lib/felhom` with folders `docker/` and `sys_drive/` (bound to `/var/lib/docker`, `/mnt/sys_drive`) | config default `system_data_path=/mnt/sys_drive` [FACT] C/internal/config/config.go:166 | path only; „same disk" = `st_dev` [FACT] C/internal/system/mounts_linux.go:225-245 | thin LV in `local-lvm`, `backup=1` [FACT] A/configs/build-golden.sh:105, :221-225; measured on all 3 boxes | app volumes; Tier-1 of apps with no drive [FACT] C/internal/backup/backup.go:429-438; Tier-2 state-only fallback [FACT] C/internal/backup/tier2.go:157-178; **inside every whole-guest copy** | „Csak egy másolat készül (nincs második meghajtó)…" — a Go constant, not i18n [FACT] backup.go:359-361, :1387-1410 |
|
| Guest `mp0` `/var/lib/felhom` with folders `docker/` and `sys_drive/` (bound to `/var/lib/docker`, `/mnt/sys_drive`) | config default `system_data_path=/mnt/sys_drive` [FACT] C/internal/config/config.go:166 | path only; „same disk" = `st_dev` [FACT] C/internal/system/mounts_linux.go:225-245 | thin LV in `local-lvm`, `backup=1` [FACT] A/configs/build-golden.sh:105, :221-225; measured on all 3 boxes | app volumes; Tier-1 of apps with no drive [FACT] C/internal/backup/backup.go:429-438; Tier-2 state-only fallback [FACT] C/internal/backup/tier2.go:157-178; **inside every whole-guest copy** | „Csak egy másolat készül (nincs második meghajtó)…" — *corrected 2026-10-24:* the i18n keys `note.backup.single_copy` / `note.backup.single_copy_unproven`, picked by the agent's verdict [FACT] C/internal/backup/backup.go:1485-1494 |
|
||||||
| Enrolled drive (`/mnt/<name>` on the host → `/mnt/felhom-drives/<name>`) | `GET /disks/candidates`: **whole disks only** [FACT] A/internal/storage/candidates.go:33-53 | mount: `uuid:<fs-uuid>`; wipe: `byid:` wwn > nvme-eui > … [FACT] A/internal/storage/durable_device.go:32-51 | systemd unit by fs-UUID through `FELHOM_MOUNT` [FACT] A/internal/storage/mountunit.go:50-65; host bind into `mp8` | `HDD_PATH` apps, media binds [FACT] C/internal/stacks/media_library.go:24-33; Tier-1/Tier-2; optionally the whole-guest target `felhom-backup`; **not in vzdump** (`mp8` is a host bind) [FACT] `07`:477-480 | Tier-2 badges; „Nincs 2. (off-drive) másolat" [FACT] C/internal/i18n/locales/hu.json:303 |
|
| Enrolled drive (`/mnt/<name>` on the host → `/mnt/felhom-drives/<name>`) | `GET /disks/candidates`: whole disks, each carrying its nested partition candidates (empty / filesystem / in use; corrected 2026-10-24) [FACT] A/internal/storage/candidates.go:84-90 | mount: `uuid:<fs-uuid>`; wipe: `byid:` wwn > nvme-eui > … [FACT] A/internal/storage/durable_device.go:32-51 | systemd unit by fs-UUID through `FELHOM_MOUNT` [FACT] A/internal/storage/mountunit.go:50-65; host bind into `mp8` | `HDD_PATH` apps, media binds [FACT] C/internal/stacks/media_library.go:24-33; Tier-1/Tier-2; optionally the whole-guest target `felhom-backup`; **not in vzdump** (`mp8` is a host bind) [FACT] `07`:477-480 | Tier-2 badges; „Nincs 2. (off-drive) másolat" [FACT] C/internal/i18n/locales/hu.json:303 |
|
||||||
| Network share | customer input [FACT] A/internal/localapi/netstorage.go:36-40 | none („no durable-id, no SMART") [FACT] netstorage.go:15-19 | `.automount` under `/mnt/felhom-drives/<name>` [FACT] A/internal/storage/netmount.go:18-31 | browse only; never media, never Tier-2 [FACT] media_library.go:83; tier2.go:141-143 | `err.backup.halozati_tarolo_nem_lehet_a_2` [FACT] tier2.go:37-40 |
|
| Network share | customer input [FACT] A/internal/localapi/netstorage.go:36-40 | none („no durable-id, no SMART") [FACT] netstorage.go:15-19 | `.automount` under `/mnt/felhom-drives/<name>` [FACT] A/internal/storage/netmount.go:18-31 | browse only; never media, never Tier-2 [FACT] media_library.go:83; tier2.go:141-143 | `err.backup.halozati_tarolo_nem_lehet_a_2` [FACT] tier2.go:37-40 |
|
||||||
| USB disk | same as an enrolled drive; transport never decides [FACT] C/internal/settings/settings.go:609-614 | same; the bridge may pass the real identity (N100: yes) [FACT] spike case 4a | same | same | same; SMART may be missing → „Nincs adat" |
|
| USB disk | same as an enrolled drive; transport never decides [FACT] C/internal/settings/settings.go:609-614 | same; the bridge may pass the real identity (N100: yes) [FACT] spike case 4a | same | same | same; SMART may be missing → „Nincs adat" |
|
||||||
|
|
||||||
@@ -126,7 +126,7 @@ binds both. **Creating or deleting partitions: never from the dashboard in this
|
|||||||
## 7. The installer
|
## 7. The installer
|
||||||
|
|
||||||
**[FACT]** Today `step_grows` gives the guest's one data volume (`mp0`) 70 / 130 / 250 GiB by the thin pool's free space
|
**[FACT]** Today `step_grows` gives the guest's one data volume (`mp0`) 70 / 130 / 250 GiB by the thin pool's free space
|
||||||
(<150 / ≥150 / ≥300 GiB), and nothing above 250 (E/scripts/felhom-host-install.sh:2261-2275). Measured pools: 128 GB
|
(<150 / ≥150 / ≥300 GiB), and nothing above 250 (E/scripts/felhom-host-install.sh `step_grows`, now at :2475 — line corrected 2026-10-24). Measured pools: 128 GB
|
||||||
disk → 54 GiB, 200 GB → 113 GiB, 512 GB → 349 GiB (PVE default layout; root capped at 96 GiB).
|
disk → 54 GiB, 200 GB → 113 GiB, 512 GB → 349 GiB (PVE default layout; root capped at 96 GiB).
|
||||||
|
|
||||||
**[DESIGN] The rule:** `mp0` = 70 GiB (pool < 300), 130 (300–800), 250 (≥ 800). Reserve = **max(16 GiB, 10 % of the
|
**[DESIGN] The rule:** `mp0` = 70 GiB (pool < 300), 130 (300–800), 250 (≥ 800). Reserve = **max(16 GiB, 10 % of the
|
||||||
@@ -231,10 +231,25 @@ sizes ruled above in §7):
|
|||||||
the public installer 1.33.0, agent 0.158.1 and golden 0.318.0 vouched) → data volume 100 GiB, internal place 213 GiB,
|
the public installer 1.33.0, agent 0.158.1 and golden 0.318.0 vouched) → data volume 100 GiB, internal place 213 GiB,
|
||||||
registered as the default drive with no press; Jellyfin installed with no extra step onto it; a whole-box copy holds nothing
|
registered as the default drive with no press; Jellyfin installed with no extra step onto it; a whole-box copy holds nothing
|
||||||
of it; partitions formatted and used through the wizard; the honest backup offer on a VM. **Not as written:** an empty part
|
of it; partitions formatted and used through the wizard; the honest backup offer on a VM. **Not as written:** an empty part
|
||||||
asks for the typed confirmation (the probe reads the parent's partition table — R-1093); a 200 GB disk gets no internal
|
asks for the typed confirmation (the probe reads the parent's partition table — R-1093) *(fixed since in agent v0.160.0:
|
||||||
|
the parent's PTTYPE is set aside when `blkid -p` proves the node is a partition of that scheme — `git tag --contains
|
||||||
|
f7b7308`; noted 2026-10-24)*; a 200 GB disk gets no internal
|
||||||
place under the ruled sizes (15 GiB < 20).
|
place under the ruled sizes (15 GiB < 20).
|
||||||
|
|
||||||
Also in build B: R-1084 (the whole-box backup offer asks the same-disk question) and R-1082 (`/mnt/sys_drive` is never
|
Also in build B: R-1084 (the whole-box backup offer asks the same-disk question) and R-1082 (`/mnt/sys_drive` is never
|
||||||
offered to attach, by device). A bundle that adds a path needs a step bundle on boxes with an older wrapper (R16, R-880):
|
offered to attach, by device). A bundle that adds a path needs a step bundle on boxes with an older wrapper (R16, R-880):
|
||||||
0.158.1 was delivered through `0.158.1-step1`.
|
0.158.1 was delivered through `0.158.1-step1`.
|
||||||
|
|
||||||
|
## Decisions in code no section records [DESIGN — confirmed by the operator 2026-10-25 (D44 = all picks, R-1265)]
|
||||||
|
|
||||||
|
Read at `file:line` on `main` by the main session:
|
||||||
|
|
||||||
|
- **`systemd-detect-virt --vm` is read once per agent process**; empty output becomes `unknown`, which is not `none`,
|
||||||
|
so every disk then reads unknown/vm — the fail-closed direction (`felhom-agent/internal/storage/identity.go:195-205`).
|
||||||
|
- **A path the controller cannot read counts as the SAME disk** (R-668, v0.269.0): no second copy is ever claimed on
|
||||||
|
it (`felhom-controller/controller/internal/backup/backup.go:1436-1450`).
|
||||||
|
- **The internal place („Belső tárhely") refuses eject, decommission and wipe** — the controller (`refuseInternalLifecycle`)
|
||||||
|
and the agent both — and allows moving its data to another drive (`internal/web/internal_place.go:20-26`).
|
||||||
|
- **The internal place's size subtracts EVERY thin volume already promised in the pool**, not only the 32 GiB rootfs and
|
||||||
|
the data volume (`felhom-agent/configs/felhom-internal-place:11-14`).
|
||||||
|
|
||||||
@@ -1,5 +1,11 @@
|
|||||||
# `05-hub-architecture.md` — critical review (grounded against felhom-hub v0.6.3 source + Parts 01–04)
|
# `05-hub-architecture.md` — critical review (grounded against felhom-hub v0.6.3 source + Parts 01–04)
|
||||||
|
|
||||||
|
> **HISTORICAL (banner added 2026-10-24, review round 2).** A dated review of hub v0.6.3. Many of its facts have moved
|
||||||
|
> since — for example the stale threshold is 45 min by config (code default 30 min), the operator cooldown is per event
|
||||||
|
> type (`notify/dispatcher.go` `operatorCooldownFor`), and the hub→controller push was retired in v0.26.0. Read
|
||||||
|
> `05-hub-architecture.md` and `08-alarm-ladder.md` for the current behaviour; this file is kept as the record of what
|
||||||
|
> the review found then.
|
||||||
|
|
||||||
Method: every claim about the existing hub was checked against `felhom.eu/hub/` source; every
|
Method: every claim about the existing hub was checked against `felhom.eu/hub/` source; every
|
||||||
cross-doc claim against Parts 01/03/04. Citations are `file:line`. Severity: **blocking** (wrong /
|
cross-doc claim against Parts 01/03/04. Citations are `file:line`. Severity: **blocking** (wrong /
|
||||||
breaks an assumption) · **should-fix** (real gap or contradiction, low blast) · **minor**.
|
breaks an assumption) · **should-fix** (real gap or contradiction, low blast) · **minor**.
|
||||||
|
|||||||
@@ -5,6 +5,21 @@
|
|||||||
> No matrix is designed here, no target architecture is proposed, and no gap is ranked.
|
> No matrix is designed here, no target architecture is proposed, and no gap is ranked.
|
||||||
>
|
>
|
||||||
> **This is the input to a redesign, not the redesign.**
|
> **This is the input to a redesign, not the redesign.**
|
||||||
|
>
|
||||||
|
> **WHAT HAS MOVED SINCE (banner added 2026-10-24, review round 2 — this inventory records 2026-07-28 and is kept as
|
||||||
|
> it was; each line below was read at `file:line` on `main` that day):**
|
||||||
|
> - The guest has ONE data volume, `mp0` at `/var/lib/felhom`, whose `docker/` and `sys_drive/` folders are bound in the
|
||||||
|
> guest; there is no `mp1` (`felhom-agent/configs/build-golden.sh:31-38,99`) — the „rootfs + mp0 + mp1" lines are history.
|
||||||
|
> - A schema-2 recovery unit (D5) carries the PORTABLE secrets; a restore takes the database password from the unit
|
||||||
|
> (R-1106) — `felhom-controller/controller/internal/backup/recovery_unit.go:24-27`, `restore_dbcreds.go`.
|
||||||
|
> - The second-drive copy's `recovery-unit/` IS read: `RestoreTier2Unit` restores the full unit from the mirror (R-102)
|
||||||
|
> — `internal/backup/tier2_restore.go:197`.
|
||||||
|
> - The off-site reconstitution replays the snapshot's named-volume tars, volumes first (R-354) —
|
||||||
|
> `internal/backup/offbox_reconstitute.go`.
|
||||||
|
> - ep0's per-household prune jobs are keep-weekly 2 at 03:30 (R-1162/R-1165, `scripts/felhom-tenantsync.sh:187-197`);
|
||||||
|
> the local whole-guest tier is keep-daily=3 (R-1149, `felhom-agent/internal/config/config.go:553-559`).
|
||||||
|
> - restic is the pinned official release, checked by version and checksum (R-1105, `felhom-controller/controller/Dockerfile:43-46`),
|
||||||
|
> no longer Debian's 0.14.0.
|
||||||
|
|
||||||
## 0. Method, baselines, and what "verified" means here
|
## 0. Method, baselines, and what "verified" means here
|
||||||
|
|
||||||
|
|||||||
@@ -25,6 +25,8 @@
|
|||||||
|
|
||||||
**Gap / roadmap (as written):** Memory-guard FIRING is not live-shown (T-RES-MEMGUARD never fired: ample RAM / auth-walled) — implemented + unit-level only
|
**Gap / roadmap (as written):** Memory-guard FIRING is not live-shown (T-RES-MEMGUARD never fired: ample RAM / auth-walled) — implemented + unit-level only
|
||||||
|
|
||||||
|
**Re-checked 2026-10-24 (catalog sweep, `audits/catalog-sweep-2026-10-24/README.md`):** a fresh box from the public ISO 1.30.0, controller 0.342.0, Tester4. Every catalog app through `POST /api/stacks/<app>/deploy` one at a time, timed to the front door (median 61 s, Cal.com 278 s), then first login, one item, backup, keep-data remove, reinstall, unit restore, remove with data. 62 installed and answered; Wanderer installed but cannot reach its database without DNS (R-739); plant-it and wger are not offered (lifecycle, by design). 52 work, 10 work with a problem (R-1269–R-1277), 0 broken. The memory guard still did not fire (one app at a time) — the gap above stands.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
<a id="b2"></a>
|
<a id="b2"></a>
|
||||||
@@ -379,3 +381,38 @@
|
|||||||
**Gap / roadmap (as written):** Remaining ~39 apps are legacy-class by design (unit-only offsite)
|
**Gap / roadmap (as written):** Remaining ~39 apps are legacy-class by design (unit-only offsite)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## B23 — An urgent security fix reaches the boxes at night (the security-fix step)
|
||||||
|
|
||||||
|
*Map row: [B23](../00-capability-map.md#b23) · added 2026-10-23.*
|
||||||
|
|
||||||
|
**Scenario:** an app has a published hole (a CVE or GHSA id with its notice) and a fixed upstream version. CC walks the
|
||||||
|
step on scratch 9202 through the product's Update (`box_walk.py --security`: install at FROM, the guarded Update, the
|
||||||
|
four version observables at TO, a 600 s memory soak), and the only writer turns a proven walk into a `security_fix`
|
||||||
|
ladder step. Every box on controller >= 0.341.0 takes it at night with the usual guards (backup first, undo on a
|
||||||
|
failed health check, held / stopped / needs_person / W+5h respected).
|
||||||
|
|
||||||
|
**Trigger (2026-10-26, R-1303):** a mail from the nightly security watch („[Felhom security] …", `08` §11) starts the
|
||||||
|
lane: it names the program, our version and the fixed version. An app notice goes this lane; a box program's (traefik,
|
||||||
|
cloudflared, the file manager) is a controller release — the cloudflared 2026.10.3 run (R-1300) is the worked example.
|
||||||
|
|
||||||
|
**Components:** catalog `3898593` (`ladder.py check_security`, the two test-record gates, `upgrade-test.py --write-ladder
|
||||||
|
--security`, `box_walk.py --security`); controller v0.341.0 (`legCandidate`).
|
||||||
|
|
||||||
|
**Status:** **PROVEN-LIVE** on 9202 with the drill catalog, 2026-10-23.
|
||||||
|
|
||||||
|
**Evidence:** `audits/security-lane-2026-10-23/` — the negative control (TO `busybox:1.36.1`) `failed` and the writer
|
||||||
|
refused it; Plex 1.43.4.10903 → 1.43.5.11029 `proven`; on 0.340.0 the night leg logged `plex: skipped — not_proven` and
|
||||||
|
a hand press took the step; on 0.341.0-rc.1 the night leg logged `plex: security fix CVE-2026-96652,… taken` and ended
|
||||||
|
`done` after a second-drive copy and an undo copy.
|
||||||
|
|
||||||
|
**Gap:** an app with no box fixture (Plex today, R-1238) is checked only by health, version and the soak — no data is
|
||||||
|
read back. `files_may_change` is a bench measurement and reads `false` when no bench ran (decision 255 item 7).
|
||||||
|
|
||||||
|
**Batch use (2026-10-27, `audits/security-app-updates-2026-10-27/`):** 14 security-fix steps in three batches, each
|
||||||
|
pushed live after the operator's yes at a batch STOP (catalog `06e603e`, `7b0071e`, `65a6b11`); 12 read the seed back
|
||||||
|
after the update, 2 are `seed: none`. demo-hp's 9201 read `06e603e` at once and holds RomM's step for its night leg
|
||||||
|
(`live-readback-batch1-9201.txt`). Notices with no CVE/GHSA id go as normal steps in the same batches (wishlist,
|
||||||
|
bookstack). Four stale fixtures were found and fixed by the walks' `inconclusive` C1 (`09` decision 255, batch note).
|
||||||
|
|
||||||
|
---
|
||||||
@@ -142,7 +142,9 @@
|
|||||||
|
|
||||||
**Evidence (as written):** New FIRST sidebar page `/launcher`: colored tiles (deterministic slug color or `.felhom.yml` `brand_color`) + white glyph/monogram, one per openable app (tile ⟺ „Megnyitás" — subdomain presence is the single criterion; controller excluded). Operational → `<a target=_blank>` to the public URL; stopped → greyed + state badge, no link. `/` stays the Vezérlőpult. Endpoint-level + render-test verified; `felhom-controller/REPORT.md` (2026-07-24)
|
**Evidence (as written):** New FIRST sidebar page `/launcher`: colored tiles (deterministic slug color or `.felhom.yml` `brand_color`) + white glyph/monogram, one per openable app (tile ⟺ „Megnyitás" — subdomain presence is the single criterion; controller excluded). Operational → `<a target=_blank>` to the public URL; stopped → greyed + state badge, no link. `/` stays the Vezérlőpult. Endpoint-level + render-test verified; `felhom-controller/REPORT.md` (2026-07-24)
|
||||||
|
|
||||||
**Gap / roadmap (as written):** Live operator click-through of a real tile → app pending (browser automation not available on DooPlex). Follow-up: curate `brand_color` for top catalog apps (R-72). Sharing the launcher outside the household is now the capability-URL guest link — see the row below
|
**Walked 2026-10-28 night (9202, endpoint-level — the page and links the browser receives):** `/launcher` answered 200 with one tile per openable app (`hike`, `paperless`, `tasks`), each a link to the app's public address; the controller has no tile. `audits/night-2026-10-28/part5-claims/launcher-walk.*`.
|
||||||
|
|
||||||
|
**Gap / roadmap (as written):** Live operator click-through of a real tile → app pending (browser automation not available on DooPlex; the endpoint walk above stands in). Follow-up: curate `brand_color` for top catalog apps (R-72). Sharing the launcher outside the household is now the capability-URL guest link — see the row below
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -157,7 +159,7 @@
|
|||||||
|
|
||||||
**Status (as written):** **IMPLEMENTED**
|
**Status (as written):** **IMPLEMENTED**
|
||||||
|
|
||||||
**Evidence and gap (as written — this row's text carries a `|` outside code, so its last cells are kept together as they stood):** 160-bit `crypto/rand` token, constant-time match (empty stored = disabled = byte-identical to the mux default 404); guest headers `noindex`/`no-referrer`/`no-store`; optional SEPARATE bcrypt share password + its own per-IP attempt map; signed cookie `HMAC(token`\ | `passwordHash)` keyed with `session_secret` (rotate-token OR change-password invalidates all cookies); token redacted in logs (`/s/<redacted>`). Groups A–G (14 tests) + 3 red-proofs; **§13 endpoint-level live validation on 9201 all-pass** (`felhom-controller/REPORT.md` 2026-07-24). **Design ruling: member accounts SUPERSEDED by this capability-URL model; per-member tile visibility parked under the SSO arc (R-15).** | Full operator browser click-through + a validation doc pending → then PROVEN-LIVE. Accepted residuals: link-preview crawlers fetch once (noindex prevents indexing); reverse-proxy/CF access logs hold the path (ops-tier); the modal link carries the request Host (LAN-IP admin ⇒ LAN-IP link)
|
**Evidence and gap (as written — this row's text carries a `|` outside code, so its last cells are kept together as they stood):** 160-bit `crypto/rand` token, constant-time match (empty stored = disabled = byte-identical to the mux default 404); guest headers `noindex`/`no-referrer`/`no-store`; optional SEPARATE bcrypt share password + its own per-IP attempt map; signed cookie `HMAC(token`\ | `passwordHash)` keyed with `session_secret` (rotate-token OR change-password invalidates all cookies); token redacted in logs (`/s/<redacted>`). Groups A–G (14 tests) + 3 red-proofs; **§13 endpoint-level live validation on 9201 all-pass** (`felhom-controller/REPORT.md` 2026-07-24). **Design ruling: member accounts SUPERSEDED by this capability-URL model; per-member tile visibility parked under the SSO arc (R-15).** | **Walked 2026-10-28 night on 9202 (endpoint-level, `audits/night-2026-10-28/part5-claims/launcher-walk.*`):** switched on from the launcher (303 `flash.share.enabled`); the link opened with NO session → 200, the household's app tiles, no admin link; a wrong token → 404; rotate → the old link 404, the new one 200; switched off → 404 (it was off before the walk). NOT-A-FINDING: an unknown non-`/s/` route answers an anonymous visitor 302 to the login page, so a wrong token is told apart from an unknown ROUTE (404 vs 302) — what the design hides is a wrong token from a disabled one, which both answer 404. → PROVEN-LIVE. Accepted residuals: link-preview crawlers fetch once (noindex prevents indexing); reverse-proxy/CF access logs hold the path (ops-tier); the modal link carries the request Host (LAN-IP admin ⇒ LAN-IP link)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -219,3 +219,25 @@ Tester 1 run controller 0.311.0. An Immich image bump the repair was not proven
|
|||||||
**Evidence:** `audits/immich-repair-2026-10-14/` — spike A on 9202 (the five candidates), the scripts on real damage
|
**Evidence:** `audits/immich-repair-2026-10-14/` — spike A on 9202 (the five candidates), the scripts on real damage
|
||||||
(9202), the live cut on demo-hp 9201 (D/). **Not exercised live:** a lost original (the cut lost none); pinned by unit
|
(9202), the live cut on demo-hp 9201 (D/). **Not exercised live:** a lost original (the cut lost none); pinned by unit
|
||||||
tests and the 9202 run.
|
tests and the 9202 run.
|
||||||
|
|
||||||
|
## F13 — A hacked app is noticed (the upload and CPU watch, the weekly line test)
|
||||||
|
|
||||||
|
*Map row: [F13](../00-capability-map.md#f13) · added 2026-10-24.*
|
||||||
|
|
||||||
|
**Scenario:** an app on a household box sends far more than it says it sends, or burns CPU with no transcode to explain
|
||||||
|
it (the Plex2Shell shape). Once a minute the controller reads every app's exact counters and asks the app (qBittorrent,
|
||||||
|
Transmission, Plex); not explained for 15 / 30 minutes → one operator-only alarm, one clear when it stops. Once a week
|
||||||
|
the host's agent measures the upload line with a 10-second iperf3 upload to ep0 inside the tunnel; qBittorrent's default
|
||||||
|
upload limit is half of the tested line, never over the household's own limit.
|
||||||
|
|
||||||
|
**Components:** controller v0.342.0 (`internal/upwatch`, `agentapi.LineTest`), agent v0.164.0 (`internal/linetest`),
|
||||||
|
hub v0.169.0 (four operator-only events; the System page's „Upload line"), ep0 `felhom-iperf3.service`.
|
||||||
|
|
||||||
|
**Status:** **PROVEN-LIVE** — the alarm path on scratch 9202 with `0.342.0-rc.1` (2026-10-23: a 50 Mbit/s flood hidden
|
||||||
|
behind a real seed → one alarm after 15 min, one clear; a CPU burner in Plex → the same; a real seed → none); the line test
|
||||||
|
on the three test boxes (2026-10-24: 94.3 / 94.4 / 92.6 Mbit/s; „server busy" retried); the qBittorrent limit on
|
||||||
|
demo-felhom (5 789 696 B/s = 50 % of 92.6 Mbit/s, three channels agree).
|
||||||
|
|
||||||
|
**Evidence:** `audits/SPIKE-upload-watch-2026-10-23.md` §7; `audits/line-test-2026-10-24/`. **Not exercised live:** an
|
||||||
|
alarm reaching the operator's mailbox from a real box (the 9202 proof had no hub); the family's line on qBittorrent's
|
||||||
|
page on demo-felhom (pinned by a render test). **Gaps:** R-1250, R-1248.
|
||||||
@@ -97,15 +97,16 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
|
|||||||
## Zone 2 — the internet
|
## Zone 2 — the internet
|
||||||
|
|
||||||
- **Cloudflare** (grey): DNS for the customer domains and for felhom.eu; the **tunnels**; the **geo-block** (only Hungarian visitors); flood protection. The hub holds a Cloudflare key per customer zone and checks that it reaches only that customer's own domain.
|
- **Cloudflare** (grey): DNS for the customer domains and for felhom.eu; the **tunnels**; the **geo-block** (only Hungarian visitors); flood protection. The hub holds a Cloudflare key per customer zone and checks that it reaches only that customer's own domain.
|
||||||
- **ep0** (grey box with an orange stripe, a Hetzner cloud server in Germany): runs the **WireGuard endpoint** and the **off-site backup server (PBS)**. Inside, one datastore with **namespaces**:
|
- **ep0** (grey box with an orange stripe, a Hetzner cloud server in Germany): runs the **WireGuard endpoint**, the **off-site backup server (PBS)** and, since 2026-10-24, a small **line-speed test server** (iperf3) that answers only inside the tunnel — each box measures its upload once a week (R-1247). Inside, one datastore with **namespaces**:
|
||||||
- one namespace **per household** (whole-guest copies, each household's own key),
|
- one namespace **per household** (whole-guest copies, each household's own key),
|
||||||
with a NEUTRAL note under it (not a red gap box): "By design: each household holds the key to its own copies; the operator cannot open them without their recovery code." — *fix (c), 2026-10-09: this is a property of the design, not a gap, and a red box said the opposite of what it means.*
|
with a NEUTRAL note under it (not a red gap box): "By design: each household holds the key to its own copies; the operator cannot open them without their recovery code." — *fix (c), 2026-10-09: this is a property of the design, not a gap, and a red box said the opposite of what it means.*
|
||||||
- the **`operator`** namespace: two groups — **"hub database"** and **"DooPlex copy"** (code, password manager, secrets, signing keys).
|
- the **`operator`** namespace: two groups — **"hub database"** and **"DooPlex copy"** (code, password manager, secrets, signing keys).
|
||||||
|
- outside PBS, a plain folder: the **release copy** (images + agent packages, D33).
|
||||||
Prune jobs on ep0: households keep one copy per week for 2 weeks (`keep-weekly 2`, R-1162, 2026-10-19; every new household's job is made with its namespace, R-1165); `operator` keeps 14 daily + 8 weekly.
|
Prune jobs on ep0: households keep one copy per week for 2 weeks (`keep-weekly 2`, R-1162, 2026-10-19; every new household's job is made with its namespace, R-1165); `operator` keeps 14 daily + 8 weekly.
|
||||||
Mark ep0 with a small shield: "**Protected** — holds the only off-premises copy of real household data."
|
Mark ep0 with a small shield: "**Protected** — holds the only off-premises copy of real household data."
|
||||||
- **Hetzner Storage Box** (grey): the shared box for the households' per-app restic copies (Tier 3), one sub-account per household, add-only keys.
|
- **Hetzner Storage Box** (grey): the shared box for the households' per-app restic copies (Tier 3), one sub-account per household, add-only keys.
|
||||||
- **Mail service** (grey, "Resend"): every mail the hub sends — to the operator and to households.
|
- **Mail service** (grey, "Resend"): every mail the hub sends — to the operator and to households.
|
||||||
- **The public website** felhom.eu (served from DooPlex — Cloudflare DNS only, the traffic does not pass through it; MEASURED 2026-10-09: the public A record is the home address, not a Cloudflare one, and no `cf-ray` header comes back) and **iso.felhom.eu** (where households download the installer): draw them as two small web pages near Cloudflare. Do not draw where iso.felhom.eu is hosted.
|
- **The public website** felhom.eu, with the installer script (**served by Cloudflare Pages since 2026-10-30**, D35 / R-1228 — MEASURED: `server: cloudflare` and a `cf-ray` header; before, it was served from DooPlex with Cloudflare DNS only) and **iso.felhom.eu** (where households download the installer): draw them as two small web pages near Cloudflare. Do not draw where iso.felhom.eu is hosted.
|
||||||
|
|
||||||
## Zone 3 — the operator's home
|
## Zone 3 — the operator's home
|
||||||
|
|
||||||
@@ -115,14 +116,15 @@ Footnote: "W = the household's backup window start, 02:30 by default. Tiers 1–
|
|||||||
- **Vaultwarden** — the operator's password manager.
|
- **Vaultwarden** — the operator's password manager.
|
||||||
- **Prometheus + Alertmanager** — the operator's alarms.
|
- **Prometheus + Alertmanager** — the operator's alarms.
|
||||||
- **The signing keys** — they sign every agent update and config bundle a box accepts.
|
- **The signing keys** — they sign every agent update and config bundle a box accepts.
|
||||||
- **The website** (felhom.eu) and the contact form's mailer.
|
- **The contact form's mailer** (`api.felhom.eu`) — and, until 2026-11-06, the old copy of the website (the way back after the 2026-10-30 move to Cloudflare).
|
||||||
- **Local backups** — several restic sets on DooPlex's own disks (red outline: "all on the same machine").
|
- **Local backups** — several restic sets on DooPlex's own disks (red outline: "all on the same machine").
|
||||||
- **ep0-copy** — a nightly pull of the whole ep0 datastore back to DooPlex (ciphertext only; DooPlex cannot read households' copies).
|
- **ep0-copy** — a nightly pull of the whole ep0 datastore back to DooPlex (ciphertext only; DooPlex cannot read households' copies).
|
||||||
|
|
||||||
Arrows from DooPlex (dashed, orange → ep0):
|
Arrows from DooPlex (dashed, orange → ep0):
|
||||||
- **00:20 "DooPlex copy"**: the code (614 MB), Gitea's database and settings, the **password manager**, the nightly secrets export, the **signing keys**. Encrypted on DooPlex with the **DooPlex off-site key**. The key that writes it **cannot delete** old copies. Tested back every Sunday.
|
- **00:20 "DooPlex copy"**: the code (614 MB), Gitea's database and settings, the **password manager**, the nightly secrets export, the **signing keys**. Encrypted on DooPlex with the **DooPlex off-site key**. The key that writes it **cannot delete** old copies. Tested back every Sunday.
|
||||||
- **02:30 "Hub database"**: encrypted with the **hub-DB key**. Tested back every Sunday at 04:30.
|
- **02:30, 08:30, 14:30, 20:30 "Hub database"** (every 6 hours since 2026-10-23, D34): encrypted with the **hub-DB key**. Tested back every Sunday at 04:30.
|
||||||
- **Left out on purpose:** the container registry (27.7 GB) — "rebuilt from the code".
|
- **01:40 "Release copy"** (since 2026-10-23, D33): the newest 3 released images of the controller, hub and samba, and the newest 5 agent packages — not encrypted (nothing secret), into a plain folder on ep0 (`/srv/felhom-release-copy`, its own limited user, outside PBS), about 0.7 GB, capped at 3 GB.
|
||||||
|
- **Left out on purpose:** the rest of the container registry (older tags, test builds, non-Felhom images) — "rebuilt from the code".
|
||||||
Arrow ep0 → DooPlex (dashed, 05:00): "**ep0-copy** pull: a second copy of ep0 at home. Keeps 8 weekly copies. A daily job removes a deleted customer's copy within 30 days."
|
Arrow ep0 → DooPlex (dashed, 05:00): "**ep0-copy** pull: a second copy of ep0 at home. Keeps 8 weekly copies. A daily job removes a deleted customer's copy within 30 days."
|
||||||
|
|
||||||
**The test machines** (small, plain tiles near DooPlex, labelled "disposable"):
|
**The test machines** (small, plain tiles near DooPlex, labelled "disposable"):
|
||||||
@@ -145,7 +147,7 @@ Title: **"Who tells a box what to do"**
|
|||||||
## Bottom strip — "One night, hour by hour"
|
## Bottom strip — "One night, hour by hour"
|
||||||
|
|
||||||
A horizontal timeline from **00:00 to 09:00**, two rows:
|
A horizontal timeline from **00:00 to 09:00**, two rows:
|
||||||
- **DooPlex row:** 00:00 database dumps · **00:20 DooPlex copy → ep0** · **02:30 hub database → ep0** · 03:30 ep0 prunes households · 03:45 ep0 prunes `operator` · **05:00 ep0-copy pull → DooPlex** · 07:30 ep0-copy prune · 08:00 deleted-customer clean-up. Sundays: 04:30 restore tests.
|
- **DooPlex row:** 00:00 database dumps · **00:20 DooPlex copy → ep0** · **02:30 hub database → ep0** (and again at 08:30; every 6 h since D34 — corrected 2026-10-24) · 03:30 ep0 prunes households · 03:45 ep0 prunes `operator` · **05:00 ep0-copy pull → DooPlex** · 07:30 ep0-copy prune · 08:00 deleted-customer clean-up. Sundays: 04:30 restore tests.
|
||||||
- **Customer box row:** **02:30 Tier 1** · **03:30 Tier 2** · **04:15 Tier 3 off-site** → app updates → **04:30–08:30 whole-guest backup** (local daily, off-site weekly) → Debian fixes → (when approved) Proxmox, Docker, kernel restart. 05:00 the hub checks every box made its backup.
|
- **Customer box row:** **02:30 Tier 1** · **03:30 Tier 2** · **04:15 Tier 3 off-site** → app updates → **04:30–08:30 whole-guest backup** (local daily, off-site weekly) → Debian fixes → (when approved) Proxmox, Docker, kernel restart. 05:00 the hub checks every box made its backup.
|
||||||
- A small note: "Household mails about a kernel restart go out the day before, 09:00–20:00."
|
- A small note: "Household mails about a kernel restart go out the day before, 09:00–20:00."
|
||||||
|
|
||||||
|
|||||||
@@ -116,7 +116,7 @@ code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 1
|
|||||||
<div style="position: absolute; left: 1552px; top: 460px; width: 136px; height: 112px; box-sizing: border-box; padding: 8px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #5b6270; border-radius: 6px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>iso.felhom.eu</b><br>households download the installer</span></div>
|
<div style="position: absolute; left: 1552px; top: 460px; width: 136px; height: 112px; box-sizing: border-box; padding: 8px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #5b6270; border-radius: 6px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b>iso.felhom.eu</b><br>households download the installer</span></div>
|
||||||
<div style="position: absolute; left: 1552px; top: 580px; width: 136px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: center;">web pages</div>
|
<div style="position: absolute; left: 1552px; top: 580px; width: 136px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: center;">web pages</div>
|
||||||
<div style="position: absolute; left: 1272px; top: 738px; width: 400px; font-size: 17px; line-height: 26px; font-weight: 400; color: #051343; text-align: left;"><b style="font-size: 22px">ep0</b> — Hetzner cloud server, Germany</div>
|
<div style="position: absolute; left: 1272px; top: 738px; width: 400px; font-size: 17px; line-height: 26px; font-weight: 400; color: #051343; text-align: left;"><b style="font-size: 22px">ep0</b> — Hetzner cloud server, Germany</div>
|
||||||
<div style="position: absolute; left: 1272px; top: 766px; width: 400px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;">WireGuard endpoint + off-site backup server (PBS). One datastore, split into namespaces:</div>
|
<div style="position: absolute; left: 1272px; top: 766px; width: 400px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;">WireGuard endpoint + off-site backup server (PBS) + a tunnel-only line-speed test (iperf3). One datastore, split into namespaces:</div>
|
||||||
<div style="position: absolute; left: 1272px; top: 818px; width: 192px; height: 150px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #0a3d73; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><span style="display: inline-block; min-width: 26px; padding: 0 6px; box-sizing: border-box; height: 26px; border-radius: 13px; background: #0a3d73; color: #ffffff; font-weight: 800; text-align: center; line-height: 26px; font-size: 16px; vertical-align: middle">G</span> <b>One namespace per household</b><br>whole-guest copies, each household's own key<br><b>keeps one per week, 2 weeks</b></span></div>
|
<div style="position: absolute; left: 1272px; top: 818px; width: 192px; height: 150px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #0a3d73; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><span style="display: inline-block; min-width: 26px; padding: 0 6px; box-sizing: border-box; height: 26px; border-radius: 13px; background: #0a3d73; color: #ffffff; font-weight: 800; text-align: center; line-height: 26px; font-size: 16px; vertical-align: middle">G</span> <b>One namespace per household</b><br>whole-guest copies, each household's own key<br><b>keeps one per week, 2 weeks</b></span></div>
|
||||||
<div style="position: absolute; left: 1480px; top: 818px; width: 192px; height: 150px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b><code style="font-size: 16px">operator</code> namespace</b><br>groups “hub database” and “DooPlex copy”<br><b>keeps 14 daily + 8 weekly</b></span></div>
|
<div style="position: absolute; left: 1480px; top: 818px; width: 192px; height: 150px; box-sizing: border-box; padding: 10px; display: flex; align-items: center; justify-content: flex-start; gap: 10px; text-align: left; background: #ffffff; border: 2px solid #a85400; border-radius: 8px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343"><span><b><code style="font-size: 16px">operator</code> namespace</b><br>groups “hub database” and “DooPlex copy”<br><b>keeps 14 daily + 8 weekly</b></span></div>
|
||||||
<div style="position: absolute; left: 1272px; top: 974px; width: 400px; font-size: 15px; line-height: 19px; font-weight: 400; color: #4a5670; text-align: left;"><b>By design:</b> each household holds the key to its own copies; the operator cannot open them without the household’s recovery code.</div>
|
<div style="position: absolute; left: 1272px; top: 974px; width: 400px; font-size: 15px; line-height: 19px; font-weight: 400; color: #4a5670; text-align: left;"><b>By design:</b> each household holds the key to its own copies; the operator cannot open them without the household’s recovery code.</div>
|
||||||
@@ -133,7 +133,7 @@ code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 1
|
|||||||
<div style="position: absolute; left: 1720px; top: 706px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: center;">The hub never connects into a box.</div>
|
<div style="position: absolute; left: 1720px; top: 706px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #4a5670; text-align: center;">The hub never connects into a box.</div>
|
||||||
<div style="position: absolute; left: 1720px; top: 784px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">00:20 DooPlex copy</div>
|
<div style="position: absolute; left: 1720px; top: 784px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">00:20 DooPlex copy</div>
|
||||||
<div style="position: absolute; left: 1720px; top: 824px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #a85400; text-align: center;">DooPlex off-site key</div>
|
<div style="position: absolute; left: 1720px; top: 824px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #a85400; text-align: center;">DooPlex off-site key</div>
|
||||||
<div style="position: absolute; left: 1720px; top: 872px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">02:30 hub database</div>
|
<div style="position: absolute; left: 1720px; top: 872px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #a85400; text-align: center;">every 6 h: hub database</div>
|
||||||
<div style="position: absolute; left: 1720px; top: 912px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #a85400; text-align: center;">hub-DB key</div>
|
<div style="position: absolute; left: 1720px; top: 912px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #a85400; text-align: center;">hub-DB key</div>
|
||||||
<div style="position: absolute; left: 1720px; top: 962px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #0a3d73; text-align: center;">05:00 ep0-copy pull</div>
|
<div style="position: absolute; left: 1720px; top: 962px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 800; color: #0a3d73; text-align: center;">05:00 ep0-copy pull</div>
|
||||||
<div style="position: absolute; left: 1720px; top: 1002px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #0a3d73; text-align: center;">ciphertext only</div>
|
<div style="position: absolute; left: 1720px; top: 1002px; width: 184px; font-size: 16px; line-height: 20px; font-weight: 400; color: #0a3d73; text-align: center;">ciphertext only</div>
|
||||||
@@ -191,7 +191,7 @@ code{font-family: ui-monospace, "SFMono-Regular", Menlo, monospace; font-size: 1
|
|||||||
<div style="position: absolute; left: 374px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #a85400; border: 2px solid #ffffff"></div>
|
<div style="position: absolute; left: 374px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #a85400; border: 2px solid #ffffff"></div>
|
||||||
<div style="position: absolute; left: 378px; top: 2040px; width: 220px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>00:20</b> DooPlex copy → ep0</div>
|
<div style="position: absolute; left: 378px; top: 2040px; width: 220px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>00:20</b> DooPlex copy → ep0</div>
|
||||||
<div style="position: absolute; left: 894px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #a85400; border: 2px solid #ffffff"></div>
|
<div style="position: absolute; left: 894px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #a85400; border: 2px solid #ffffff"></div>
|
||||||
<div style="position: absolute; left: 898px; top: 2000px; width: 230px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>02:30</b> hub database → ep0</div>
|
<div style="position: absolute; left: 898px; top: 2000px; width: 230px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>02:30</b> hub DB → ep0, 6-hourly</div>
|
||||||
<div style="position: absolute; left: 1134px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #0a3d73; border: 2px solid #ffffff"></div>
|
<div style="position: absolute; left: 1134px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #0a3d73; border: 2px solid #ffffff"></div>
|
||||||
<div style="position: absolute; left: 1138px; top: 2040px; width: 236px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>03:30</b> ep0 prunes households</div>
|
<div style="position: absolute; left: 1138px; top: 2040px; width: 236px; font-size: 16px; line-height: 20px; font-weight: 400; color: #051343; text-align: left;"><b>03:30</b> ep0 prunes households</div>
|
||||||
<div style="position: absolute; left: 1194px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #0a3d73; border: 2px solid #ffffff"></div>
|
<div style="position: absolute; left: 1194px; top: 2024px; width: 12px; height: 12px; box-sizing: border-box; border-radius: 6px; background: #0a3d73; border: 2px solid #ffffff"></div>
|
||||||
|
|||||||
@@ -6,17 +6,17 @@
|
|||||||
</style></head><body><div class="wrap">
|
</style></head><body><div class="wrap">
|
||||||
<h1>Where Felhom stands</h1>
|
<h1>Where Felhom stands</h1>
|
||||||
<div style="color:#8fa0b8;font-size:12.5px;margin:7px 0 4px;max-width:1100px;line-height:1.5">What we built, what happens when things go wrong, and what is still missing. <b style="color:#cbd7e8">Generated from <code>where-felhom-stands.yaml</code></b>, which cites the capability map, register row or evidence document behind every claim — and which is checked by <code>scripts/check_stands.py</code>.</div>
|
<div style="color:#8fa0b8;font-size:12.5px;margin:7px 0 4px;max-width:1100px;line-height:1.5">What we built, what happens when things go wrong, and what is still missing. <b style="color:#cbd7e8">Generated from <code>where-felhom-stands.yaml</code></b>, which cites the capability map, register row or evidence document behind every claim — and which is checked by <code>scripts/check_stands.py</code>.</div>
|
||||||
<div style="color:#6b7a91;font-size:11.5px;margin-bottom:16px">Verified 2026-08-22 against felhom-agent <code>40d857b527</code>, felhom-controller <code>2da259af38</code>, hub <code>877fcd2a38</code>. <b style="color:#fbbf24">15 claim(s) carry a recorded status move</b> (8 re-checked in this pass) — each is marked on the page.</div>
|
<div style="color:#6b7a91;font-size:11.5px;margin-bottom:16px">Verified 2026-08-22 against felhom-agent <code>40d857b527</code>, felhom-controller <code>2da259af38</code>, hub <code>877fcd2a38</code>. <b style="color:#fbbf24">17 claim(s) carry a recorded status move</b> (4 re-checked in this pass) — each is marked on the page.</div>
|
||||||
<div style="display:flex;gap:22px;flex-wrap:wrap;background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:11px 14px;margin-bottom:18px"><div style="display:flex;gap:8px;align-items:flex-start"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:11.5px"><b>Walked</b> <span style="color:#7c8aa3">(20)</span><span style="display:block;color:#6b7a91;font-size:10.5px">done end to end on real hardware, evidence on file</span></span></div><div style="display:flex;gap:8px;align-items:flex-start"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:11.5px"><b>Built</b> <span style="color:#7c8aa3">(15)</span><span style="display:block;color:#6b7a91;font-size:10.5px">shipped and tested, the real path never walked</span></span></div><div style="display:flex;gap:8px;align-items:flex-start"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:11.5px"><b>Partial</b> <span style="color:#7c8aa3">(16)</span><span style="display:block;color:#6b7a91;font-size:10.5px">some walked, some not — the note says which</span></span></div><div style="display:flex;gap:8px;align-items:flex-start"><span style="width:8px;height:8px;border-radius:50%;background:#64748b;flex:0 0 8px;margin-top:5px"></span><span style="font-size:11.5px"><b>Missing</b> <span style="color:#7c8aa3">(4)</span><span style="display:block;color:#6b7a91;font-size:10.5px">does not exist</span></span></div><div style="display:flex;gap:8px;align-items:flex-start"><span style="font-size:11.5px"><b style="color:#7dd3fc">needs-hardware</b><span style="display:block;color:#6b7a91;font-size:10.5px">only a running box could settle it</span></span></div></div>
|
<div style="display:flex;gap:22px;flex-wrap:wrap;background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:11px 14px;margin-bottom:18px"><div style="display:flex;gap:8px;align-items:flex-start"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:11.5px"><b>Walked</b> <span style="color:#7c8aa3">(32)</span><span style="display:block;color:#6b7a91;font-size:10.5px">done end to end on real hardware, evidence on file</span></span></div><div style="display:flex;gap:8px;align-items:flex-start"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:11.5px"><b>Built</b> <span style="color:#7c8aa3">(4)</span><span style="display:block;color:#6b7a91;font-size:10.5px">shipped and tested, the real path never walked</span></span></div><div style="display:flex;gap:8px;align-items:flex-start"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:11.5px"><b>Partial</b> <span style="color:#7c8aa3">(15)</span><span style="display:block;color:#6b7a91;font-size:10.5px">some walked, some not — the note says which</span></span></div><div style="display:flex;gap:8px;align-items:flex-start"><span style="width:8px;height:8px;border-radius:50%;background:#64748b;flex:0 0 8px;margin-top:5px"></span><span style="font-size:11.5px"><b>Missing</b> <span style="color:#7c8aa3">(4)</span><span style="display:block;color:#6b7a91;font-size:10.5px">does not exist</span></span></div><div style="display:flex;gap:8px;align-items:flex-start"><span style="font-size:11.5px"><b style="color:#7dd3fc">needs-hardware</b><span style="display:block;color:#6b7a91;font-size:10.5px">only a running box could settle it</span></span></div></div>
|
||||||
<div style="font-size:15px;font-weight:600;margin:0 0 9px">The journey</div>
|
<div style="font-size:15px;font-weight:600;margin:0 0 9px">The journey</div>
|
||||||
<div style="color:#6b7a91;font-size:11.5px;margin-bottom:10px">Seven stages, left to right, as they happen to a person.</div>
|
<div style="color:#6b7a91;font-size:11.5px;margin-bottom:10px">Seven stages, left to right, as they happen to a person.</div>
|
||||||
<div class="grid7">
|
<div class="grid7">
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">1</span> Getting a box</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A blank machine installs itself from our own boot image and registers itself as unclaimed — proven on two different boards<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Two boards: N100 2026-07-18, HP t740 2026-07-21.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Bare-metal Felhom ISO (blank hardware → zero-touch auto-ins… · evidence tests/VALIDATION-n100-rehearsal-2026-07-18.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The installer is published rather than pushed: rolling it back is one act<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Gate 6 of hostinstall_gates asserts the manifest names an installer-v tag; ran green tonight.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: gate 6 asserts the manifest names an installer tag, but no walk of a rollback is on file</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map The installer is PUBLISHED, not pushed · register R-110</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Installing onto hardware the customer already owns — the path exists, the first real one has not happened<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">UPGRADED: a real --mode byo install completed on demo-hp 2026-08-09 (Day-0 provision SUCCESS, 3m49s). Still not a customer's own hardware, so not 'walked' — but 'has not happened' is now false.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A machine that ends up on the wrong network port explains itself on screen and finds its way back — proven in a virtual drill, never on metal<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Map says the same: PROVEN-LIVE (nested drill — nested != metal).</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Box survives a wrong-NIC install · evidence audits/SPIKE-firstboot-nic-sweep-2026-07-22.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Reinstalling a machine we previously installed refuses, because our own removal leaves a name service holding the port our own installer checks<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Warning card. Confirmed live 2026-08-09; dnsmasq restarted by --uninstall seizes :53.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-272 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">1</span> Getting a box</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A blank machine installs itself from our own boot image and registers itself as unclaimed — proven on two different boards<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Two boards: N100 2026-07-18, HP t740 2026-07-21.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Bare-metal Felhom ISO (blank hardware → zero-touch auto-ins… · evidence tests/VALIDATION-n100-rehearsal-2026-07-18.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The installer is published rather than pushed: rolling it back is one act<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Gate 6 of hostinstall_gates asserts the manifest names an installer-v tag; ran green tonight.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: gate 6 asserts the manifest names an installer tag, but no walk of a rollback is on file</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map The installer is PUBLISHED, not pushed · register R-110</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Installing onto hardware the customer already owns — the path exists, the first real one has not happened<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">UPGRADED: a real --mode byo install completed on demo-hp 2026-08-09 (Day-0 provision SUCCESS, 3m49s). Still not a customer's own hardware, so not 'walked' — but 'has not happened' is now false.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A machine that ends up on the wrong network port explains itself on screen and finds its way back — proven in a virtual drill, never on metal<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Map says the same: PROVEN-LIVE (nested drill — nested != metal).</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Box survives a wrong-NIC install · evidence audits/SPIKE-firstboot-nic-sweep-2026-07-22.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Reinstalling a machine we previously installed refuses, because our own removal leaves a name service holding the port our own installer checks<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Warning card. Confirmed live 2026-08-09; dnsmasq restarted by --uninstall seizes :53.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-272 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">2</span> Making it theirs</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">An emailed one-time code; the customer sets their own password and the operator never sees it<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Exercised live 2026-08-09: reset code accepted at /claim, 302 + session.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Customer claim: one-time emailed code → customer sets own p… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The customer can bind their own machine from a link, with no operator present — done once, for real<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">attempts=0, locked=0, source customer_selfbind, 2026-07-18.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Customer binds their own appliance (self-service) · evidence tests/VALIDATION-n100-rehearsal-2026-07-18.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">It has never been done by a person who is not the operator<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Map records MISSING (as evidence). Still true after 2026-08-09.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map A customer (not the operator) performs a restore via UI alone</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Each of the two secrets has one name on every surface, and the re-enrol mail points at a page a rebuilt machine shows<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Shipped (controller v0.211.0, hub v0.104.0/v0.105.0); no customer has typed the passphrase yet. Operator ruling 2026-10-12 (S13). The old defect cost a real code on 2026-08-09.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-282 · register R-295 · register R-323 · register R-327</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">2</span> Making it theirs</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">An emailed one-time code; the customer sets their own password and the operator never sees it<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Exercised live 2026-08-09: reset code accepted at /claim, 302 + session.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Customer claim: one-time emailed code → customer sets own p… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The customer can bind their own machine from a link, with no operator present — done once, for real<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">attempts=0, locked=0, source customer_selfbind, 2026-07-18.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Customer binds their own appliance (self-service) · evidence tests/VALIDATION-n100-rehearsal-2026-07-18.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">It has never been done by a person who is not the operator<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Map records MISSING (as evidence). Still true after 2026-08-09.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map A customer (not the operator) performs a restore via UI alone</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Each of the two secrets has one name on every surface, and the re-enrol mail points at a page a rebuilt machine shows<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Shipped (controller v0.211.0, hub v0.104.0/v0.105.0); no customer has typed the passphrase yet. Operator ruling 2026-10-12 (S13). The old defect cost a real code on 2026-08-09.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-282 · register R-295 · register R-323 · register R-327</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">3</span> Using it</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Apps installed from a catalog of about 52, with a memory guard and health-aware progress<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">53 templates listed on the rebuilt box 2026-08-09; deploy driven live.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Deploy an app from the catalog (env config, memory guard, h… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Start, stop, restart, update, logs, remove — and the parts that must not be stopped cannot be<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: no walk document cited by the map row or anywhere else</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map App lifecycle: start/stop/restart/update/logs/remove/redeploy · register R-108</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Reachable from anywhere through a tunnel, per-app addresses<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Verified tonight: the rebuilt box answered on its public URL from outside.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Remote access via Cloudflare Tunnel + Traefik (per-app subd… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Reachable on the home network when the internet is down<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#1b2740;color:#7dd3fc;font-size:10px;white-space:nowrap">needs-hardware</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Only an observation on a running box with WAN pulled could settle it. Boxes are off.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map LAN access when internet is down (lan_resolver)</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Phone photos, documents with text recognition, files from Windows Explorer or a Mac<span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Files from Windows Explorer / Mac Finder (SMB server) · evidence audits/SPIKE-lan-discovery-2026-07-18.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A one-tap launcher, and a read-only guest link for visitors<span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Indítópult (app launcher) — one-tap grid of the household's…</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#64748b;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-heigLine truncated
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">3</span> Using it</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Apps installed from a catalog of about 52, with a memory guard and health-aware progress<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">53 templates listed on the rebuilt box 2026-08-09; deploy driven live.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Deploy an app from the catalog (env config, memory guard, h… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Start, stop, restart, update, logs, remove — and the parts that must not be stopped cannot be<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was built</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Walked 2026-10-28 night on scratch 9202: install, stop, start, restart, logs; traefik and cloudflared refuse a stop (403); remove with data leaves no volume and no container. The update half: the same day's guarded updates on 9202 (bench + box + undo walks).</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: a walk document is now cited (2026-10-28 night)</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map App lifecycle: start/stop/restart/update/logs/remove/redeploy · register R-108 · evidence audits/night-2026-10-28/part5-claims/lifecycle-walk.log · evidence audits/dockerhub-bigsteps-2026-10-28</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Reachable from anywhere through a tunnel, per-app addresses<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Verified tonight: the rebuilt box answered on its public URL from outside.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Remote access via Cloudflare Tunnel + Traefik (per-app subd… · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Reachable on the home network when the internet is down<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Walked 2026-11-03: with the internet gone the dashboard answers at the box's own address, and the box's DNS answers every app's name. By NAME it needs the home network's DNS to point at the box, and nothing tells the household; on a host whose FQDN is the dashboard's name (Tester 1) that one name answers the host and fails (R-1364).</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">evidence audits/security-2026-11-02/part5/lan-resolver.txt · evidence audits/security-2026-11-02/part5/outage-wan.log · register R-1364 · capability-map LAN access when internet is down (lan_resolver)</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Phone photos, documents with text recognition, files from Windows Explorer or a Mac<span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Files from Windows Explorer / Mac Finder (SMB server) · evidence audits/SPIKE-lan-discovery-2026-07-18.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;heigLine truncated
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">4</span> Drives</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A new drive is found, offered, formatted, mounted and enrolled — including on awkward older boot layouts<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">moved twice: 2026-08-09 walked -> built, then 2026-08-10 back to walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Applies to a NEW drive. Re-attaching an existing one after a reinstall is R-280 and fails.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: receipt found 2026-08-10: a live throwaway drive on felhom-pve walked scan -> format -> mount -> PVE dir storage -> one-click Regisztralas, with the resulting storage.cfg entry and systemd mount unit recorded. The map already read PROVEN-LIVE; the dataset was behind it. Scope unchanged: this is a NEW drive. Re-attaching an existing one after a reinstall is R-280 and still fails.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Drive wizard: scan/format/mount/enroll, incl. legacy-boot L… · register R-220 · evidence audits/SPIKE-raw-drive-enroll-2026-06-15.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Moving data between drives, crash-safe; removing a drive safely; unplug detected<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: no walk document cited</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Data migration between drives (all / per-app), crash-safe</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A network drive can be browsed and hold bulk media, but may not hold an app's data — enforced<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">RefuseAsAppNamespace is the fail-closed predicate.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Network storage (NAS) is browse + bulk-media only · register R-108 · evidence audits/R108-network-app-namespace-2026-07-30.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">After a reinstall the data drive cannot be re-attached through any dashboard route<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Warning card. /api/disks/candidates returns empty; the restore page promises two clicks.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-280 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">4</span> Drives</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A new drive is found, offered, formatted, mounted and enrolled — including on awkward older boot layouts<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">moved twice: 2026-08-09 walked -> built, then 2026-08-10 back to walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Applies to a NEW drive. Re-attaching an existing one after a reinstall is R-280 and fails.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: receipt found 2026-08-10: a live throwaway drive on felhom-pve walked scan -> format -> mount -> PVE dir storage -> one-click Regisztralas, with the resulting storage.cfg entry and systemd mount unit recorded. The map already read PROVEN-LIVE; the dataset was behind it. Scope unchanged: this is a NEW drive. Re-attaching an existing one after a reinstall is R-280 and still fails.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Drive wizard: scan/format/mount/enroll, incl. legacy-boot L… · register R-220 · evidence audits/SPIKE-raw-drive-enroll-2026-06-15.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Moving data between drives, crash-safe; removing a drive safely; unplug detected<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: no walk document cited</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Data migration between drives (all / per-app), crash-safe</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A network drive can be browsed and hold bulk media, but may not hold an app's data — enforced<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">RefuseAsAppNamespace is the fail-closed predicate.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Network storage (NAS) is browse + bulk-media only · register R-108 · evidence audits/R108-network-app-namespace-2026-07-30.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">After a reinstall the data drive cannot be re-attached through any dashboard route<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Warning card. /api/disks/candidates returns empty; the restore page promises two clicks.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-280 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">5</span> Backing up</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">App data on the machine, nightly database dumps, a copy on a second drive<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">moved twice: 2026-08-09 walked -> built, then 2026-08-10 back to walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">True for every app but one, and that one was found on 2026-08-21: paperless-ngx's 72-table PostgreSQL was dumped nightly and written into a folder named after an app that does not exist, so it never entered the recovery unit, the second-drive copy or the off-site copy. Fixed and proven live 2026-08-22 (R-355) — the dump is now in the app's own unit and in the off-site snapshot for the first time. A catalogue-wide sweep, itself proven able to convict a planted case, says 1 of 53 was affected.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: receipt found 2026-08-10: an adversarial, destructive, unattended overnight campaign across both boxes and ep0, with A2 (one quiesce, two tiers) proven end to end. Map already read PROVEN-LIVE.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Tier-2 secondary-drive copy: class-driven legs · register R-355 · evidence audits/CAMPAIGN-8-backup-restore-2026-07-27.md · evidence audits/DRILL-backup-truth-2026-08-21/evidence/phase4-paperl…</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A whole-machine archive that lands off the guest's own disk — a single-drive machine is recorded as degraded rather than pretending<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: no walk document cited</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Whole-guest backup lands OFF the guest's own physical device · register R-165</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">An encrypted off-site copy, sealed with a key the operator cannot read<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">The COPY is real and the key is still unreadable to us — what failed is DAILY and UNBROKEN. The 2026-08-09 note said '18 snapshots, daily, unbroken'; the next snapshot after 2026-08-09 08:30 was 2026-08-21 22:17, put there by hand during the drill. Twelve days, no alarm. Two causes in series: the 2026-08-21 rebuild lost the off-box target (R-193's shape), and after the self-heal restored it EVERY per-app switch was still off, so the first run logged 'backup OK: 0 app(s) backed up, 14s'.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: the claim is about a CONTINUING daily copy; a 12-day silent gap was found on 2026-08-21 and nothing reported it</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Offsite (restic → Hetzner Storage Box) · register R-199 · register R-193 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The backups prove themselves: a restore is actually performed, uLine truncated
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">5</span> Backing up</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">App data on the machine, nightly database dumps, a copy on a second drive<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">moved twice: 2026-08-09 walked -> built, then 2026-08-10 back to walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">True for every app but one, and that one was found on 2026-08-21: paperless-ngx's 72-table PostgreSQL was dumped nightly and written into a folder named after an app that does not exist, so it never entered the recovery unit, the second-drive copy or the off-site copy. Fixed and proven live 2026-08-22 (R-355) — the dump is now in the app's own unit and in the off-site snapshot for the first time. A catalogue-wide sweep, itself proven able to convict a planted case, says 1 of 53 was affected.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: receipt found 2026-08-10: an adversarial, destructive, unattended overnight campaign across both boxes and ep0, with A2 (one quiesce, two tiers) proven end to end. Map already read PROVEN-LIVE.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Tier-2 secondary-drive copy: class-driven legs · register R-355 · evidence audits/CAMPAIGN-8-backup-restore-2026-07-27.md · evidence audits/DRILL-backup-truth-2026-08-21/evidence/phase4-paperl…</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A whole-machine archive that lands off the guest's own disk — a single-drive machine is recorded as degraded rather than pretending<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: no walk document cited</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Whole-guest backup lands OFF the guest's own physical device · register R-165</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">An encrypted off-site copy, sealed with a key the operator cannot read<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">DAILY and UNBROKEN, read from the repositories 2026-11-02: demo-hp 14–15 snapshots every day 10-21 → 11-02, Tester 1 4 every day 10-27 → 11-02. The 2026-08 gap's two causes were fixed (R-199, R-193). The key stays the household's.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: the claim is about a CONTINUING daily copy; a 12-day silent gap was found on 2026-08-21 and nothing reported it</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">evidence audits/security-2026-11-02/part5/offsite-demo-hp.txt · evidence audits/security-2026-11-02/part5/offsite-tester1.txt · capability-map Offsite (restic → Hetzner Storage Box) · register R-199 · register R-193 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The backups prove themselves: a restore is actually performed, unattended, on every tier, on both machines<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;bLine truncated
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">6</span> When something goes wrong</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The machine watches itself and repairs some faults without telling anyone it had to<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">R-264 records that the self-heal counters reach the hub and are decoded nowhere.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: no walk document cited</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Box survives an unattended app or guest-network failure · register R-264</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Failures reach the operator by email, one mail per run, every failing app named<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">CONFIRMED against source: backup_run_failures digest is allowlisted (api/handler.go:1837), operator-only (dispatcher.go:423), templated (templates.go:48); recovery_unit_capture_failed is record-only (dispatcher.go:376). NOTE: R-182's register row still describes the PRE-FIX behaviour — see R-289.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: the digest is wired and source-verified, but no run of it has been observed delivering</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map A failed per-app Tier-1 backup reaches the OPERATOR — EVERY… · register R-182</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Failures reach the customer<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">The customer leg is built; today's log shows customer-channel rows skipped as operator_only.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map App crashes → customer notified (one event per transition, …</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">An already-paired box is still told to pair itself<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Warning card.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-214 · register R-235</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">6</span> When something goes wrong</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The machine watches itself and repairs some faults without telling anyone it had to<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was built</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Walked 2026-10-29 night on scratch 9202: a crashed app part (kill -9 of paperless-redis) ran again in 1 s and the health probe read the app healthy; a guest reboot brought every app back in under 2 minutes, the boot sweep found nothing to start; no event, no banner, no mail. Docker's restart policy did the repair in both legs; the controller's own boot repair had nothing to do (its last proof: 6 of 6 hard resets, 2026-08-02). R-264 records that the self-heal counters reach the hub and are decoded nowhere.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: walked 2026-10-29 on scratch 9202 (a walk document is now cited)</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Box survives an unattended app or guest-network failure · register R-264 · evidence audits/night-2026-10-28/part5-claims/selfheal/WALK.md · evidence audits/DIAG-bootrecon-drive-absent-2026-08-02.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Failures reach the operator by email, one mail per run, every failing app named<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Seen in the mailbox: „Tester4: 3 of 8 apps failed to back up (manual run)\" names immich, nextcloud and paperless-ngx in one mail (2026-10-20); seven of this shape in 60 days.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: the digest is wired and source-verified, but no run of it has been observed delivering</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">evidence audits/security-2026-11-02/part5/mail-operator-digest.txt · capability-map A failed per-app Tier-1 backup reaches the OPERATOR — EVERY… · register R-182</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Failures reach the customer<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Seen in the mailbox: the crash mail and the fill warning to tester1@ (2026-11-02/03), „A szerver nem elérhető!\" to tester4@ (2026-10-24), each in Hungarian. It needs an address on the household's page (Tester 1 had none).</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">evidence audits/security-2026-11-02/part5/mail-crash-household.txt · evidence audits/security-2026-11-02/part5/mail-fill-household.txt · evidence audits/security-2026-11-02/part5/mail-operator-digest.txt · capability-map App crashes → customer notified (one event per transition, …</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">An already-paired box no longer tells anyone to pair it — its console says there is nothing to do<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was partial</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Was a warning card (the console kept showing the pairing code). Fixed by ISO 1.28.0 (R-535; R-214, R-235 closed). WalkedLine truncated
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">7</span> Getting everything back</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A rebuilt machine shows a full-page recovery screen without anyone looking for it, and says plainly that nobody can replace a lost recovery code<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Seen unsought on the rebuilt demo-hp 2026-08-09, seal date matching host_escrow.created_at.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-193 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The customer's code opens the sealed package and the data returns byte for byte — including accented Hungarian filenames, verified as raw bytes<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Still true, and re-proven 2026-08-22 (5/5 byte-identical, both accented names as raw bytes) — but the SCOPE is narrower than the sentence sounds and was silently narrower still until v0.218.0. The drill found the off-site restore had NO named-volume leg at all: the archive sat in the unit, the snapshot and the checking folder and was never replayed, under a success message (R-354, fixed and proven 2026-08-22). AND 40 of the 53 catalogue apps STILL cannot run this route at all — it refuses first, saying a running app is not installed (R-356, open). So: proven for an app that declares a data drive; unproven and currently unreachable for the class whose entire dataset is a named volume.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-201 · register R-354 · register R-356 · evidence tests/walk5-r201-2026-08-07/journal.md · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Walked end to end with no command line inside the machine (2026-08-07)<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#1b2740;color:#a78bfa;font-size:10px;white-space:nowrap">contested</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">CONTESTED-RESOLVED: not a contradiction. The walk proves the ROUTE needs no guest shell; the map's MISSING row is about a NON-OPERATOR doing it, which has still never happened. Two questions, one word 'customer'.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-201 · evidence tests/walk5-r201-2026-08-07/journal.md · capability-map A customer (not the operator) performs a restore via UI alone</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Putting restored files back where they belong is still manual<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Warning card. Confirmed 2026-08-09: the restore lands in a verification folder and says so.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-213</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The tripwire that says someone is opening this customer's backups does fire<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">CONFIRMED tonight from the hub store: escrow_blob_served 2026-08-09 10:19:41Z = 12:19 CEST in the operator's mailbox. R-281's original claim of silence is WITHDRAWN.</span><span style="display:block;color:#4b5bLine truncated
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44"><span style="color:#4b5b76;font-family:ui-monospace,monospace">7</span> Getting everything back</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A rebuilt machine shows a full-page recovery screen without anyone looking for it, and says plainly that nobody can replace a lost recovery code<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Seen unsought on the rebuilt demo-hp 2026-08-09, seal date matching host_escrow.created_at.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-193 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The customer's code opens the sealed package and the data returns byte for byte — including accented Hungarian filenames, verified as raw bytes<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Still true, and re-proven 2026-08-22 (5/5 byte-identical, both accented names as raw bytes) — but the SCOPE is narrower than the sentence sounds and was silently narrower still until v0.218.0. The drill found the off-site restore had NO named-volume leg at all: the archive sat in the unit, the snapshot and the checking folder and was never replayed, under a success message (R-354, fixed and proven 2026-08-22). AND 40 of the 53 catalogue apps STILL cannot run this route at all — it refuses first, saying a running app is not installed (R-356, open). So: proven for an app that declares a data drive; unproven and currently unreachable for the class whose entire dataset is a named volume.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-201 · register R-354 · register R-356 · evidence tests/walk5-r201-2026-08-07/journal.md · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Walked end to end with no command line inside the machine (2026-08-07)<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#1b2740;color:#a78bfa;font-size:10px;white-space:nowrap">contested</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">CONTESTED-RESOLVED: not a contradiction. The walk proves the ROUTE needs no guest shell; the map's MISSING row is about a NON-OPERATOR doing it, which has still never happened. Two questions, one word 'customer'.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-201 · evidence tests/walk5-r201-2026-08-07/journal.md · capability-map A customer (not the operator) performs a restore via UI alone</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Putting restored files back where they belong is still manual<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Warning card. Confirmed 2026-08-09: the restore lands in a verification folder and says so.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-213</span></span></div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The tripwire that says someone is opening this customer's backups does fire<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">CONFIRMED tonight from the hub store: escrow_blob_served 2026-08-09 10:19:41Z = 12:19 CEST in the operator's mailbox. R-281's original claim of silence is WITHDRAWN.</span><span style="display:block;color:#4b5bLine truncated
|
||||||
</div>
|
</div>
|
||||||
<div style="font-size:15px;font-weight:600;margin:22px 0 9px">When things go wrong</div>
|
<div style="font-size:15px;font-weight:600;margin:22px 0 9px">When things go wrong</div>
|
||||||
@@ -24,8 +24,8 @@
|
|||||||
<div class="grid3">
|
<div class="grid3">
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">A disk starts failing</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A disk starts failing — healthy path only; a genuinely failing disk has never been seen<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#1b2740;color:#7dd3fc;font-size:10px;white-space:nowrap">needs-hardware</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Only a failing disk on a running box could settle it.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Lemez-egészség felügyelet: per-disk SMART kártya · evidence audits/SPIKE-smart-coverage-2026-07-25.md</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">A disk starts failing</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A disk starts failing — healthy path only; a genuinely failing disk has never been seen<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#1b2740;color:#7dd3fc;font-size:10px;white-space:nowrap">needs-hardware</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Only a failing disk on a running box could settle it.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Lemez-egészség felügyelet: per-disk SMART kártya · evidence audits/SPIKE-smart-coverage-2026-07-25.md</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The backup drive is unplugged</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The backup drive is unplugged<span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map An ABSENT backup-target drive raises its OWN alarm · evidence audits/R116-v0116-2026-07-30.md</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The backup drive is unplugged</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The backup drive is unplugged<span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map An ABSENT backup-target drive raises its OWN alarm · evidence audits/R116-v0116-2026-07-30.md</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">A drive is filling up</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A drive is filling up<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: no walk document cited</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-167</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">A drive is filling up</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A drive is filling up<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Same walk as backup.fill-warning: warned before it filled; the file removed, nothing else was needed.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: no walk document cited</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">evidence audits/security-2026-11-02/part5/fill-walk.log · register R-167</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">An app crashes</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">An app crashes — the email leg has never been confirmed end to end<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Consistent with fault.operator-email: the digest is wired but its delivery is unobserved.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map App crashes → customer notified (one event per transition, …</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">An app crashes</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">An app crashes — the box stops it and both the operator and the household get a mail<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Walked 2026-11-02 on Tester 1: kill -9 of a throwaway app, 15 s apart; at 6 restarts in 10 minutes the box stopped and held it, the operator's mail arrived in 1 s; with an address on the household's page the Hungarian mail arrived too. The household's Start lifted it. One flaw: the mail ends with raw JSON (R-1363).</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">evidence audits/security-2026-11-02/part5/crash-walk.log · evidence audits/security-2026-11-02/part5/crash-walk-customer.log · evidence audits/security-2026-11-02/part5/mail-crash-operator.txt · evidence audits/security-2026-11-02/part5/mail-crash-household.txt · register R-1363 · capability-map App crashes → customer notified (one event per transition, …</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">Power cut mid-backup</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Power cut mid-backup<span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">evidence audits/AUDIT-power-outage-recovery-2026-07-22.md</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">Power cut mid-backup</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">Power cut mid-backup<span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">evidence audits/AUDIT-power-outage-recovery-2026-07-22.md</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The guest is destroyed</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The guest is destroyed<span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-201 · evidence audits/DRILL-r201-night-run-2026-08-04.md · evidence audits/DRILL-r201-night-run-2026-08-04.md</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The guest is destroyed</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The guest is destroyed<span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-201 · evidence audits/DRILL-r201-night-run-2026-08-04.md · evidence audits/DRILL-r201-night-run-2026-08-04.md</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The whole machine is wiped and reinstalled</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The whole machine is wiped and reinstalled — the data comes back<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">The 2026-08-09 rehearsal really did return 4/4 byte-identical, and that stands. What the next real reinstall showed (2026-08-21, demo-hp) is that the rebuild ORPHANS BOTH OFF-PREMISES TIERS at once, quietly: the restic target was lost and needed a self-heal plus a per-app re-enable before any copy resumed (R-193), and the PBS archives from before the reinstall cannot be opened by the rebuilt box at all, because it now presents a different key (R-366). The data came back in the rehearsal because the rehearsal restored it immediately; a machine left alone after a reinstall is not protected in the meantime and nothing says so.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: a real reinstall on 2026-08-21 left both off-premises tiers broken - one silently for 12 days, the other unreadable - so 'the data comes back' holds only if someone restores it at once</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-193 · register R-366 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The whole machine is wiped and reinstalled</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The whole machine is wiped and reinstalled — the data comes back<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">The 2026-08-09 rehearsal really did return 4/4 byte-identical, and that stands. What the next real reinstall showed (2026-08-21, demo-hp) is that the rebuild ORPHANS BOTH OFF-PREMISES TIERS at once, quietly: the restic target was lost and needed a self-heal plus a per-app re-enable before any copy resumed (R-193), and the PBS archives from before the reinstall cannot be opened by the rebuilt box at all, because it now presents a different key (R-366). The data came back in the rehearsal because the rehearsal restored it immediately; a machine left alone after a reinstall is not protected in the meantime and nothing says so.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: a real reinstall on 2026-08-21 left both off-premises tiers broken - one silently for 12 days, the other unreadable - so 'the data comes back' holds only if someone restores it at once</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-193 · register R-366 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md</span></span></div></div>
|
||||||
@@ -34,13 +34,13 @@
|
|||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The customer forgets their dashboard password</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The customer forgets their dashboard password<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Exercised 2026-08-09 via the reset code.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The customer forgets their dashboard password</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The customer forgets their dashboard password<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Exercised 2026-08-09 via the reset code.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The customer loses their recovery code</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The customer loses their recovery code — by design, the data is unrecoverable<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">moved twice: 2026-08-09 walked -> built, then 2026-08-10 back to walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">The recovery screen states it in Hungarian.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: receipt found 2026-08-10, and then proven the hard way: the recovery screen states in Hungarian that nobody can replace a lost code, and on 2026-08-10 demo-felhom's pre-fix key was confirmed unrecoverable by construction (identity_blob NULL; the restic password lives only in that bundle). The by-design refusal is real.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Escrow ceremony: customer-facing wizard, one-shot R claim, … · register R-198 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The customer loses their recovery code</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The customer loses their recovery code — by design, the data is unrecoverable<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">moved twice: 2026-08-09 walked -> built, then 2026-08-10 back to walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">The recovery screen states it in Hungarian.</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: receipt found 2026-08-10, and then proven the hard way: the recovery screen states in Hungarian that nobody can replace a lost code, and on 2026-08-10 demo-felhom's pre-fix key was confirmed unrecoverable by construction (identity_blob NULL; the restic password lives only in that bundle). The by-design refusal is real.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Escrow ceremony: customer-facing wizard, one-shot R claim, … · register R-198 · evidence audits/REHEARSAL-byo-reinstall-2026-08-09.md</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The machine moves house / new network</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The machine moves house / new network<span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Box survives a site/network change (relocation, different s… · evidence audits/AUDIT-vacation-remote-ops-2026-07-20.md</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The machine moves house / new network</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The machine moves house / new network<span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Box survives a site/network change (relocation, different s… · evidence audits/AUDIT-vacation-remote-ops-2026-07-20.md</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The internet goes down</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The internet goes down — built, never walked<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#1b2740;color:#7dd3fc;font-size:10px;white-space:nowrap">needs-hardware</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Needs a running box with WAN pulled.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map LAN access when internet is down (lan_resolver)</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The internet goes down</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#34d399;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The internet goes down<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Walked 2026-11-03 on Tester 1: packets to the outside dropped on host and guest for 13 minutes, the line's route left in place. The dashboard answered on the home network, 10 apps ran, the public name gave 530; 60 s after the line came back the public name worked by itself, nothing to do. Slow: the Backups page took over 2 minutes (R-1362). An earlier try removed the route: the agent's self-heal put it back in about 100 s.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">evidence audits/security-2026-11-02/part5/outage-wan.log · evidence audits/security-2026-11-02/part5/outage-wan-3min.txt · register R-1362 · capability-map LAN access when internet is down (lan_resolver)</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The hub is down</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#60a5fa;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The hub is down<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: no walk document cited</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map Config/state change round-trips in seconds (hub↔box immediacy</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The hub is down</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The hub is down<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#3a2a12;color:#fbbf24;font-size:10px;white-space:nowrap">changed 2026-08-09, was walked</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Walked 2026-11-03 on Tester 1 (the hub's address dropped for 13 minutes; DooPlex untouched): the box works on — dashboard, apps, an install from the cached catalog, a crash-loop stop. What fails: every event sent in that time is lost after 9 s and nothing re-sends it, so the operator never heard of the held app (R-1361).</span><span style="display:block;color:#9a7b3a;font-size:11px;margin-top:2px">why it moved: no walk document cited</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">evidence audits/security-2026-11-02/part5/outage-hub.log · register R-1361 · capability-map Config/state change round-trips in seconds (hub↔box immediacy</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">We ship a broken release</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#64748b;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">We ship a broken release — the guard is missing<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Proven the hard way on 2026-08-09: a vouched agent version had no git tag and every install died at 5/8. Both guards still owed.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-273 · register R-287</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">We ship a broken release</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#64748b;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">We ship a broken release — the guard is missing<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Proven the hard way on 2026-08-09: a vouched agent version had no git tag and every install died at 5/8. Both guards still owed.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-273 · register R-287</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">A fresh install picks up an old image</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A fresh install picks up an old image<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Narrowed 2026-08-09: the resume path fetched the vouched golden; a fresh install still takes the newest LOCAL archive with no manifest comparison.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-274</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">A fresh install picks up an old image</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A fresh install picks up an old image<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Narrowed 2026-08-09: the resume path fetched the vouched golden; a fresh install still takes the newest LOCAL archive with no manifest comparison.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-274</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The off-site provider account is deleted</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The off-site provider account is deleted<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">The credential that holds the customer's documents can still delete.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-95</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The off-site provider account is deleted</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The off-site provider account is deleted<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">The credential that holds the customer's documents can still delete.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-95</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The machine is switched off for an afternoon</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#64748b;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The machine is switched off for an afternoon — no notion of expected downtime<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Confirmed 2026-08-09: eight operator mails for deliberate, attended work.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-285</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">The machine is switched off for an afternoon</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#64748b;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">The machine is switched off for an afternoon — no notion of expected downtime<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Confirmed 2026-08-09: eight operator mails for deliberate, attended work.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">register R-285</span></span></div></div>
|
||||||
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">A customer restores their own data with no help</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A customer restores their own data with no help<span style="display:inline-block;margin-left:6px;padding:1px 5px;border-radius:3px;background:#1b2740;color:#a78bfa;font-size:10px;white-space:nowrap">contested</span><span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">CONTESTED-RESOLVED: the map's MISSING is about a NON-OPERATOR performing it; the walks prove the route, not the person. Neither record was wrong. SINCE 2026-08-21 there is a second, harder blocker and it is not about the person at all: for the 40 of 53 apps that declare no data drive the off-site restore REFUSES before it starts, telling the customer a running app 'nincs telepítve' and to reinstall it to the same place — which those apps give them no way to choose (R-356). Those are exactly the apps whose whole dataset is a named volume. Until that is fixed, most customers cannot self-restore off-site even in principle.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map A customer (not the operator) performs a restore via UI alone · register R-201 · register R-356 · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md</span></span></div></div>
|
<div style="background:#121b2c;border:1px solid #1f2c44;border-radius:6px;padding:14px 13px"><div style="font-size:14px;font-weight:500;padding-bottom:9px;margin-bottom:10px;border-bottom:1px solid #1f2c44">A customer restores their own data with no help</div><div style="display:flex;gap:8px;align-items:flex-start;margin-bottom:9px"><span style="width:8px;height:8px;border-radius:50%;background:#fbbf24;flex:0 0 8px;margin-top:5px"></span><span style="font-size:12px;line-height:1.45;color:#b8c4d6;text-wrap:pretty">A customer restores their own data with no help<span style="display:block;color:#7c8aa3;font-size:11px;margin-top:3px">Walked 2026-10-29 night on scratch 9202 through the household's restore button (the page's own POST): vikunja from the 02:30 local point in 79 s — the project made before the point came back, the one made after it is gone, the app runs, no hold; a restore that meets a busy volume puts the live data back (R-1256 H5). The route needs no help. Still open: no person who is not the operator has done it (claim.never-by-non-operator). The old off-site blocker (R-356) shipped in controller v0.219.0.</span><span style="display:block;color:#4b5b76;font-size:10px;margin-top:3px;font-family:ui-monospace,SFMono-Regular,Menlo,monospace">capability-map A customer (not the operator) performs a restore via UI alone · register R-201 · register R-356 · evidence audits/REPORT-DRILL-backup-truth-2026-08-21.md · evidence audits/night-2026-10-28/part5-claims/selfrestore-walk.log</span></span></div></div>
|
||||||
</div>
|
</div>
|
||||||
<div style="margin-top:26px;padding-top:12px;border-top:1px solid #1f2c44;color:#4b5b76;font-size:10.5px;line-height:1.6">Generated by <code>scripts/render_stands.py</code> from <code>where-felhom-stands.yaml</code>. Do not hand-edit this file — edit the data and regenerate. The 2026-08-09 React bundle is kept as <code>where-felhom-stands-2026-08-09-snapshot.html</code> and is not maintained.</div>
|
<div style="margin-top:26px;padding-top:12px;border-top:1px solid #1f2c44;color:#4b5b76;font-size:10.5px;line-height:1.6">Generated by <code>scripts/render_stands.py</code> from <code>where-felhom-stands.yaml</code>. Do not hand-edit this file — edit the data and regenerate. The 2026-08-09 React bundle is kept as <code>where-felhom-stands-2026-08-09-snapshot.html</code> and is not maintained.</div>
|
||||||
</div></body></html>
|
</div></body></html>
|
||||||
@@ -155,17 +155,20 @@ claims:
|
|||||||
band: journey
|
band: journey
|
||||||
stage: 3
|
stage: 3
|
||||||
title: "Start, stop, restart, update, logs, remove — and the parts that must not be stopped cannot be"
|
title: "Start, stop, restart, update, logs, remove — and the parts that must not be stopped cannot be"
|
||||||
status: built
|
status: walked
|
||||||
|
note: "Walked 2026-10-28 night on scratch 9202: install, stop, start, restart, logs; traefik and cloudflared refuse a stop (403); remove with data leaves no volume and no container. The update half: the same day's guarded updates on 9202 (bench + box + undo walks)."
|
||||||
sources:
|
sources:
|
||||||
- capability-map: "App lifecycle: start/stop/restart/update/logs/remove/redeploy"
|
- capability-map: "App lifecycle: start/stop/restart/update/logs/remove/redeploy"
|
||||||
- register: "R-108"
|
- register: "R-108"
|
||||||
|
- evidence: "audits/night-2026-10-28/part5-claims/lifecycle-walk.log"
|
||||||
|
- evidence: "audits/dockerhub-bigsteps-2026-10-28"
|
||||||
changed:
|
changed:
|
||||||
from: walked
|
from: built
|
||||||
reason: "no walk document cited by the map row or anywhere else"
|
reason: "a walk document is now cited (2026-10-28 night)"
|
||||||
verified:
|
verified:
|
||||||
date: 2026-08-09
|
date: 2026-10-28
|
||||||
verdict: downgraded
|
verdict: upgraded
|
||||||
depth: register+map
|
depth: source-read
|
||||||
- id: use.tunnel
|
- id: use.tunnel
|
||||||
band: journey
|
band: journey
|
||||||
stage: 3
|
stage: 3
|
||||||
@@ -183,14 +186,17 @@ claims:
|
|||||||
band: journey
|
band: journey
|
||||||
stage: 3
|
stage: 3
|
||||||
title: "Reachable on the home network when the internet is down"
|
title: "Reachable on the home network when the internet is down"
|
||||||
status: built
|
status: partial
|
||||||
note: "Only an observation on a running box with WAN pulled could settle it. Boxes are off."
|
note: "Walked 2026-11-03: with the internet gone the dashboard answers at the box's own address, and the box's DNS answers every app's name. By NAME it needs the home network's DNS to point at the box, and nothing tells the household; on a host whose FQDN is the dashboard's name (Tester 1) that one name answers the host and fails (R-1364)."
|
||||||
sources:
|
sources:
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/lan-resolver.txt"
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/outage-wan.log"
|
||||||
|
- register: "R-1364"
|
||||||
- capability-map: "LAN access when internet is down (lan_resolver)"
|
- capability-map: "LAN access when internet is down (lan_resolver)"
|
||||||
verified:
|
verified:
|
||||||
date: 2026-08-09
|
date: 2026-11-03
|
||||||
verdict: needs-hardware
|
verdict: upgraded
|
||||||
depth: needs-hardware
|
depth: source-read
|
||||||
- id: use.files
|
- id: use.files
|
||||||
band: journey
|
band: journey
|
||||||
stage: 3
|
stage: 3
|
||||||
@@ -207,13 +213,18 @@ claims:
|
|||||||
band: journey
|
band: journey
|
||||||
stage: 3
|
stage: 3
|
||||||
title: "A one-tap launcher, and a read-only guest link for visitors"
|
title: "A one-tap launcher, and a read-only guest link for visitors"
|
||||||
status: built
|
status: walked
|
||||||
|
note: "Walked 2026-10-28 night on scratch 9202 through the pages the browser receives: the launcher lists each openable app; the guest link opens with no login and no admin link, a wrong token is 404, rotate and switch-off kill the old link."
|
||||||
sources:
|
sources:
|
||||||
- capability-map: "Indítópult (app launcher) — one-tap grid of the household's openable apps"
|
- capability-map: "Indítópult (app launcher) — one-tap grid of the household's openable apps"
|
||||||
|
- evidence: "audits/night-2026-10-28/part5-claims/launcher-walk.log"
|
||||||
|
changed:
|
||||||
|
from: built
|
||||||
|
reason: "walked live 2026-10-28 night (map E8, E9 → PROVEN-LIVE)"
|
||||||
verified:
|
verified:
|
||||||
date: 2026-08-09
|
date: 2026-10-28
|
||||||
verdict: confirmed
|
verdict: upgraded
|
||||||
depth: register+map
|
depth: source-read
|
||||||
- id: use.dlna
|
- id: use.dlna
|
||||||
band: journey
|
band: journey
|
||||||
stage: 3
|
stage: 3
|
||||||
@@ -337,9 +348,11 @@ claims:
|
|||||||
band: journey
|
band: journey
|
||||||
stage: 5
|
stage: 5
|
||||||
title: "An encrypted off-site copy, sealed with a key the operator cannot read"
|
title: "An encrypted off-site copy, sealed with a key the operator cannot read"
|
||||||
status: partial
|
status: walked
|
||||||
note: "The COPY is real and the key is still unreadable to us — what failed is DAILY and UNBROKEN. The 2026-08-09 note said '18 snapshots, daily, unbroken'; the next snapshot after 2026-08-09 08:30 was 2026-08-21 22:17, put there by hand during the drill. Twelve days, no alarm. Two causes in series: the 2026-08-21 rebuild lost the off-box target (R-193's shape), and after the self-heal restored it EVERY per-app switch was still off, so the first run logged 'backup OK: 0 app(s) backed up, 14s'."
|
note: "DAILY and UNBROKEN, read from the repositories 2026-11-02: demo-hp 14–15 snapshots every day 10-21 → 11-02, Tester 1 4 every day 10-27 → 11-02. The 2026-08 gap's two causes were fixed (R-199, R-193). The key stays the household's."
|
||||||
sources:
|
sources:
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/offsite-demo-hp.txt"
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/offsite-tester1.txt"
|
||||||
- capability-map: "Offsite (restic → Hetzner Storage Box)"
|
- capability-map: "Offsite (restic → Hetzner Storage Box)"
|
||||||
- register: "R-199"
|
- register: "R-199"
|
||||||
- register: "R-193"
|
- register: "R-193"
|
||||||
@@ -349,16 +362,19 @@ claims:
|
|||||||
from: walked
|
from: walked
|
||||||
reason: "the claim is about a CONTINUING daily copy; a 12-day silent gap was found on 2026-08-21 and nothing reported it"
|
reason: "the claim is about a CONTINUING daily copy; a 12-day silent gap was found on 2026-08-21 and nothing reported it"
|
||||||
verified:
|
verified:
|
||||||
date: 2026-08-22
|
date: 2026-11-03
|
||||||
verdict: downgraded
|
verdict: upgraded
|
||||||
depth: source-read
|
depth: source-read
|
||||||
- id: backup.restore-proof
|
- id: backup.restore-proof
|
||||||
band: journey
|
band: journey
|
||||||
stage: 5
|
stage: 5
|
||||||
title: "The backups prove themselves: a restore is actually performed, unattended, on every tier, on both machines"
|
title: "The backups prove themselves: a restore is actually performed, unattended, on every tier, on both machines"
|
||||||
status: built
|
status: walked
|
||||||
note: "STILL GREY, and for a sharper reason than in August. The scheduler runs and the check works — it failed again on 2026-08-21, unprompted, and named the cause exactly: after the reinstall the box presents a different PBS key than its own older archives were sealed with, so those archives cannot be opened at all (R-366). A tier whose archives are orphaned is a different alarm from a tier whose test failed, and only the second is being said."
|
note: "Read 2026-11-02 from the journals of all three boxes: a scheduled restore-test passed on every tier in 10 days — demo-hp local and PBS, Tester 1 local and PBS, the N100 its USB drive and PBS, the newest on 2026-11-02 22:45. R-366's orphaned key was a one-off of the 2026-08 reinstall."
|
||||||
sources:
|
sources:
|
||||||
|
- evidence: "audits/security-2026-11-02/part4/restore-test-demo-hp.txt"
|
||||||
|
- evidence: "audits/security-2026-11-02/part4/restore-test-tester1.txt"
|
||||||
|
- evidence: "audits/security-2026-11-02/part4/restore-test-n100.txt"
|
||||||
- capability-map: "Restore-proof is UNATTENDED — the scheduler covers EVERY tier"
|
- capability-map: "Restore-proof is UNATTENDED — the scheduler covers EVERY tier"
|
||||||
- register: "R-86"
|
- register: "R-86"
|
||||||
- register: "R-366"
|
- register: "R-366"
|
||||||
@@ -369,16 +385,18 @@ claims:
|
|||||||
decay: "PROOF-DECAY RULE FIRED (first time it has). A receipt EXISTS - architecture/_recovery-inventory-2026-07-28.md carries live journal lines for scheduled restore-tests on both boxes and both tiers - but it is superseded by later observation: demo-hp logged restore_test_failed on 2026-08-05, and the box has since been wiped and reinstalled (2026-08-09). The claim is about a CONTINUING scheduled behaviour, so a 2026-07-28 observation cannot carry it. Stays grey until a scheduled restore-test is seen passing on the rebuilt box. THE CAPABILITY MAP STILL READS PROVEN-LIVE (2026-08-03) AND IS NOW THE THING OUT OF STEP."
|
decay: "PROOF-DECAY RULE FIRED (first time it has). A receipt EXISTS - architecture/_recovery-inventory-2026-07-28.md carries live journal lines for scheduled restore-tests on both boxes and both tiers - but it is superseded by later observation: demo-hp logged restore_test_failed on 2026-08-05, and the box has since been wiped and reinstalled (2026-08-09). The claim is about a CONTINUING scheduled behaviour, so a 2026-07-28 observation cannot carry it. Stays grey until a scheduled restore-test is seen passing on the rebuilt box. THE CAPABILITY MAP STILL READS PROVEN-LIVE (2026-08-03) AND IS NOW THE THING OUT OF STEP."
|
||||||
worse_2026_08_22: "It failed AGAIN, unprompted, on 2026-08-21 21:59 - and the cause is worse than 'untested'. Hub event 3016: the PBS archive of 2026-08-18 could not be restored because the manifest's key does not match the key the rebuilt box now presents. The archives that predate the 2026-08-21 reinstall are UNREADABLE to the machine that made them (R-366). Credit where due: the mechanism caught it and named the key mismatch precisely. The gap is that it is reported as 'a restore test failed' rather than 'your older whole-machine backups cannot be opened on this box'."
|
worse_2026_08_22: "It failed AGAIN, unprompted, on 2026-08-21 21:59 - and the cause is worse than 'untested'. Hub event 3016: the PBS archive of 2026-08-18 could not be restored because the manifest's key does not match the key the rebuilt box now presents. The archives that predate the 2026-08-21 reinstall are UNREADABLE to the machine that made them (R-366). Credit where due: the mechanism caught it and named the key mismatch precisely. The gap is that it is reported as 'a restore test failed' rather than 'your older whole-machine backups cannot be opened on this box'."
|
||||||
verified:
|
verified:
|
||||||
date: 2026-08-22
|
date: 2026-11-03
|
||||||
verdict: downgraded
|
verdict: upgraded
|
||||||
depth: needs-hardware
|
depth: source-read
|
||||||
- id: backup.fill-warning
|
- id: backup.fill-warning
|
||||||
band: journey
|
band: journey
|
||||||
stage: 5
|
stage: 5
|
||||||
title: "The customer is warned before a drive fills, per drive, in their own language"
|
title: "The customer is warned before a drive fills, per drive, in their own language"
|
||||||
status: partial
|
status: walked
|
||||||
note: "The warning is real and was SEEN firing on 2026-08-21 with the right Hungarian copy, naming the drive and the free space. What 'BEFORE' cannot survive is the cadence: the watcher runs once a day at 03:30 plus once at startup (R-363), so a filesystem that fills at 03:31 goes unannounced for ~24 h. Watched live: the 69 GB volume carrying all 40-class app data was filled to 99% and the watcher said nothing, while the backup reserve was already refusing an app per run and telling the hub about it."
|
note: "Walked 2026-11-03 on Tester 1: a data drive taken to 89 %; the watcher (every 10 minutes since R-363) warned at its next tick, 2 minutes later: the household's Hungarian mail names the drive and 5,2 GB free, once. The August gap (a daily check) is gone. The cadence leaves a drive up to 10 minutes."
|
||||||
sources:
|
sources:
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/fill-walk.log"
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/mail-fill-household.txt"
|
||||||
- capability-map: "The customer is warned BEFORE a filesystem fills"
|
- capability-map: "The customer is warned BEFORE a filesystem fills"
|
||||||
- register: "R-167"
|
- register: "R-167"
|
||||||
- register: "R-363"
|
- register: "R-363"
|
||||||
@@ -388,8 +406,8 @@ claims:
|
|||||||
from: walked
|
from: walked
|
||||||
reason: "a daily check cannot carry the word BEFORE; observed silent for the whole window a filesystem sat at 99%"
|
reason: "a daily check cannot carry the word BEFORE; observed silent for the whole window a filesystem sat at 99%"
|
||||||
verified:
|
verified:
|
||||||
date: 2026-08-22
|
date: 2026-11-03
|
||||||
verdict: downgraded
|
verdict: upgraded
|
||||||
depth: source-read
|
depth: source-read
|
||||||
- id: backup.sikeres
|
- id: backup.sikeres
|
||||||
band: journey
|
band: journey
|
||||||
@@ -410,59 +428,69 @@ claims:
|
|||||||
band: journey
|
band: journey
|
||||||
stage: 6
|
stage: 6
|
||||||
title: "The machine watches itself and repairs some faults without telling anyone it had to"
|
title: "The machine watches itself and repairs some faults without telling anyone it had to"
|
||||||
status: built
|
status: walked
|
||||||
note: "R-264 records that the self-heal counters reach the hub and are decoded nowhere."
|
note: "Walked 2026-10-29 night on scratch 9202: a crashed app part (kill -9 of paperless-redis) ran again in 1 s and the health probe read the app healthy; a guest reboot brought every app back in under 2 minutes, the boot sweep found nothing to start; no event, no banner, no mail. Docker's restart policy did the repair in both legs; the controller's own boot repair had nothing to do (its last proof: 6 of 6 hard resets, 2026-08-02). R-264 records that the self-heal counters reach the hub and are decoded nowhere."
|
||||||
sources:
|
sources:
|
||||||
- capability-map: "Box survives an unattended app or guest-network failure"
|
- capability-map: "Box survives an unattended app or guest-network failure"
|
||||||
- register: "R-264"
|
- register: "R-264"
|
||||||
|
- evidence: "audits/night-2026-10-28/part5-claims/selfheal/WALK.md"
|
||||||
|
- evidence: "audits/DIAG-bootrecon-drive-absent-2026-08-02.md"
|
||||||
changed:
|
changed:
|
||||||
from: walked
|
from: built
|
||||||
reason: "no walk document cited"
|
reason: "walked 2026-10-29 on scratch 9202 (a walk document is now cited)"
|
||||||
verified:
|
verified:
|
||||||
date: 2026-08-09
|
date: 2026-10-29
|
||||||
verdict: downgraded
|
verdict: upgraded
|
||||||
depth: source-read
|
depth: source-read
|
||||||
- id: fault.operator-email
|
- id: fault.operator-email
|
||||||
band: journey
|
band: journey
|
||||||
stage: 6
|
stage: 6
|
||||||
title: "Failures reach the operator by email, one mail per run, every failing app named"
|
title: "Failures reach the operator by email, one mail per run, every failing app named"
|
||||||
status: built
|
status: walked
|
||||||
note: "CONFIRMED against source: backup_run_failures digest is allowlisted (api/handler.go:1837), operator-only (dispatcher.go:423), templated (templates.go:48); recovery_unit_capture_failed is record-only (dispatcher.go:376). NOTE: R-182's register row still describes the PRE-FIX behaviour — see R-289."
|
note: "Seen in the mailbox: „Tester4: 3 of 8 apps failed to back up (manual run)\" names immich, nextcloud and paperless-ngx in one mail (2026-10-20); seven of this shape in 60 days."
|
||||||
sources:
|
sources:
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/mail-operator-digest.txt"
|
||||||
- capability-map: "A failed per-app Tier-1 backup reaches the OPERATOR — EVERY failing app, in ONE mail per run"
|
- capability-map: "A failed per-app Tier-1 backup reaches the OPERATOR — EVERY failing app, in ONE mail per run"
|
||||||
- register: "R-182"
|
- register: "R-182"
|
||||||
changed:
|
changed:
|
||||||
from: walked
|
from: walked
|
||||||
reason: "the digest is wired and source-verified, but no run of it has been observed delivering"
|
reason: "the digest is wired and source-verified, but no run of it has been observed delivering"
|
||||||
verified:
|
verified:
|
||||||
date: 2026-08-09
|
date: 2026-11-03
|
||||||
verdict: downgraded
|
verdict: upgraded
|
||||||
depth: source-read
|
depth: source-read
|
||||||
- id: fault.customer-email
|
- id: fault.customer-email
|
||||||
band: journey
|
band: journey
|
||||||
stage: 6
|
stage: 6
|
||||||
title: "Failures reach the customer"
|
title: "Failures reach the customer"
|
||||||
status: built
|
status: walked
|
||||||
note: "The customer leg is built; today's log shows customer-channel rows skipped as operator_only."
|
note: "Seen in the mailbox: the crash mail and the fill warning to tester1@ (2026-11-02/03), „A szerver nem elérhető!\" to tester4@ (2026-10-24), each in Hungarian. It needs an address on the household's page (Tester 1 had none)."
|
||||||
sources:
|
sources:
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/mail-crash-household.txt"
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/mail-fill-household.txt"
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/mail-operator-digest.txt"
|
||||||
- capability-map: "App crashes → customer notified (one event per transition, no flapping spam)"
|
- capability-map: "App crashes → customer notified (one event per transition, no flapping spam)"
|
||||||
verified:
|
verified:
|
||||||
date: 2026-08-09
|
date: 2026-11-03
|
||||||
verdict: confirmed
|
verdict: upgraded
|
||||||
depth: source-read
|
depth: source-read
|
||||||
- id: fault.already-paired
|
- id: fault.already-paired
|
||||||
band: journey
|
band: journey
|
||||||
stage: 6
|
stage: 6
|
||||||
title: "An already-paired box is still told to pair itself"
|
title: "An already-paired box no longer tells anyone to pair it — its console says there is nothing to do"
|
||||||
status: partial
|
status: walked
|
||||||
note: "Warning card."
|
note: "Was a warning card (the console kept showing the pairing code). Fixed by ISO 1.28.0 (R-535; R-214, R-235 closed). Walked 2026-10-29 00:04 on Tester 1 (paired, claimed): the console reads, in Hungarian and English, that there is nothing to do on this machine and no need to log in — no pairing code."
|
||||||
sources:
|
sources:
|
||||||
- register: "R-214"
|
- register: "R-214"
|
||||||
- register: "R-235"
|
- register: "R-235"
|
||||||
|
- evidence: "audits/night-2026-10-28/part5-claims/tester1-console-2026-10-29.png"
|
||||||
|
changed:
|
||||||
|
from: partial
|
||||||
|
reason: "the defect is closed and the console was read live on a paired box (2026-10-29)"
|
||||||
verified:
|
verified:
|
||||||
date: 2026-08-09
|
date: 2026-10-29
|
||||||
verdict: confirmed
|
verdict: upgraded
|
||||||
depth: register+map
|
depth: source-read
|
||||||
- id: recover.screen
|
- id: recover.screen
|
||||||
band: journey
|
band: journey
|
||||||
stage: 7
|
stage: 7
|
||||||
@@ -559,26 +587,33 @@ claims:
|
|||||||
- id: fail.drive-filling
|
- id: fail.drive-filling
|
||||||
band: failures
|
band: failures
|
||||||
title: "A drive is filling up"
|
title: "A drive is filling up"
|
||||||
status: built
|
status: walked
|
||||||
|
note: "Same walk as backup.fill-warning: warned before it filled; the file removed, nothing else was needed."
|
||||||
sources:
|
sources:
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/fill-walk.log"
|
||||||
- register: "R-167"
|
- register: "R-167"
|
||||||
changed:
|
changed:
|
||||||
from: walked
|
from: walked
|
||||||
reason: "no walk document cited"
|
reason: "no walk document cited"
|
||||||
verified:
|
verified:
|
||||||
date: 2026-08-09
|
date: 2026-11-03
|
||||||
verdict: downgraded
|
verdict: upgraded
|
||||||
depth: register+map
|
depth: source-read
|
||||||
- id: fail.app-crash
|
- id: fail.app-crash
|
||||||
band: failures
|
band: failures
|
||||||
title: "An app crashes — the email leg has never been confirmed end to end"
|
title: "An app crashes — the box stops it and both the operator and the household get a mail"
|
||||||
status: built
|
status: walked
|
||||||
note: "Consistent with fault.operator-email: the digest is wired but its delivery is unobserved."
|
note: "Walked 2026-11-02 on Tester 1: kill -9 of a throwaway app, 15 s apart; at 6 restarts in 10 minutes the box stopped and held it, the operator's mail arrived in 1 s; with an address on the household's page the Hungarian mail arrived too. The household's Start lifted it. One flaw: the mail ends with raw JSON (R-1363)."
|
||||||
sources:
|
sources:
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/crash-walk.log"
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/crash-walk-customer.log"
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/mail-crash-operator.txt"
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/mail-crash-household.txt"
|
||||||
|
- register: "R-1363"
|
||||||
- capability-map: "App crashes → customer notified (one event per transition, no flapping spam)"
|
- capability-map: "App crashes → customer notified (one event per transition, no flapping spam)"
|
||||||
verified:
|
verified:
|
||||||
date: 2026-08-09
|
date: 2026-11-03
|
||||||
verdict: confirmed
|
verdict: upgraded
|
||||||
depth: source-read
|
depth: source-read
|
||||||
- id: fail.power-cut
|
- id: fail.power-cut
|
||||||
band: failures
|
band: failures
|
||||||
@@ -686,28 +721,34 @@ claims:
|
|||||||
depth: register+map
|
depth: register+map
|
||||||
- id: fail.internet-down
|
- id: fail.internet-down
|
||||||
band: failures
|
band: failures
|
||||||
title: "The internet goes down — built, never walked"
|
title: "The internet goes down"
|
||||||
status: built
|
status: walked
|
||||||
note: "Needs a running box with WAN pulled."
|
note: "Walked 2026-11-03 on Tester 1: packets to the outside dropped on host and guest for 13 minutes, the line's route left in place. The dashboard answered on the home network, 10 apps ran, the public name gave 530; 60 s after the line came back the public name worked by itself, nothing to do. Slow: the Backups page took over 2 minutes (R-1362). An earlier try removed the route: the agent's self-heal put it back in about 100 s."
|
||||||
sources:
|
sources:
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/outage-wan.log"
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/outage-wan-3min.txt"
|
||||||
|
- register: "R-1362"
|
||||||
- capability-map: "LAN access when internet is down (lan_resolver)"
|
- capability-map: "LAN access when internet is down (lan_resolver)"
|
||||||
verified:
|
verified:
|
||||||
date: 2026-08-09
|
date: 2026-11-03
|
||||||
verdict: needs-hardware
|
verdict: upgraded
|
||||||
depth: needs-hardware
|
depth: source-read
|
||||||
- id: fail.hub-down
|
- id: fail.hub-down
|
||||||
band: failures
|
band: failures
|
||||||
title: "The hub is down"
|
title: "The hub is down"
|
||||||
status: built
|
status: partial
|
||||||
|
note: "Walked 2026-11-03 on Tester 1 (the hub's address dropped for 13 minutes; DooPlex untouched): the box works on — dashboard, apps, an install from the cached catalog, a crash-loop stop. What fails: every event sent in that time is lost after 9 s and nothing re-sends it, so the operator never heard of the held app (R-1361)."
|
||||||
sources:
|
sources:
|
||||||
|
- evidence: "audits/security-2026-11-02/part5/outage-hub.log"
|
||||||
|
- register: "R-1361"
|
||||||
- capability-map: "Config/state change round-trips in seconds (hub↔box immediacy"
|
- capability-map: "Config/state change round-trips in seconds (hub↔box immediacy"
|
||||||
changed:
|
changed:
|
||||||
from: walked
|
from: walked
|
||||||
reason: "no walk document cited"
|
reason: "no walk document cited"
|
||||||
verified:
|
verified:
|
||||||
date: 2026-08-09
|
date: 2026-11-03
|
||||||
verdict: downgraded
|
verdict: upgraded
|
||||||
depth: register+map
|
depth: source-read
|
||||||
- id: fail.broken-release
|
- id: fail.broken-release
|
||||||
band: failures
|
band: failures
|
||||||
title: "We ship a broken release — the guard is missing"
|
title: "We ship a broken release — the guard is missing"
|
||||||
@@ -757,13 +798,14 @@ claims:
|
|||||||
band: failures
|
band: failures
|
||||||
title: "A customer restores their own data with no help"
|
title: "A customer restores their own data with no help"
|
||||||
status: partial
|
status: partial
|
||||||
note: "CONTESTED-RESOLVED: the map's MISSING is about a NON-OPERATOR performing it; the walks prove the route, not the person. Neither record was wrong. SINCE 2026-08-21 there is a second, harder blocker and it is not about the person at all: for the 40 of 53 apps that declare no data drive the off-site restore REFUSES before it starts, telling the customer a running app 'nincs telepítve' and to reinstall it to the same place — which those apps give them no way to choose (R-356). Those are exactly the apps whose whole dataset is a named volume. Until that is fixed, most customers cannot self-restore off-site even in principle."
|
note: "Walked 2026-10-29 night on scratch 9202 through the household's restore button (the page's own POST): vikunja from the 02:30 local point in 79 s — the project made before the point came back, the one made after it is gone, the app runs, no hold; a restore that meets a busy volume puts the live data back (R-1256 H5). The route needs no help. Still open: no person who is not the operator has done it (claim.never-by-non-operator). The old off-site blocker (R-356) shipped in controller v0.219.0."
|
||||||
sources:
|
sources:
|
||||||
- capability-map: "A customer (not the operator) performs a restore via UI alone"
|
- capability-map: "A customer (not the operator) performs a restore via UI alone"
|
||||||
- register: "R-201"
|
- register: "R-201"
|
||||||
- register: "R-356"
|
- register: "R-356"
|
||||||
- evidence: "audits/REPORT-DRILL-backup-truth-2026-08-21.md"
|
- evidence: "audits/REPORT-DRILL-backup-truth-2026-08-21.md"
|
||||||
|
- evidence: "audits/night-2026-10-28/part5-claims/selfrestore-walk.log"
|
||||||
verified:
|
verified:
|
||||||
date: 2026-08-22
|
date: 2026-10-29
|
||||||
verdict: contested
|
verdict: confirmed
|
||||||
depth: source-read
|
depth: source-read
|
||||||
@@ -0,0 +1,173 @@
|
|||||||
|
# SPIKE — who is using the upload line, and does the app explain it? (2026-10-23 night)
|
||||||
|
|
||||||
|
Operator ruling 2026-10-23 19:22 (`CONTEXT.md`): measure the line; on a sustained high upload attribute it to an app
|
||||||
|
and ask the app; a mismatch is the alarm; the same shape for CPU; qBittorrent gets a default upload limit from the
|
||||||
|
measured line, household changes respected. Brief Part 2.1. Design written from these numbers: §6 here and
|
||||||
|
`architecture/08-alarm-ladder.md` §9 [DESIGN — proposed]. Evidence and tools: `SPIKE-upload-watch-2026-10-23/`.
|
||||||
|
|
||||||
|
Read first: `08-alarm-ladder.md` (R-388, §8.1), `07-backup-architecture.md` (the night chain: W = 02:30 local, the
|
||||||
|
off-site push after it, the proof at 03:30 UTC), `15-media-library.md`, `13-threat-model.md` (attacker j),
|
||||||
|
`audits/plex2shell-felhom-impact-2026-10-23/README.md` (the 1.5 TB Plex upload). Controller: `internal/metrics/`
|
||||||
|
(`container_metrics` — one row per container per minute, 30 days), `internal/monitor/healthcheck.go` (the instant
|
||||||
|
whole-box CPU warning), `internal/mediawire/` (the app clients and their credentials, `09` §3 decision 248).
|
||||||
|
|
||||||
|
Machines: the three test boxes READ ONLY (their stored metrics; their off-site record; tonight a one-second read of
|
||||||
|
the controller's own counter during the real push); scratch 9202 on demo-hp (throwaway qBittorrent, Transmission, Plex,
|
||||||
|
Jellyfin through the product's deploy); a throwaway receiver LXC **9471 `cc-uw-recv`** on demo-hp's scratch storage
|
||||||
|
(never the thin pool). Every flood went 9202 → 9471 on demo-hp's own bridge; nothing went toward the internet.
|
||||||
|
|
||||||
|
## 1. Normal levels (stored metrics, three boxes)
|
||||||
|
|
||||||
|
History: **demo-hp** 2026-09-23 02:00 → 10-23 18:09 UTC (30.7 d, 998 216 rows); **demo-felhom** (the N100) same span
|
||||||
|
(30.7 d, 268 120 rows); **Tester 1** 2026-10-04 19:41 → 10-23 18:08 (18.9 d, 294 433 rows). Per app: upload from the
|
||||||
|
`net_tx_bytes` deltas (one-minute samples), CPU in percent of ONE core. Full tables: `s1-levels-per-app.txt`.
|
||||||
|
|
||||||
|
| box | the busiest uploaders (p50 / p95 / max Mbit/s) | the busiest CPU (p95 / max % of a core) |
|
||||||
|
|---|---|---|
|
||||||
|
| demo-hp | qbittorrent 0.02 / 1.2 / 4.0; immich-server 0.03 / 0.13 / 243 (one minute); felhom-controller 0 / 0.003 / 17.8 (the push) | grimmory 272 / 305 (a scan, 94 samples); immich-ML 32 / 548; romm 3 / 442; paperless 0.3 / 404 |
|
||||||
|
| demo-felhom | immich-server 0.67 / 5.95 / 12.9 (32 samples); jellyfin 0 / 0.85 / 2.9 | immich-ML 295 / 296 (an import); onlyoffice 131 / 148; paperless 137 / 140 |
|
||||||
|
| Tester 1 | qbittorrent 0.007 / 0.13 / 42.4; transmission 0.001 / 0.013 / 57.4; controller 0 / 0.003 / 37.5 | paperless 0.7 / 508; controller 46 / 423; plex 4.3 / 198 |
|
||||||
|
|
||||||
|
**How long a normal high lasts** (`s1-longest-runs.txt` — the longest run of consecutive minutes): upload ≥ 5 Mbit/s:
|
||||||
|
**3 min** at most (qBittorrent, Tester 1); ≥ 1 Mbit/s: 9 min. CPU ≥ 150 %: **7 min** at most (Immich machine learning);
|
||||||
|
≥ 80 %: 13 min (Kimai). Busiest upload hours (UTC): demo-hp 03h, 02h; demo-felhom 16h, 05h; Tester 1 07h, 18h — no fixed
|
||||||
|
pattern beyond the night push. For scale: DooPlex's hacked Plex sent ~474 GB in one day (≈ 44 Mbit/s on average, for
|
||||||
|
hours) and used ~10 cores.
|
||||||
|
|
||||||
|
`felhom-samba` runs with the host network: its counter is always 0 (no own network namespace) — the watch cannot see it.
|
||||||
|
|
||||||
|
## 2. The line without a test
|
||||||
|
|
||||||
|
**What the box already sees:** the off-site push (restic, inside the controller container) is the box's only big
|
||||||
|
upload of its own. At the stored one-minute sample (`s2-controller-push-per-night.txt`): the controller's best five
|
||||||
|
minutes per night exceeded 2 Mbit/s on **2 of 31 nights** on demo-hp (5.1 and 6.9 Mbit/s; best single minute 17.0),
|
||||||
|
**1 of 19** on Tester 1 (7.6; best minute 37.5) and **0 of 31** on demo-felhom. The off-site record keeps the push's
|
||||||
|
duration (33 s on demo-felhom, 1 m 8 s on Tester 1, 3 m 19 s on demo-hp on 2026-10-23) and **no byte count**.
|
||||||
|
|
||||||
|
**Conclusion:** at a one-minute sample the push shows the night's DATA SIZE (a few MB, incremental), not the line. Not
|
||||||
|
enough as it stands. Two changes make it usable: sample once a SECOND during the push (the controller knows when its
|
||||||
|
push runs and can read its own counter), and let a night count only when its best 10 seconds moved ≥ 10 MB. Even then
|
||||||
|
it is a **lower bound** (restic over SFTP to the Storage Box may not fill the line; the family uploads too).
|
||||||
|
|
||||||
|
**Tonight's one-second read of the real push** (2026-10-24 ~02:16Z, three boxes, read only): §2.1 below.
|
||||||
|
|
||||||
|
### 2.1 One-second samples of the real push
|
||||||
|
|
||||||
|
Read 2026-10-24 02:08–02:30Z, once a second, the controller container's own transmit counter (the counter the watch
|
||||||
|
reads: `eth0` in the controller's network namespace), on the three test boxes, read only
|
||||||
|
(`SPIKE-upload-watch-2026-10-23/night-push-1s-<box>.txt`; the reader `tools/nightsample.sh`).
|
||||||
|
|
||||||
|
| Box | Sent in the 22 min | Best 10 s | 1-second peak | Counts (≥ 10 MiB in the best 10 s)? |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| demo-hp | 19.3 MB | **12.6 MB in 10.1 s = 10.0 Mbit/s** (02:17:32Z) | 70.3 Mbit/s | **yes** → line „at least 10 Mbit/s" |
|
||||||
|
| demo-felhom | 3.3 MB | 3.2 MB = 2.5 Mbit/s (02:15:22Z) | 13.7 Mbit/s | no |
|
||||||
|
| Tester 1 | 3.6 MB | 3.5 MB = 2.7 Mbit/s (02:15:40Z) | 14.7 Mbit/s | no |
|
||||||
|
|
||||||
|
**What it says:** one night in three counted. The push is a burst of a few seconds (3–9 seconds above 100 kB/s), so
|
||||||
|
even the counting night gives a low floor: 10 Mbit/s, while one second of the same push reached 70 Mbit/s. The design
|
||||||
|
keeps the best counting night of the last 14, so the line rises only when a bigger night comes. **Consequence for the
|
||||||
|
qBittorrent limit (R-1245):** 50 % of tonight's floor is ~5 Mbit/s on demo-hp — likely far below its real line. This is
|
||||||
|
the known lower-bound limit (R-1247), now measured; it is written into R-1245 so the operator decides with it.
|
||||||
|
|
||||||
|
## 3. The line with a test — NOT run tonight
|
||||||
|
|
||||||
|
| Candidate | Ours? | Terms | Cost per test | Tonight |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| ep0 (a receiver on our Hetzner box) | yes | ours | 10 s × line ≈ 125 MB at 100 Mbit/s | **not built** — a change on ep0 is forbidden tonight |
|
||||||
|
| the box's own Storage Box sub-account, through the same transport as the push | yes (Hetzner, unlimited traffic) | Hetzner's — ours | same | not tried — a test file in the account that holds the restic repository needs its own design (the repo check, the quota) |
|
||||||
|
| Cloudflare R2 (where the ISO lives) | yes | ours | same + R2 class-A writes | not used — its write key is the operator's ISO-publish key; a test object in a public bucket is wrong |
|
||||||
|
| public speed tests (speed.cloudflare.com, LibreSpeed servers) | no | **not read** | — | not used (the brief: no third-party test without its terms) |
|
||||||
|
|
||||||
|
A quiet hour: never 02:30–05:30 local (dumps 02:30, second drive, off-site push, the 03:30 UTC proof). The family shares
|
||||||
|
the line, so take the highest of several runs. **Not needed for the first build** — §6 works with the push's lower
|
||||||
|
bound and an absolute floor; an active test is a decide row (R-1247).
|
||||||
|
|
||||||
|
## 4. Can the box ask each app? (measured on 9202 with throwaway installs; a private torrent 9202 → 9471)
|
||||||
|
|
||||||
|
Set-up: qBittorrent 5.2.4 and Transmission 4.1.3 installed through the product (`POST /api/stacks/<n>/deploy`; the
|
||||||
|
first try was refused BEFORE anything was made — the data place must be the drive itself; two empty helper folders
|
||||||
|
removed). The same 2 GiB file (an AES-CTR keystream of a fixed key) on both sides; a private torrent; a 40-line HTTP
|
||||||
|
tracker on 9471 (`tools/tracker.py`); `transmission-daemon` on 9471 as the leecher. The admin passwords were read from
|
||||||
|
the containers at use and never printed.
|
||||||
|
|
||||||
|
| App | Its own number | Measured | Limit | Household's limit visible? |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| **qBittorrent** | `/api/v2/transfer/info` → `up_info_data` (cumulative), `up_info_speed` | seeding: app **131.1 Mbit/s** vs container **134.6–140.5** → **×1.03–1.07** (`s4-qb-seed-nolimit.jsonl`) | `transfer/setUploadLimit` (bytes/s) = `preferences.up_limit`; stored in KiB/s (2 500 000 → **2 499 584**); survives a restart (`Session\GlobalUPSpeedLimit=2441`); measured: ~20 Mbit/s held (`s4-qb-seed-limit20.jsonl`) | yes: `transfer/uploadLimit` ≠ the value the box stored |
|
||||||
|
| **Transmission** | RPC `session-stats.current-stats.uploadedBytes`, `uploadSpeed` | seeding: app 487.1 / 390.3 / 115.8 vs container 487.6 / 390.7 / 117.9 Mbit/s → **×1.001–1.02** (`s4-tr-seed-nolimit.jsonl`) | `session-set speed-limit-up` (kB/s) + `-enabled` (read back); **its rate under the limit was NOT observed** — the re-added leecher got no connection in 2 minutes | yes: `speed-limit-up-enabled` + value |
|
||||||
|
| **Plex** | `/status/sessions`, `/transcode/sessions` (`MediaContainer.size`) | unclaimed 1.43.5: both 200 without a token from the docker network, `{"size":0}`; a claimed server needs its PlexOnlineToken (read at use from Preferences.xml, as mediawire does — no new secret) | — | — |
|
||||||
|
| **Jellyfin** | `/Sessions` | **401** without a key; the key named `felhom` exists only when the household made it (mediawire's „one household step") — the box cannot create one at install without the admin login it does not have | — | — |
|
||||||
|
| Nextcloud, Immich | **no upload-explaining number found** (not measured further); Immich's job queue needs an API key the box does not hold | — | — | → the plain rule (§6) |
|
||||||
|
|
||||||
|
**The fake hacked app** (code inside qBittorrent's own container, `docker exec … python3` → 9471:5001, Plex2Shell's
|
||||||
|
shape): the container sent **~10 100 Mbit/s** (LAN, one bridge) while qBittorrent reported **0** (`s4-qb-flood.jsonl`).
|
||||||
|
|
||||||
|
**The counter itself:** the Docker Engine API's one-shot stats (`networks.*.tx_bytes`) equal the kernel's counter in
|
||||||
|
the container's network namespace to the byte; `docker stats`' text (what `internal/metrics` stores) has three
|
||||||
|
significant digits — ~10 MB steps at a GB total, ~1.3 Mbit/s at a one-minute sample. The watch reads the Engine API.
|
||||||
|
|
||||||
|
**A limit the hacked app controls:** an attacker with code in the app's container can also answer its API. The check
|
||||||
|
catches an attacker who does not know about it; one who does could lie. §5 is the answer to that.
|
||||||
|
|
||||||
|
## 5. A universal upload ceiling — spike only
|
||||||
|
|
||||||
|
A helper container sharing the app's network namespace (`docker run --rm --network container:qbittorrent --cap-add
|
||||||
|
NET_ADMIN alpine:3.20` + `iproute2-tc`) added `tc qdisc … root tbf rate 20mbit burst 64kb latency 400ms` on the app's
|
||||||
|
`eth0` **inside the unprivileged 9202 guest**. The same flood then ran at **19.9–20.1 Mbit/s** (`s4-qb-flood-tbf.jsonl`),
|
||||||
|
qBittorrent's API kept answering, and `tc qdisc del` restored it. Costs: a helper image with `tc` (a new external image,
|
||||||
|
or `tc` shipped in the controller image), `NET_ADMIN` for that helper, re-applied after every app restart (the qdisc
|
||||||
|
dies with the namespace), upload only. **Not built** (brief). It would cap a hacked app's flood whatever its API says.
|
||||||
|
|
||||||
|
## 6. The design (numbers from §1–§5; full text `08-alarm-ladder.md` §9)
|
||||||
|
|
||||||
|
1. **Line:** one-second samples of the controller's own sent bytes during the off-site push; the best ≥ 10-second rate
|
||||||
|
per night; a night counts when that window moved ≥ 10 MB; the line = the highest counting night of the last 14;
|
||||||
|
settings.json `upwatch.line`; shown on the system settings page as „legalább N Mbit/s" („még mérjük" until known).
|
||||||
|
2. **Trigger:** an app (its containers summed, exact counters, once a minute) sends ≥ **max(5 Mbit/s, 25 % of the line)**
|
||||||
|
for **15 minutes** — five times the longest normal run at ≥ 5 Mbit/s (3 min). Per app, not per box: the box cannot
|
||||||
|
tell LAN from internet bytes inside its controller container, and the per-app view is what the check compares.
|
||||||
|
3. **Attribution:** the app's compose project; infrastructure (controller, traefik, cloudflared, Samba) never judged.
|
||||||
|
4. **Explanation:** qBittorrent / Transmission: explained when container bytes ≤ 1.25 × the app's own count + 1 Mbit/s
|
||||||
|
(measured overhead 1–7 %); Plex: any viewer explains the upload (session bandwidth not measured — the conservative
|
||||||
|
half; no viewer + a big upload = the Plex2Shell signature). Not explained → **`app_unexplained_upload`**,
|
||||||
|
operator-only, one per episode, a clear after 15 quiet minutes. No explainer: alarm only at ≥ 10 × its own p95 for
|
||||||
|
60 minutes, after ~20 h of history, worded „no app number to compare".
|
||||||
|
5. **CPU:** ≥ **max(150 %, 50 % × cores)** of one core for **30 minutes** (four times the longest normal run at ≥ 150 %),
|
||||||
|
explained by a Plex transcode or the box's own work on the app (backup, update, deploy) → else
|
||||||
|
**`app_unexplained_cpu`**. No explainer: ≥ max(200 %, 75 % × cores) for 60 minutes.
|
||||||
|
6. **qBittorrent limit:** 50 % of the line (CC's pick — R-1245), every 15 minutes, re-set when the line moves > 20 %;
|
||||||
|
never over a household's limit (a limit there first, or any change from the stored value). No line → nothing set.
|
||||||
|
The household's line (HU + EN in the report).
|
||||||
|
7. **Edge cases:** the night push is the controller's (never judged); the first day: apps with an explainer are judged
|
||||||
|
on the floor (a mismatch is evidence whatever the line), apps without one are not judged before ~20 h of history; a
|
||||||
|
slow or changing line (a Wi-Fi laptop): the line is a max over 14 nights — a falling line lowers nothing until it
|
||||||
|
rolls out (R-1247 notes the active test); two apps at once: each judged on its own bytes; the household's own devices:
|
||||||
|
invisible to the box — the trigger uses the box's own bytes only; an app reinstalled from scratch reads as „the
|
||||||
|
household removed the limit" and is left alone.
|
||||||
|
|
||||||
|
## 7. Live proofs on 9202 (controller `0.342.0-rc.1`, commit `a9c1997`, CI run 2270 success)
|
||||||
|
|
||||||
|
Set by hand in 9202's `/etc/felhom-controller-image` (an `-rc` tag is never offered to a household); 9202's hub link
|
||||||
|
is off by design, so every event was handed to the notifier and DROPPED there — the hub side (allow-list, operator-only,
|
||||||
|
the mail line) is proven by hub 0.169.0's tests. 9202's off-site is off, so one line record (40 Mbit/s) was seeded by
|
||||||
|
hand into its settings.json. Logs: `SPIKE-upload-watch-2026-10-23/live/2-9202-watch-log-full.txt`.
|
||||||
|
|
||||||
|
| # | What | Result (UTC) |
|
||||||
|
|---|---|---|
|
||||||
|
| 1 | qBittorrent unlimited, line 40 Mbit/s → the box sets the limit | **19:47:42** „qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)" — read back from qBittorrent: 2 499 584 |
|
||||||
|
| 2 | **a real torrent seeding at the limit** (5 GB, private, 9202 → 9471) | **20:04:42** „qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s)"; no alarm (counter ratio 1.04) |
|
||||||
|
| 3 | **a CPU burner in Plex with no transcode** (4 busy loops, 400 % on a 7-core guest; trigger 350 %) | **20:07:42** „plex used 399% CPU … for 30 min with no work that explains it — operator alarm" → `app_unexplained_cpu`; ONE |
|
||||||
|
| 4 | **the fake hacked app**: a 50 Mbit/s flood from inside qBittorrent's container to 9471, hidden behind the real seed | **20:19:42** „qbittorrent sent 65.0 Mbit/s for 15 min — NOT explained by the app (qbittorrent: 20.0 Mbit/s) — operator alarm" → `app_unexplained_upload`; ONE, none more while it ran |
|
||||||
|
| 5 | the burner stopped 20:11 | **20:25:41** „plex: the unexplained CPU stopped … — cleared" → `app_unexplained_cpu_cleared`; ONE |
|
||||||
|
| 6 | the household changes qBittorrent's limit by hand (1 500 000 → stored 1 499 136) at 20:21 | **20:32:42** „qBittorrent's upload limit is the household's — the box leaves it alone from now on"; read back at 20:43: still 1 499 136 |
|
||||||
|
| 7 | the flood stopped 20:26 (the seed went on, explained) | **20:41:41** „qbittorrent: the unexplained upload stopped … — cleared" → `app_unexplained_upload_cleared`; ONE |
|
||||||
|
| 8 | the system settings page (`GET /settings`) | „Feltöltési sebesség: legalább 40 Mbit/s (az éjszakai távoli mentésből mérve)"; qBittorrent's page carried no limit line after step 6 (the household's) — the box-owned branch is pinned by `TestUpwatch_QBLimitLineOnQBittorrentsPageOnly`, not captured live |
|
||||||
|
|
||||||
|
**Found live:** the very first minute after a start failed to reach the apps by name (the bootstrap joins the controller
|
||||||
|
to the app network a few seconds after the start); every later minute answered. That first minute only sets baselines —
|
||||||
|
**NOT-A-FINDING**. The media wiring's generated `docker-compose.override.yml` stays beside the template after a removal
|
||||||
|
(Jellyfin tonight; Plex's from the morning session) — rewritten before every start, harmless — **NOT-A-FINDING**.
|
||||||
|
|
||||||
|
**Teardown, three layers:** machine — the four throwaway apps removed through the product (containers and volumes gone,
|
||||||
|
verified), my test files and probes deleted, the `alpine:3.20` helper image removed, 9202 back on 0.341.0 with the seeded
|
||||||
|
record removed, the rc image removed; host — the receiver LXC 9471 destroyed (`--purge`, its disk gone); hub — none (9202
|
||||||
|
is not on the hub). The night sampler on the three test boxes is reads only.
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
gitea.dooplex.hu/admin/felhom-controller:0.342.0-rc.1
|
||||||
|
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0-rc.1 Up 6 minutes (healthy)
|
||||||
|
filebrowser gtstef/filebrowser:1.5.6-stable Up 4 hours (healthy)
|
||||||
|
immich-machine-learning ghcr.io/immich-app/immich-machine-learning:v3.2.4 Up 4 hours (healthy)
|
||||||
|
immich-postgres ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0 Up 4 hours (healthy)
|
||||||
|
immich-redis redis:7-alpine Up 4 hours (healthy)
|
||||||
|
immich-server ghcr.io/immich-app/immich-server:v3.2.4 Up 4 hours (healthy)
|
||||||
|
jellyfin jellyfin/jellyfin:10.11.11 Up About an hour (healthy)
|
||||||
|
paperless-postgres postgres:18-alpine Up 4 hours (healthy)
|
||||||
|
paperless-redis redis:7-alpine Up 4 hours (healthy)
|
||||||
|
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15 Up 4 hours (healthy)
|
||||||
|
plex plexinc/pms-docker:1.43.5.11029-6bf8af3f0 Up About an hour (healthy)
|
||||||
|
qbittorrent qbittorrentofficial/qbittorrent-nox:5.2.4-1 Up About an hour (healthy)
|
||||||
|
traefik traefik:v3.7.13 Up 4 hours
|
||||||
|
transmission lscr.io/linuxserver/transmission:version-4.1.3-r0 Up About an hour (healthy)
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
VMID Status Lock Name
|
||||||
|
9201 running demo-hp
|
||||||
|
9202 running demo-hp-scratch
|
||||||
|
9401 stopped upgrade-harness
|
||||||
|
9471 running cc-uw-recv
|
||||||
|
description: CC throwaway 2026-10-23 night%3A upload-watch spike receiver/leecher. Remove after the session.%0A
|
||||||
|
hostname: cc-uw-recv
|
||||||
|
net0: name=eth0,bridge=vmbr0,hwaddr=BC:24:11:04:68:80,ip=dhcp,type=veth
|
||||||
|
rootfs: nvme-scratch:9471/vm-9471-disk-0.raw,size=8G
|
||||||
|
tags: cc;scratch;uw-spike
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
2026/10/23 19:32:41 upwatch_wiring.go:97: [INFO] [upwatch] upload and CPU watch started (line at least 40 Mbit/s, qBittorrent share 50 %)
|
||||||
|
2026/10/23 19:32:41 watch.go:237: [DEBUG] [upwatch] plex: the app did not answer (resolving the plex container: lookup plex on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||||
|
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] transmission: the app did not answer (Post "http://transmission:9091/transmission/rpc": dial tcp: lookup transmission on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||||
|
2026/10/23 19:32:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_definition_refused (severity warning) — further app_definition_refused events are logged at DEBUG only
|
||||||
|
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] qbittorrent: the app did not answer (Post "http://qbittorrent:8080/api/v2/auth/login": dial tcp: lookup qbittorrent on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||||
|
2026/10/23 19:47:42 service.go:108: [INFO] [upwatch] qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)
|
||||||
|
2026/10/23 20:04:42 watch.go:303: [INFO] [upwatch] qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)); normal use
|
||||||
|
2026/10/23 20:07:42 watch.go:343: [WARN] [upwatch] plex used 399% CPU (of one core; 7 cores) for 30 min with no work that explains it (plex transcodes) — operator alarm
|
||||||
|
2026/10/23 20:07:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu (severity warning) — further app_unexplained_cpu events are logged at DEBUG only
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
2026/10/23 19:32:41 upwatch_wiring.go:97: [INFO] [upwatch] upload and CPU watch started (line at least 40 Mbit/s, qBittorrent share 50 %)
|
||||||
|
2026/10/23 19:32:41 watch.go:237: [DEBUG] [upwatch] plex: the app did not answer (resolving the plex container: lookup plex on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||||
|
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] transmission: the app did not answer (Post "http://transmission:9091/transmission/rpc": dial tcp: lookup transmission on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||||
|
2026/10/23 19:32:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_definition_refused (severity warning) — further app_definition_refused events are logged at DEBUG only
|
||||||
|
2026/10/23 19:32:42 watch.go:237: [DEBUG] [upwatch] qbittorrent: the app did not answer (Post "http://qbittorrent:8080/api/v2/auth/login": dial tcp: lookup qbittorrent on 192.168.0.250:53: no such host) — treated as no number this minute
|
||||||
|
2026/10/23 19:47:42 service.go:108: [INFO] [upwatch] qBittorrent upload limit set to 2499584 B/s (50 % of the measured line, 40.0 Mbit/s)
|
||||||
|
2026/10/23 20:04:42 watch.go:303: [INFO] [upwatch] qbittorrent sent 20.8 Mbit/s for 15 min — explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)); normal use
|
||||||
|
2026/10/23 20:07:42 watch.go:343: [WARN] [upwatch] plex used 399% CPU (of one core; 7 cores) for 30 min with no work that explains it (plex transcodes) — operator alarm
|
||||||
|
2026/10/23 20:07:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu (severity warning) — further app_unexplained_cpu events are logged at DEBUG only
|
||||||
|
2026/10/23 20:19:42 watch.go:307: [WARN] [upwatch] qbittorrent sent 65.0 Mbit/s for 15 min — NOT explained by the app (qbittorrent: 20.0 Mbit/s, 0 viewer(s)) — operator alarm
|
||||||
|
2026/10/23 20:19:42 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_upload (severity warning) — further app_unexplained_upload events are logged at DEBUG only
|
||||||
|
2026/10/23 20:25:41 watch.go:378: [INFO] [upwatch] plex: the unexplained CPU stopped (15 min below the trigger or explained) — cleared
|
||||||
|
2026/10/23 20:25:41 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_cpu_cleared (severity info) — further app_unexplained_cpu_cleared events are logged at DEBUG only
|
||||||
|
2026/10/23 20:32:42 service.go:111: [INFO] [upwatch] qBittorrent's upload limit is the household's — the box leaves it alone from now on
|
||||||
|
2026/10/23 20:41:41 watch.go:370: [INFO] [upwatch] qbittorrent: the unexplained upload stopped (15 min below the trigger or explained) — cleared
|
||||||
|
2026/10/23 20:41:41 notifier.go:1463: [WARN] notifier disabled (no hub configured): DROPPED event app_unexplained_upload_cleared (severity info) — further app_unexplained_upload_cleared events are logged at DEBUG only
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
/settings 56743 data-uplink> <span class="settings-label">Feltöltési sebesség</span> <span class="settings-value">legalább <span class="mono">40</span> Mbit/s (az éjszakai távoli mentésből mérve)</span> </div> | qb line: none
|
||||||
|
/apps/qbittorrent 49281 no uplink row | qb line: none
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
version v5.2.4 webapi 2.15.1
|
||||||
|
transfer/info {'connection_status': 'connected', 'dht_nodes': 375, 'dl_info_data': 0, 'dl_info_speed': 0, 'dl_rate_limit': 0, 'last_external_address_v4': '37.191.56.193', 'last_external_address_v6': '', 'up_info_data': 5566595787, 'up_info_speed': 0, 'up_rate_limit': 1499136}
|
||||||
|
prefs {'up_limit': 1499136, 'dl_limit': 0, 'alt_up_limit': 10240, 'alt_dl_limit': 10240, 'scheduler_enabled': False, 'limit_tcp_overhead': False, 'limit_lan_peers': True, 'limit_utp_rate': True, 'max_uploads': 20, 'enable_upload_suggestions': False}
|
||||||
|
uploadLimit 1499136 speedLimitsMode 0
|
||||||
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,186 @@
|
|||||||
|
|
||||||
|
=== demo-hp: 2026-09-23 02:00 -> 2026-10-23 18:09 UTC (30.7 d), 998216 rows
|
||||||
|
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
|
||||||
|
qbittorrent 18 0.019 1.199 4.01 | 0.4 5.4 11.3 |
|
||||||
|
metube 3 0.146 0.146 0.15 | 0.0 0.1 0.1 |
|
||||||
|
immich-server 318 0.031 0.133 242.93 | 0.1 5.6 138.1 | resets=7
|
||||||
|
grimmory-db 102 0.000 0.116 0.35 | 0.0 16.9 102.5 | resets=12
|
||||||
|
immich-postgres 320 0.000 0.066 2.20 | 0.0 8.7 70.2 | resets=7
|
||||||
|
grimmory 94 0.001 0.047 0.77 | 1.1 272.0 305.0 | resets=9
|
||||||
|
immich-redis 320 0.013 0.029 0.18 | 0.6 4.3 5.1 | resets=7
|
||||||
|
romm 26492 0.019 0.027 0.20 | 0.7 3.2 442.4 | resets=97
|
||||||
|
docmost 43499 0.013 0.027 0.23 | 0.1 0.8 158.1 | resets=147
|
||||||
|
rallly 71 0.000 0.017 0.12 | 0.0 8.5 152.1 | resets=10
|
||||||
|
radarr 16 0.002 0.015 0.03 | 0.2 7.2 11.8 |
|
||||||
|
romm-redis 26523 0.005 0.013 0.10 | 0.7 4.4 61.5 | resets=101
|
||||||
|
paperless-webserver 43488 0.005 0.013 0.04 | 0.1 0.3 404.2 | resets=144
|
||||||
|
nextcloud-db 663 0.001 0.013 1.74 | 0.0 6.1 71.8 | resets=11
|
||||||
|
docmost-redis 43532 0.005 0.013 0.13 | 0.7 4.4 6.6 | resets=147
|
||||||
|
cloudflared 43604 0.000 0.013 1.21 | 0.3 0.5 13.8 | resets=6
|
||||||
|
rallly-postgres 71 0.000 0.007 0.01 | 0.0 5.0 8.8 | resets=7
|
||||||
|
prowlarr 5 0.002 0.004 0.00 | 0.1 0.2 0.2 |
|
||||||
|
sonarr 18 0.000 0.004 0.01 | 2.5 8.2 8.8 |
|
||||||
|
felhom-controller 43608 0.000 0.003 17.79 | 1.0 57.9 251.4 | resets=133
|
||||||
|
paperless-redis 43521 0.001 0.003 0.02 | 0.6 4.3 6.1 | resets=148
|
||||||
|
bentopdf 43567 0.000 0.003 0.01 | 0.0 0.0 5.2 | resets=58
|
||||||
|
homepage 78 0.000 0.002 0.01 | 0.0 4.2 4.5 | resets=1
|
||||||
|
bookstack-db 43565 0.001 0.001 0.01 | 0.0 6.1 102.8 | resets=147
|
||||||
|
romm-db 26526 0.000 0.001 0.19 | 0.0 6.0 105.3 | resets=89
|
||||||
|
plex 11974 0.001 0.001 1.10 | 0.3 0.6 124.6 | resets=46
|
||||||
|
nextcloud 662 0.000 0.001 0.80 | 0.0 5.7 150.0 | resets=11
|
||||||
|
immich-machine-learning 320 0.000 0.001 0.39 | 0.2 32.0 548.0 | resets=7
|
||||||
|
docmost-postgres 43533 0.000 0.001 0.02 | 0.0 4.7 29.6 | resets=147
|
||||||
|
bookstack 43546 0.000 0.001 0.01 | 0.0 0.0 63.0 | resets=148
|
||||||
|
adventurelog-postgres 43567 0.000 0.001 1.35 | 0.0 4.7 37.9 | resets=138
|
||||||
|
adventurelog 43549 0.000 0.001 2.16 | 0.0 0.7 204.3 | resets=146
|
||||||
|
ghost 20 0.000 0.001 0.00 | 0.0 6.8 132.9 | resets=2
|
||||||
|
glance 20 0.000 0.001 0.00 | 0.0 0.0 0.0 | resets=2
|
||||||
|
privatebin 43518 0.000 0.001 0.00 | 0.1 0.1 8.1 | resets=140
|
||||||
|
paperless-postgres 43520 0.000 0.001 0.05 | 0.0 4.7 19.1 | resets=148
|
||||||
|
kimai 30136 0.000 0.000 0.00 | 0.0 0.0 107.2 | resets=102
|
||||||
|
home-assistant 14403 0.000 0.000 1.00 | 0.0 0.5 106.9 | resets=62
|
||||||
|
kimai-db 30150 0.000 0.000 0.05 | 0.0 5.9 100.8 | resets=96
|
||||||
|
adventurelog-frontend 43559 0.000 0.000 0.21 | 0.0 3.8 22.8 | resets=126
|
||||||
|
vaultwarden 329 0.000 0.000 0.00 | 0.0 0.1 4.0 | resets=7
|
||||||
|
calibre-web 43547 0.000 0.000 0.01 | 0.0 0.3 127.1 | resets=145
|
||||||
|
opengist 43530 0.000 0.000 0.01 | 0.0 0.0 9.8 | resets=140
|
||||||
|
jellyfin 297 0.000 0.000 5.87 | 0.0 4.4 160.5 | resets=7
|
||||||
|
vikunja 8 0.000 0.000 0.00 | 0.0 0.0 0.0 | resets=2
|
||||||
|
a1-echo 113 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||||
|
upbeat_perlman 1 0.000 0.000 0.00 | 31.2 31.2 31.2 |
|
||||||
|
traefik 43608 0.000 0.000 1.21 | 0.1 3.5 92.1 | resets=11
|
||||||
|
practical_johnson 1 0.000 0.000 0.00 | 9.9 9.9 9.9 |
|
||||||
|
peaceful_ptolemy 1 0.000 0.000 0.00 | 14.1 14.1 14.1 |
|
||||||
|
nextcloud-redis 663 0.000 0.000 0.03 | 0.6 4.1 7.9 | resets=5
|
||||||
|
friendly_wilbur 1 0.000 0.000 0.00 | 59.0 59.0 59.0 |
|
||||||
|
filebrowser 43607 0.000 0.000 0.74 | 0.0 0.0 35.1 | resets=17
|
||||||
|
felhom-samba 10 0.000 0.000 0.00 | 0.0 13.5 17.6 | tx counter always 0 (host network?)
|
||||||
|
determined_satoshi 1 0.000 0.000 0.00 | 17.4 17.4 17.4 |
|
||||||
|
busiest UTC hours by bytes sent: 03h 14%, 02h 7%, 13h 5%, 00h 4%
|
||||||
|
|
||||||
|
=== demo-felhom: 2026-09-23 02:00 -> 2026-10-23 18:08 UTC (30.7 d), 268120 rows
|
||||||
|
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
|
||||||
|
immich-server 32 0.667 5.953 12.90 | 0.6 120.1 130.8 |
|
||||||
|
jellyfin 16 0.000 0.854 2.85 | 0.0 72.7 290.7 |
|
||||||
|
immich-postgres 32 0.000 0.766 2.24 | 0.3 8.5 197.6 |
|
||||||
|
romm 16 0.020 0.339 1.07 | 0.5 25.5 100.5 |
|
||||||
|
dawarich 20 0.000 0.313 3.01 | 0.1 5.3 104.4 |
|
||||||
|
immich-machine-learning 32 0.003 0.243 0.42 | 0.2 294.7 295.7 |
|
||||||
|
immich-redis 32 0.017 0.144 0.16 | 0.5 3.9 4.2 |
|
||||||
|
nextcloud-db 16 0.000 0.137 0.43 | 0.0 4.9 5.8 |
|
||||||
|
nextcloud 16 0.000 0.099 0.33 | 0.0 21.7 86.7 |
|
||||||
|
paperless-webserver 21 0.005 0.073 0.08 | 0.0 137.4 139.7 |
|
||||||
|
paperless-postgres 22 0.000 0.048 0.06 | 0.0 4.6 5.0 |
|
||||||
|
kimai-db 19 0.000 0.023 0.15 | 0.0 8.4 38.8 |
|
||||||
|
dawarich-sidekiq 19 0.020 0.023 0.02 | 0.1 0.2 0.2 |
|
||||||
|
claper 20 0.019 0.020 0.02 | 0.5 0.7 0.9 |
|
||||||
|
docmost 22 0.020 0.020 0.02 | 0.0 0.2 0.3 |
|
||||||
|
wanderer 19 0.002 0.015 0.02 | 0.0 0.0 0.0 |
|
||||||
|
cloudflared 43863 0.000 0.013 1.33 | 0.2 0.3 8.3 | resets=11
|
||||||
|
adventurelog 22 0.000 0.013 3.24 | 0.0 0.1 65.8 |
|
||||||
|
grafana 20 0.000 0.012 0.12 | 1.0 1.9 17.3 |
|
||||||
|
wanderer-db 19 0.007 0.011 0.01 | 0.0 3.7 3.7 |
|
||||||
|
dawarich-redis 20 0.009 0.011 0.01 | 0.4 3.7 3.8 |
|
||||||
|
claper-postgres 20 0.011 0.011 0.01 | 0.2 0.5 4.4 |
|
||||||
|
outline 19 0.008 0.008 0.01 | 0.0 7.4 7.7 |
|
||||||
|
nextcloud-redis 16 0.000 0.008 0.02 | 0.4 0.7 1.4 |
|
||||||
|
romm-redis 16 0.006 0.007 0.01 | 0.5 3.7 3.8 |
|
||||||
|
docmost-redis 23 0.005 0.005 0.07 | 0.4 0.7 3.9 |
|
||||||
|
dawarich-db 20 0.001 0.005 0.03 | 0.0 0.1 0.1 |
|
||||||
|
bookstack-db 20 0.001 0.005 0.04 | 0.0 4.7 4.8 |
|
||||||
|
grocy 19 0.000 0.005 0.01 | 0.0 0.0 0.0 |
|
||||||
|
bookstack 20 0.000 0.005 0.04 | 0.0 7.9 47.2 |
|
||||||
|
zipline 20 0.003 0.003 0.00 | 0.0 0.0 0.0 |
|
||||||
|
paperless-redis 22 0.002 0.003 0.00 | 0.4 0.4 0.4 |
|
||||||
|
outline-redis 19 0.003 0.003 0.00 | 0.4 0.4 0.5 |
|
||||||
|
kimai 19 0.000 0.003 0.02 | 0.0 2.4 24.3 |
|
||||||
|
zipline-postgres 20 0.003 0.003 0.00 | 0.0 0.0 0.0 |
|
||||||
|
qbittorrent 11759 0.001 0.003 0.01 | 0.1 0.2 97.5 | resets=25
|
||||||
|
felhom-controller 43863 0.000 0.003 1.73 | 0.1 15.5 122.0 | resets=92
|
||||||
|
bentopdf 18 0.000 0.002 0.00 | 0.0 0.0 0.0 |
|
||||||
|
emby 19 0.001 0.002 0.01 | 0.0 0.1 0.1 |
|
||||||
|
homepage 20 0.000 0.002 0.00 | 0.0 0.0 0.0 |
|
||||||
|
sonarr 11756 0.000 0.001 0.01 | 0.1 0.3 74.7 | resets=25
|
||||||
|
grimmory-db 24 0.000 0.001 0.35 | 0.0 5.0 100.2 |
|
||||||
|
grimmory 23 0.001 0.001 0.00 | 5.7 25.3 35.0 |
|
||||||
|
lubelogger 19 0.000 0.001 0.00 | 0.0 0.2 0.6 |
|
||||||
|
radarr 11756 0.000 0.001 0.17 | 0.0 0.6 79.9 | resets=25
|
||||||
|
glance 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||||
|
metube 20 0.000 0.001 0.00 | 0.0 1.3 26.1 |
|
||||||
|
actualbudget 18 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||||
|
privatebin 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||||
|
plex 18 0.000 0.001 0.00 | 0.7 0.8 0.8 |
|
||||||
|
papra 20 0.000 0.001 0.00 | 0.0 6.8 7.5 |
|
||||||
|
home-assistant 16 0.000 0.001 0.00 | 0.0 19.8 79.0 |
|
||||||
|
wanderer-search 19 0.000 0.001 0.00 | 0.1 0.1 0.1 |
|
||||||
|
karakeep 19 0.001 0.001 0.00 | 1.8 2.0 2.0 |
|
||||||
|
romm-db 16 0.001 0.001 0.00 | 0.0 4.6 4.7 |
|
||||||
|
radicale 19 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||||
|
gramps-web 20 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||||
|
tandoor 19 0.000 0.001 0.00 | 0.1 4.7 4.8 |
|
||||||
|
karakeep-chrome 19 0.001 0.001 0.00 | 0.0 0.0 0.0 |
|
||||||
|
outline-postgres 19 0.000 0.001 0.00 | 0.0 0.4 0.7 |
|
||||||
|
seerr 18 0.000 0.001 0.00 | 0.0 0.0 0.0 |
|
||||||
|
n8n 22 0.000 0.000 0.00 | 0.8 1.0 110.6 |
|
||||||
|
termix 20 0.000 0.000 0.00 | 0.0 0.4 2.3 |
|
||||||
|
tandoor-postgres 19 0.000 0.000 0.00 | 0.4 1.5 4.1 |
|
||||||
|
audiobookshelf 19 0.000 0.000 0.00 | 0.0 0.0 0.1 |
|
||||||
|
adventurelog-postgres 22 0.000 0.000 0.02 | 0.0 3.8 33.6 |
|
||||||
|
ghost 21 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||||
|
transmission 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
|
||||||
|
komga 19 0.000 0.000 0.00 | 0.3 6.8 10.6 |
|
||||||
|
calcom 18 0.000 0.000 0.00 | 0.0 3.4 22.3 |
|
||||||
|
vaultwarden 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||||
|
sparkyfitness 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||||
|
homebox 19 0.000 0.000 0.00 | 0.1 0.2 0.2 |
|
||||||
|
docmost-postgres 23 0.000 0.000 0.01 | 0.0 3.8 73.9 |
|
||||||
|
vikunja 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||||
|
adventurelog-frontend 22 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||||
|
rallly 20 0.000 0.000 0.00 | 0.0 5.6 110.3 |
|
||||||
|
calibre-web 17 0.000 0.000 0.00 | 0.0 21.3 106.4 |
|
||||||
|
wishlist 20 0.000 0.000 0.00 | 0.0 6.3 125.1 |
|
||||||
|
prowlarr 11738 0.000 0.000 0.00 | 0.0 0.1 77.0 | resets=26
|
||||||
|
opengist 43862 0.000 0.000 0.01 | 0.0 0.0 6.0 | resets=107
|
||||||
|
onlyoffice 22 0.000 0.000 0.00 | 0.0 130.5 147.5 |
|
||||||
|
sabnzbd 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||||
|
gitea 19 0.000 0.000 0.00 | 0.0 0.4 4.0 |
|
||||||
|
navidrome 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
|
||||||
|
code-server 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||||
|
uptime-kuma 19 0.000 0.000 0.00 | 0.7 0.7 0.7 |
|
||||||
|
nzbget 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||||
|
gokapi 19 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||||
|
recipe-importer 20 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||||
|
crafty-controller 19 0.000 0.000 0.00 | 0.1 8.3 8.5 |
|
||||||
|
mealie 20 0.000 0.000 0.00 | 0.1 0.1 0.1 |
|
||||||
|
rallly-postgres 20 0.000 0.000 0.00 | 0.0 3.7 3.8 |
|
||||||
|
traefik 43863 0.000 0.000 0.96 | 0.0 3.6 17.9 | resets=13
|
||||||
|
sparkyfitness-server 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
|
||||||
|
sparkyfitness-db 19 0.000 0.000 0.00 | 0.0 3.8 3.8 |
|
||||||
|
karakeep-meilisearch 19 0.000 0.000 0.00 | 0.1 0.1 0.1 |
|
||||||
|
filebrowser 43863 0.000 0.000 0.74 | 0.0 0.0 15.7 | resets=16
|
||||||
|
calcom-postgres 18 0.000 0.000 0.00 | 0.0 3.9 4.0 |
|
||||||
|
busiest UTC hours by bytes sent: 16h 16%, 05h 12%, 09h 6%, 02h 5%
|
||||||
|
|
||||||
|
=== tester-1: 2026-10-04 19:41 -> 2026-10-23 18:08 UTC (18.9 d), 294433 rows
|
||||||
|
container n tx p50 p95 max Mbit/s | cpu p50 p95 max %core | notes
|
||||||
|
qbittorrent 3857 0.007 0.133 42.40 | 0.1 2.6 92.3 | resets=20
|
||||||
|
transmission 3857 0.001 0.013 57.43 | 0.1 1.6 82.5 | resets=19
|
||||||
|
paperless-webserver 26710 0.005 0.013 0.09 | 0.1 0.7 508.1 | resets=63
|
||||||
|
cloudflared 26751 0.000 0.013 1.63 | 0.4 1.2 29.4 | resets=6
|
||||||
|
prowlarr 3725 0.000 0.012 0.26 | 0.1 0.3 101.8 | resets=8
|
||||||
|
sonarr 3855 0.001 0.004 1.21 | 0.1 4.5 131.6 | resets=20
|
||||||
|
radarr 3855 0.001 0.003 1.52 | 0.1 4.3 103.3 | resets=19
|
||||||
|
paperless-redis 26740 0.001 0.003 0.01 | 0.7 4.2 27.5 | resets=68
|
||||||
|
felhom-controller 26751 0.000 0.003 37.47 | 0.5 46.4 422.6 | resets=62
|
||||||
|
plex 3739 0.001 0.002 1.07 | 0.3 4.3 198.4 | resets=17
|
||||||
|
bookstack-db 26729 0.001 0.001 0.03 | 0.0 5.7 130.6 | resets=66
|
||||||
|
plex-k6 514 0.001 0.001 0.01 | 0.4 5.4 7.2 |
|
||||||
|
bookstack 26722 0.000 0.001 0.04 | 0.0 0.1 72.5 | resets=65
|
||||||
|
privatebin 26744 0.000 0.001 0.00 | 0.0 0.1 14.7 | resets=64
|
||||||
|
paperless-postgres 26740 0.000 0.001 0.02 | 0.0 4.4 65.8 | resets=68
|
||||||
|
traefik 26751 0.000 0.000 1.56 | 0.0 3.4 100.6 | resets=3
|
||||||
|
jellyfin 3648 0.000 0.000 2.83 | 0.0 0.2 181.5 | resets=19
|
||||||
|
xfp-echo 1 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||||
|
vaultwarden 1 0.000 0.000 0.00 | 0.0 0.0 0.0 |
|
||||||
|
filebrowser 26743 0.000 0.000 0.01 | 0.0 0.0 80.1 | resets=22
|
||||||
|
busiest UTC hours by bytes sent: 07h 13%, 18h 11%, 02h 9%, 17h 9%
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
== demo-hp
|
||||||
|
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
|
||||||
|
kimai 13 0 (-) 0 0 (-)
|
||||||
|
paperless-webserver 6 1 (09-23 04:56) 0 0 (-)
|
||||||
|
immich-machine-learning 4 4 (10-14 00:54) 0 0 (-)
|
||||||
|
romm 3 2 (10-23 15:36) 0 0 (-)
|
||||||
|
adventurelog 2 1 (09-30 11:29) 3 0 (-)
|
||||||
|
adventurelog-postgres 0 0 (-) 3 0 (-)
|
||||||
|
felhom-controller 3 1 (09-23 11:37) 2 2 (09-28 13:44)
|
||||||
|
immich-server 2 0 (-) 3 2 (10-14 00:53)
|
||||||
|
== demo-felhom
|
||||||
|
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
|
||||||
|
immich-server 2 0 (-) 8 1 (10-13 05:32)
|
||||||
|
immich-machine-learning 7 7 (10-13 16:39) 0 0 (-)
|
||||||
|
paperless-webserver 6 0 (-) 0 0 (-)
|
||||||
|
onlyoffice 4 0 (-) 0 0 (-)
|
||||||
|
== tester-1
|
||||||
|
container cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]
|
||||||
|
plex 10 5 (10-14 17:20) 1 0 (-)
|
||||||
|
qbittorrent 2 0 (-) 9 3 (10-14 18:08)
|
||||||
|
sonarr 9 0 (-) 2 0 (-)
|
||||||
|
transmission 1 0 (-) 6 2 (10-16 07:17)
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
|
||||||
|
== demo-hp: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
|
||||||
|
2026-09-23 max1= 3.14 best5= 0.91 busy_min= 0 peak_at=02:17Z
|
||||||
|
2026-09-24 max1= 3.04 best5= 0.73 busy_min= 0 peak_at=02:17Z
|
||||||
|
2026-09-25 max1= 2.13 best5= 0.67 busy_min= 0 peak_at=02:18Z
|
||||||
|
2026-09-26 max1= 1.77 best5= 0.63 busy_min= 0 peak_at=02:18Z
|
||||||
|
2026-09-27 max1= 2.04 best5= 0.75 busy_min= 0 peak_at=02:18Z
|
||||||
|
2026-09-28 max1= 17.79 best5= 6.89 busy_min= 3 peak_at=13:43Z
|
||||||
|
2026-09-29 max1= 2.45 best5= 0.65 busy_min= 0 peak_at=02:17Z
|
||||||
|
2026-09-30 max1= 2.62 best5= 0.76 busy_min= 0 peak_at=02:18Z
|
||||||
|
2026-10-01 max1= 1.02 best5= 0.54 busy_min= 0 peak_at=02:18Z
|
||||||
|
2026-10-02 max1= 2.02 best5= 0.62 busy_min= 0 peak_at=02:18Z
|
||||||
|
2026-10-03 max1= 2.01 best5= 0.60 busy_min= 0 peak_at=15:25Z
|
||||||
|
2026-10-04 max1= 2.08 best5= 0.42 busy_min= 0 peak_at=02:17Z
|
||||||
|
2026-10-05 max1= 2.36 best5= 0.48 busy_min= 0 peak_at=05:26Z
|
||||||
|
2026-10-06 max1= 1.58 best5= 0.32 busy_min= 0 peak_at=02:17Z
|
||||||
|
2026-10-07 max1= 1.05 best5= 0.30 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-08 max1= 1.43 best5= 0.29 busy_min= 0 peak_at=02:17Z
|
||||||
|
2026-10-09 max1= 1.47 best5= 0.29 busy_min= 0 peak_at=02:17Z
|
||||||
|
2026-10-10 max1= 0.90 best5= 0.27 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-11 max1= 1.34 best5= 0.30 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-12 max1= 4.96 best5= 0.99 busy_min= 0 peak_at=02:17Z
|
||||||
|
2026-10-13 max1= 1.31 best5= 0.28 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-14 max1= 6.31 best5= 1.27 busy_min= 1 peak_at=02:18Z
|
||||||
|
2026-10-15 max1= 2.25 best5= 0.45 busy_min= 0 peak_at=02:17Z
|
||||||
|
2026-10-16 max1= 2.78 best5= 0.66 busy_min= 0 peak_at=20:55Z
|
||||||
|
2026-10-17 max1= 2.26 best5= 0.45 busy_min= 0 peak_at=02:17Z
|
||||||
|
2026-10-18 max1= 2.08 best5= 0.47 busy_min= 0 peak_at=02:18Z
|
||||||
|
2026-10-19 max1= 16.95 best5= 5.08 busy_min= 2 peak_at=02:18Z
|
||||||
|
2026-10-20 max1= 1.20 best5= 0.48 busy_min= 0 peak_at=02:17Z
|
||||||
|
2026-10-21 max1= 2.27 best5= 0.46 busy_min= 0 peak_at=02:18Z
|
||||||
|
2026-10-22 max1= 2.13 best5= 0.52 busy_min= 0 peak_at=02:18Z
|
||||||
|
2026-10-23 max1= 1.37 best5= 0.48 busy_min= 0 peak_at=02:18Z
|
||||||
|
nights with a push (best5>2): 2; best5 min/median/max = 5.1/6.9/6.9 Mbit/s
|
||||||
|
|
||||||
|
== demo-felhom: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
|
||||||
|
2026-09-23 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=02:15Z
|
||||||
|
2026-09-24 max1= 0.06 best5= 0.01 busy_min= 0 peak_at=02:15Z
|
||||||
|
2026-09-25 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-09-26 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-09-27 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=00:19Z
|
||||||
|
2026-09-28 max1= 0.03 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-09-29 max1= 0.03 best5= 0.01 busy_min= 0 peak_at=02:15Z
|
||||||
|
2026-09-30 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-01 max1= 0.07 best5= 0.01 busy_min= 0 peak_at=19:54Z
|
||||||
|
2026-10-02 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-03 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-04 max1= 0.02 best5= 0.00 busy_min= 0 peak_at=06:58Z
|
||||||
|
2026-10-05 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=05:21Z
|
||||||
|
2026-10-06 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=17:34Z
|
||||||
|
2026-10-07 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=02:15Z
|
||||||
|
2026-10-08 max1= 0.09 best5= 0.02 busy_min= 0 peak_at=13:04Z
|
||||||
|
2026-10-09 max1= 0.10 best5= 0.02 busy_min= 0 peak_at=05:34Z
|
||||||
|
2026-10-10 max1= 0.01 best5= 0.00 busy_min= 0 peak_at=02:15Z
|
||||||
|
2026-10-11 max1= 0.04 best5= 0.01 busy_min= 0 peak_at=10:46Z
|
||||||
|
2026-10-12 max1= 0.08 best5= 0.02 busy_min= 0 peak_at=06:18Z
|
||||||
|
2026-10-13 max1= 0.05 best5= 0.01 busy_min= 0 peak_at=13:54Z
|
||||||
|
2026-10-14 max1= 0.07 best5= 0.02 busy_min= 0 peak_at=07:59Z
|
||||||
|
2026-10-15 max1= 0.14 best5= 0.03 busy_min= 0 peak_at=14:25Z
|
||||||
|
2026-10-16 max1= 0.84 best5= 0.17 busy_min= 0 peak_at=02:15Z
|
||||||
|
2026-10-17 max1= 0.49 best5= 0.10 busy_min= 0 peak_at=02:15Z
|
||||||
|
2026-10-18 max1= 0.33 best5= 0.07 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-19 max1= 1.73 best5= 0.35 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-20 max1= 0.23 best5= 0.07 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-21 max1= 0.36 best5= 0.07 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-22 max1= 0.52 best5= 0.11 busy_min= 0 peak_at=02:15Z
|
||||||
|
2026-10-23 max1= 0.48 best5= 0.10 busy_min= 0 peak_at=02:15Z
|
||||||
|
|
||||||
|
== tester-1: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak
|
||||||
|
2026-10-04 max1= 0.17 best5= 0.14 busy_min= 0 peak_at=19:55Z
|
||||||
|
2026-10-05 max1= 1.44 best5= 0.29 busy_min= 0 peak_at=03:10Z
|
||||||
|
2026-10-06 max1= 0.64 best5= 0.13 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-07 max1= 0.64 best5= 0.13 busy_min= 0 peak_at=02:15Z
|
||||||
|
2026-10-08 max1= 0.42 best5= 0.08 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-09 max1= 0.48 best5= 0.10 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-10 max1= 0.84 best5= 0.17 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-11 max1= 0.46 best5= 0.09 busy_min= 0 peak_at=02:15Z
|
||||||
|
2026-10-12 max1= 1.71 best5= 0.34 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-13 max1= 0.99 best5= 0.20 busy_min= 0 peak_at=02:15Z
|
||||||
|
2026-10-14 max1= 0.45 best5= 0.09 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-15 max1= 37.47 best5= 7.58 busy_min= 1 peak_at=02:17Z
|
||||||
|
2026-10-16 max1= 3.85 best5= 0.77 busy_min= 0 peak_at=02:17Z
|
||||||
|
2026-10-17 max1= 5.11 best5= 1.21 busy_min= 1 peak_at=02:16Z
|
||||||
|
2026-10-18 max1= 0.50 best5= 0.10 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-19 max1= 5.14 best5= 1.03 busy_min= 1 peak_at=02:16Z
|
||||||
|
2026-10-20 max1= 0.52 best5= 0.10 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-21 max1= 0.61 best5= 0.12 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-22 max1= 0.53 best5= 0.11 busy_min= 0 peak_at=02:16Z
|
||||||
|
2026-10-23 max1= 0.80 best5= 0.18 busy_min= 0 peak_at=02:16Z
|
||||||
|
nights with a push (best5>2): 1; best5 min/median/max = 7.6/7.6/7.6 Mbit/s
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
login 204 cookie
|
||||||
|
{"t": 0.2, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65136369198, "engine_tx": 65136367684, "cli": "65.5MB / 65.1GB"}
|
||||||
|
{"t": 6.5, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65152041285, "engine_tx": 65152039771, "cli": "65.9MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 19.9, "rate_engine_Mbit": 19.9}
|
||||||
|
{"t": 12.7, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65167646902, "engine_tx": 65167645388, "cli": "66.2MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 20.14, "rate_engine_Mbit": 20.14}
|
||||||
|
{"t": 19.0, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65183320503, "engine_tx": 65183318989, "cli": "66.6MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 19.9, "rate_engine_Mbit": 19.9}
|
||||||
|
{"t": 25.2, "qb_up_speed": 0, "qb_up_data": 197886667, "qb_tor_uploaded": 1660504484, "proc_tx": 65198925974, "engine_tx": 65198924460, "cli": "66.9MB / 65.2GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 20.14, "rate_engine_Mbit": 20.14}
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
LISTEN 0 5 0.0.0.0:5001 0.0.0.0:*
|
||||||
|
login 204 cookie
|
||||||
|
{"t": 0.0, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 1559315619, "engine_tx": 1558369279, "cli": "2.94MB / 2.85GB"}
|
||||||
|
{"t": 6.0, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 9161893432, "engine_tx": 9161333668, "cli": "9.34MB / 10.5GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10136.77, "rate_engine_Mbit": 10137.29}
|
||||||
|
{"t": 12.1, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 17020963755, "engine_tx": 17019975365, "cli": "16.7MB / 18.7GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10306.98, "rate_engine_Mbit": 10306.42}
|
||||||
|
{"t": 18.1, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 25140960556, "engine_tx": 25139940620, "cli": "23.6MB / 26.5GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10826.66, "rate_engine_Mbit": 10826.62}
|
||||||
|
{"t": 24.2, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 33049634829, "engine_tx": 33048966257, "cli": "30.6MB / 34.4GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10372.03, "rate_engine_Mbit": 10372.49}
|
||||||
|
{"t": 30.2, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 41055773948, "engine_tx": 41055105508, "cli": "37.6MB / 42.3GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10674.85, "rate_engine_Mbit": 10674.85}
|
||||||
|
{"t": 36.3, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 48735514801, "engine_tx": 48734576785, "cli": "44.3MB / 50GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10071.79, "rate_engine_Mbit": 10071.44}
|
||||||
|
{"t": 42.3, "qb_up_speed": 0, "qb_up_data": 0, "qb_tor_uploaded": 1462617817, "proc_tx": 56370706217, "engine_tx": 56370119481, "cli": "52.4MB / 58.8GB", "rate_qb_data_Mbit": 0.0, "rate_proc_Mbit": 10180.26, "rate_engine_Mbit": 10180.72}
|
||||||
@@ -0,0 +1,9 @@
|
|||||||
|
login 204 cookie
|
||||||
|
setUploadLimit (200, '')
|
||||||
|
uploadLimit now 2499584 prefs.up_limit 2499584
|
||||||
|
login 204 cookie
|
||||||
|
{"t": 0.0, "qb_up_speed": 16386315, "qb_up_data": 1346633728, "qb_tor_uploaded": 1346633728, "proc_tx": 1422747831, "engine_tx": 1422747831, "cli": "27.2MB / 1.42GB"}
|
||||||
|
{"t": 6.0, "qb_up_speed": 0, "qb_up_data": 1374387364, "qb_tor_uploaded": 1374387364, "proc_tx": 1434635707, "engine_tx": 1434635707, "cli": "27.6MB / 1.44GB", "rate_qb_data_Mbit": 37.0, "rate_proc_Mbit": 15.85, "rate_engine_Mbit": 15.85}
|
||||||
|
{"t": 12.1, "qb_up_speed": 2872275, "qb_up_data": 1392093322, "qb_tor_uploaded": 1391692420, "proc_tx": 1457229455, "engine_tx": 1457229455, "cli": "27.9MB / 1.46GB", "rate_qb_data_Mbit": 23.22, "rate_proc_Mbit": 29.63, "rate_engine_Mbit": 29.63}
|
||||||
|
{"t": 18.1, "qb_up_speed": 1975559, "qb_up_data": 1406540742, "qb_tor_uploaded": 1406540742, "proc_tx": 1471460371, "engine_tx": 1471460371, "cli": "28.2MB / 1.47GB", "rate_qb_data_Mbit": 19.26, "rate_proc_Mbit": 18.97, "rate_engine_Mbit": 18.97}
|
||||||
|
{"t": 24.2, "qb_up_speed": 3254241, "qb_up_data": 1423113257, "qb_tor_uploaded": 1420865709, "proc_tx": 1488294067, "engine_tx": 1488294067, "cli": "28.5MB / 1.49GB", "rate_qb_data_Mbit": 21.73, "rate_proc_Mbit": 22.08, "rate_engine_Mbit": 22.08}
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
login 204 cookie
|
||||||
|
{"t": 0.0, "qb_up_speed": 16266049, "qb_up_data": 117833728, "qb_tor_uploaded": 117701484, "proc_tx": 123161204, "engine_tx": 123161204, "cli": "6.74MB / 140MB"}
|
||||||
|
{"t": 6.0, "qb_up_speed": 16384087, "qb_up_data": 216137728, "qb_tor_uploaded": 216137728, "proc_tx": 225760308, "engine_tx": 225760308, "cli": "8.38MB / 243MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 136.8, "rate_engine_Mbit": 136.8}
|
||||||
|
{"t": 12.1, "qb_up_speed": 16387266, "qb_up_data": 314441728, "qb_tor_uploaded": 314441728, "proc_tx": 328359626, "engine_tx": 328359626, "cli": "10.1MB / 346MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 134.56, "rate_engine_Mbit": 134.56}
|
||||||
|
{"t": 18.1, "qb_up_speed": 16391332, "qb_up_data": 412745728, "qb_tor_uploaded": 412745728, "proc_tx": 431576783, "engine_tx": 431540471, "cli": "11.8MB / 450MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 137.62, "rate_engine_Mbit": 137.57}
|
||||||
|
{"t": 24.2, "qb_up_speed": 16377743, "qb_up_data": 511049728, "qb_tor_uploaded": 511049728, "proc_tx": 536615588, "engine_tx": 536559607, "cli": "13.4MB / 556MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 137.76, "rate_engine_Mbit": 137.73}
|
||||||
|
{"t": 30.2, "qb_up_speed": 16387474, "qb_up_data": 609353728, "qb_tor_uploaded": 609353728, "proc_tx": 641957804, "engine_tx": 641903336, "cli": "14.9MB / 660MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 140.46, "rate_engine_Mbit": 140.46}
|
||||||
|
{"t": 36.3, "qb_up_speed": 16383934, "qb_up_data": 707657728, "qb_tor_uploaded": 707657728, "proc_tx": 746552576, "engine_tx": 746552576, "cli": "16.8MB / 764MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 137.17, "rate_engine_Mbit": 137.24}
|
||||||
|
{"t": 42.3, "qb_up_speed": 16392852, "qb_up_data": 805961728, "qb_tor_uploaded": 805961728, "proc_tx": 849903723, "engine_tx": 849903723, "cli": "18.3MB / 867MB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 137.8, "rate_engine_Mbit": 137.8}
|
||||||
|
{"t": 48.4, "qb_up_speed": 16382132, "qb_up_data": 904265728, "qb_tor_uploaded": 904265728, "proc_tx": 952502020, "engine_tx": 952502020, "cli": "19.9MB / 970MB", "rate_qb_data_Mbit": 128.92, "rate_proc_Mbit": 134.56, "rate_engine_Mbit": 134.56}
|
||||||
|
{"t": 54.4, "qb_up_speed": 16376106, "qb_up_data": 1002569728, "qb_tor_uploaded": 1002569728, "proc_tx": 1055099637, "engine_tx": 1055099637, "cli": "21.5MB / 1.07GB", "rate_qb_data_Mbit": 131.07, "rate_proc_Mbit": 136.8, "rate_engine_Mbit": 136.8}
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
{'arguments': {}, 'result': 'success'}
|
||||||
|
{'speed-limit-up': 2500, 'speed-limit-up-enabled': True}
|
||||||
|
{"t": 0.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157242728}
|
||||||
|
{"t": 5.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157244479, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 10.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157246230, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 15.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157248133, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157249884, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 25.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157251635, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
{"t": 0.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157256302}
|
||||||
|
{"t": 10.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157258205, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157261594, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 30.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157263345, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 40.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157265096, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 50.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157268485, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 60.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157271188, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 70.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157272939, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 80.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157276328, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 90.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157278588, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
login 204 cookie
|
||||||
|
/api/v2/torrents/stop (200, '')
|
||||||
|
{'arguments': {}, 'result': 'success'}
|
||||||
|
{"t": 0.0, "tr_upspeed": 40648163, "tr_cur_up": 1528429958, "tr_tor_up": 1528429958, "tx": 1534546958}
|
||||||
|
{"t": 5.0, "tr_upspeed": 65588000, "tr_cur_up": 1832873057, "tr_tor_up": 1832873057, "tx": 1839315356, "rate_tr_Mbit": 487.11, "rate_container_Mbit": 487.63}
|
||||||
|
{"t": 10.0, "tr_upspeed": 23587084, "tr_cur_up": 2076811226, "tr_tor_up": 2076811226, "tx": 2083512878, "rate_tr_Mbit": 390.3, "rate_container_Mbit": 390.72}
|
||||||
|
{"t": 15.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157213873, "rate_tr_Mbit": 115.8, "rate_container_Mbit": 117.92}
|
||||||
|
{"t": 20.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157215624, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 25.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157218947, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.01}
|
||||||
|
{"t": 30.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157220698, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 35.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157222449, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 40.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157224200, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 45.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157225951, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 50.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157227702, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.0}
|
||||||
|
{"t": 55.0, "tr_upspeed": 0, "tr_cur_up": 2149187584, "tr_tor_up": 2149187584, "tx": 2157231243, "rate_tr_Mbit": 0.0, "rate_container_Mbit": 0.01}
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
# usage: gen.sh <dir> ; writes <dir>/uwtest.bin, 2 GiB, identical on every machine (AES-CTR keystream of a fixed key)
|
||||||
|
mkdir -p "$1"; openssl enc -aes-128-ctr -nosalt -K 00112233445566778899aabbccddeeff -iv 0 -in /dev/zero 2>/dev/null | head -c 2147483648 > "$1/uwtest.bin"; sha256sum "$1/uwtest.bin" | cut -c1-16
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
import sqlite3,sys,collections,datetime,statistics
|
||||||
|
def pct(a,p):
|
||||||
|
if not a: return 0
|
||||||
|
a=sorted(a); k=(len(a)-1)*p/100; f=int(k); c=min(f+1,len(a)-1); return a[f]+(a[c]-a[f])*(k-f)
|
||||||
|
for box in sys.argv[1:]:
|
||||||
|
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
|
||||||
|
lo,hi,n=db.execute('select min(ts),max(ts),count(*) from container_metrics').fetchone()
|
||||||
|
print(f'\n=== {box.rsplit("/",1)[1]}: {datetime.datetime.utcfromtimestamp(lo):%Y-%m-%d %H:%M} -> {datetime.datetime.utcfromtimestamp(hi):%Y-%m-%d %H:%M} UTC ({(hi-lo)/86400:.1f} d), {n} rows')
|
||||||
|
rows=collections.defaultdict(list)
|
||||||
|
for ts,name,cpu,tx in db.execute('select ts,container_name,cpu_percent,net_tx_bytes from container_metrics order by ts'):
|
||||||
|
rows[name].append((ts,cpu,tx))
|
||||||
|
hour=collections.defaultdict(float)
|
||||||
|
out=[]
|
||||||
|
for name,r in rows.items():
|
||||||
|
rates=[];resets=0;zero=all((x[2] or 0)==0 for x in r)
|
||||||
|
for (t0,_,a),(t1,_,b) in zip(r,r[1:]):
|
||||||
|
if a is None or b is None or t1-t0<=0 or t1-t0>180: continue
|
||||||
|
d=b-a
|
||||||
|
if d<0: resets+=1; continue
|
||||||
|
mbps=d*8/(t1-t0)/1e6; rates.append(mbps)
|
||||||
|
hour[datetime.datetime.utcfromtimestamp(t1).hour]+=d
|
||||||
|
cpus=[x[1] for x in r]
|
||||||
|
out.append((pct(rates,95),name,len(r),pct(rates,50),pct(rates,95),max(rates or [0]),pct(cpus,50),pct(cpus,95),max(cpus),zero,resets))
|
||||||
|
print(f'{"container":32} {"n":>6} {"tx p50":>8} {"p95":>8} {"max":>8} Mbit/s | {"cpu p50":>7} {"p95":>7} {"max":>7} %core | notes')
|
||||||
|
for o in sorted(out,reverse=True):
|
||||||
|
_,name,nn,a,b,c,d,e,f,zero,res=o
|
||||||
|
note=('tx counter always 0 (host network?)' if zero else '')+(f' resets={res}' if res else '')
|
||||||
|
print(f'{name[:32]:32} {nn:6} {a:8.3f} {b:8.3f} {c:8.2f} | {d:7.1f} {e:7.1f} {f:7.1f} | {note}')
|
||||||
|
tot=sum(hour.values()) or 1
|
||||||
|
print('busiest UTC hours by bytes sent:',', '.join(f'{h:02d}h {100*v/tot:.0f}%' for h,v in sorted(hour.items(),key=lambda x:-x[1])[:4]))
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
import sqlite3,sys,collections,datetime
|
||||||
|
for box in sys.argv[1:]:
|
||||||
|
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
|
||||||
|
r=db.execute("select ts,net_tx_bytes from container_metrics where container_name='felhom-controller' order by ts").fetchall()
|
||||||
|
per=collections.defaultdict(list)
|
||||||
|
for (t0,a),(t1,b) in zip(r,r[1:]):
|
||||||
|
if a is None or b is None or b<a or not 0<t1-t0<=180: continue
|
||||||
|
d=datetime.datetime.utcfromtimestamp(t1)
|
||||||
|
per[d.date()].append((t1,(b-a)*8/(t1-t0)/1e6))
|
||||||
|
print(f'\n== {box.rsplit("/",1)[1]}: per UTC day — max 1-min, best 5-min mean, minutes above 5 Mbit/s, hour of the peak')
|
||||||
|
peaks=[]
|
||||||
|
for day in sorted(per):
|
||||||
|
v=per[day]; rates=[x[1] for x in v]
|
||||||
|
best5=max((sum(rates[i:i+5])/5 for i in range(max(1,len(rates)-4))),default=0)
|
||||||
|
m=max(v,key=lambda x:x[1])
|
||||||
|
busy=sum(1 for x in rates if x>5)
|
||||||
|
peaks.append(best5)
|
||||||
|
print(f'{day} max1={m[1]:6.2f} best5={best5:6.2f} busy_min={busy:4d} peak_at={datetime.datetime.utcfromtimestamp(m[0]):%H:%M}Z')
|
||||||
|
big=[p for p in peaks if p>2]
|
||||||
|
if big:
|
||||||
|
big.sort(); print(f'nights with a push (best5>2): {len(big)}; best5 min/median/max = {big[0]:.1f}/{big[len(big)//2]:.1f}/{big[-1]:.1f} Mbit/s')
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Waits until 02:08Z on 2026-10-24, then reads the controller container's eth0 tx bytes once a second for 22 minutes on
|
||||||
|
# each of the three test boxes (read only: /proc/<pid>/net/dev inside guest 9201). Output: one file per box.
|
||||||
|
OUT=$1
|
||||||
|
target=$(date -u -d '2026-10-24T02:08:00Z' +%s); now=$(date -u +%s); [ $target -gt $now ] && sleep $((target-now))
|
||||||
|
INNER='P=$(docker inspect -f {{.State.Pid}} felhom-controller); end=$(( $(date +%s) + 1320 )); while [ $(date +%s) -lt $end ]; do echo "$(date +%s.%N | cut -c1-14) $(awk "/eth0:/{print \$10}" /proc/$P/net/dev)"; sleep 1; done'
|
||||||
|
for pair in "demo-hp:demo-hp" "felhom-pve:demo-felhom" "root@192.168.0.154:tester-1"; do h=${pair%%:*}; n=${pair##*:}
|
||||||
|
ssh -o BatchMode=yes -o ServerAliveInterval=30 $h "pct exec 9201 -- bash -c '$INNER'" > "$OUT/night-push-1s-$n.txt" 2>&1 &
|
||||||
|
done
|
||||||
|
wait; echo "done $(date -u +%FT%TZ)"
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
# Runs INSIDE guest 9202. Asks qBittorrent's own Web API and reads the container's exact byte counters.
|
||||||
|
# The admin password is read from the container env at use (as the controller does) and never printed.
|
||||||
|
import json, subprocess, sys, time, urllib.request, urllib.parse, http.client, socket, re, uuid
|
||||||
|
def sh(*a): return subprocess.run(a, capture_output=True, text=True).stdout.strip()
|
||||||
|
C = sys.argv[1] if len(sys.argv) > 1 else 'qbittorrent'
|
||||||
|
IP = sh('docker', 'inspect', '-f', '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}', C).split()[0]
|
||||||
|
PID = sh('docker', 'inspect', '-f', '{{.State.Pid}}', C)
|
||||||
|
CID = sh('docker', 'inspect', '-f', '{{.Id}}', C)
|
||||||
|
BASE = f'http://{IP}:8080'
|
||||||
|
PW = sh('docker', 'exec', C, 'printenv', 'QBT_ADMIN_PASSWORD')
|
||||||
|
def req(method, path, data=None, hdr=None, raw=False):
|
||||||
|
h = {'Referer': BASE}
|
||||||
|
h.update(hdr or {})
|
||||||
|
if COOKIE: h['Cookie'] = COOKIE
|
||||||
|
r = urllib.request.Request(BASE + path, data=data, headers=h, method=method)
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(r, timeout=20) as resp:
|
||||||
|
b = resp.read(); return resp.status, (b if raw else (json.loads(b) if b and b[:1] in b'[{' else b.decode()))
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
return e.code, e.read().decode()[:200]
|
||||||
|
COOKIE = None
|
||||||
|
code, _ = None, None
|
||||||
|
r = urllib.request.Request(BASE + '/api/v2/auth/login', data=urllib.parse.urlencode({'username': 'admin', 'password': PW}).encode(),
|
||||||
|
headers={'Referer': BASE, 'Content-Type': 'application/x-www-form-urlencoded'}, method='POST')
|
||||||
|
with urllib.request.urlopen(r, timeout=20) as resp:
|
||||||
|
m = re.search(r'((?:QBT_)?SID[A-Za-z0-9_]*)=([^;]+)', '; '.join(resp.headers.get_all('Set-Cookie') or []))
|
||||||
|
print('login', resp.status, 'cookie' if m else 'NO COOKIE')
|
||||||
|
COOKIE = f'{m.group(1)}={m.group(2)}'
|
||||||
|
def netdev():
|
||||||
|
for line in open(f'/proc/{PID}/net/dev'):
|
||||||
|
if line.strip().startswith('eth0:'):
|
||||||
|
f = line.split(':', 1)[1].split(); return int(f[8])
|
||||||
|
def engine_stats():
|
||||||
|
s = socket.socket(socket.AF_UNIX); s.connect('/var/run/docker.sock')
|
||||||
|
s.sendall(f'GET /containers/{CID}/stats?stream=false&one-shot=true HTTP/1.0\r\n\r\n'.encode())
|
||||||
|
b = b''
|
||||||
|
while True:
|
||||||
|
x = s.recv(65536)
|
||||||
|
if not x: break
|
||||||
|
b += x
|
||||||
|
j = json.loads(b.split(b'\r\n\r\n', 1)[1])
|
||||||
|
return sum(n['tx_bytes'] for n in j.get('networks', {}).values()), j['cpu_stats']['cpu_usage']['total_usage']
|
||||||
|
def cli_stats():
|
||||||
|
return sh('docker', 'stats', '--no-stream', '--format', '{{.NetIO}}', C)
|
||||||
|
mode = sys.argv[2] if len(sys.argv) > 2 else 'info'
|
||||||
|
if mode == 'info':
|
||||||
|
print('version', req('GET', '/api/v2/app/version')[1], 'webapi', req('GET', '/api/v2/app/webapiVersion')[1])
|
||||||
|
print('transfer/info', req('GET', '/api/v2/transfer/info')[1])
|
||||||
|
p = req('GET', '/api/v2/app/preferences')[1]
|
||||||
|
print('prefs', {k: p.get(k) for k in ('up_limit', 'dl_limit', 'alt_up_limit', 'alt_dl_limit', 'scheduler_enabled', 'limit_tcp_overhead', 'limit_lan_peers', 'limit_utp_rate', 'max_uploads', 'enable_upload_suggestions')})
|
||||||
|
print('uploadLimit', req('GET', '/api/v2/transfer/uploadLimit')[1], 'speedLimitsMode', req('GET', '/api/v2/transfer/speedLimitsMode')[1])
|
||||||
|
elif mode == 'add':
|
||||||
|
bnd = uuid.uuid4().hex
|
||||||
|
tor = open('/root/uwtest.torrent', 'rb').read()
|
||||||
|
parts = []
|
||||||
|
for k, v in (('savepath', '/mnt/felhom-drives/scratch_hdd/userdata/media/torrents'), ('skip_checking', 'true'), ('category', '')):
|
||||||
|
parts.append(f'--{bnd}\r\nContent-Disposition: form-data; name="{k}"\r\n\r\n{v}\r\n'.encode())
|
||||||
|
parts.append(f'--{bnd}\r\nContent-Disposition: form-data; name="torrents"; filename="uwtest.torrent"\r\nContent-Type: application/x-bittorrent\r\n\r\n'.encode() + tor + b'\r\n')
|
||||||
|
parts.append(f'--{bnd}--\r\n'.encode())
|
||||||
|
print('add', req('POST', '/api/v2/torrents/add', b''.join(parts), {'Content-Type': f'multipart/form-data; boundary={bnd}'}))
|
||||||
|
time.sleep(3)
|
||||||
|
print('torrents', [(t['name'], t['state'], t['progress']) for t in req('GET', '/api/v2/torrents/info')[1]])
|
||||||
|
elif mode == 'setlimit':
|
||||||
|
print('setUploadLimit', req('POST', '/api/v2/transfer/setUploadLimit', urllib.parse.urlencode({'limit': sys.argv[3]}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
|
||||||
|
print('uploadLimit now', req('GET', '/api/v2/transfer/uploadLimit')[1], 'prefs.up_limit', req('GET', '/api/v2/app/preferences')[1].get('up_limit'))
|
||||||
|
elif mode == 'setpref':
|
||||||
|
print('setPreferences', req('POST', '/api/v2/app/setPreferences', urllib.parse.urlencode({'json': json.dumps({'up_limit': int(sys.argv[3])})}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
|
||||||
|
print('uploadLimit now', req('GET', '/api/v2/transfer/uploadLimit')[1], 'prefs.up_limit', req('GET', '/api/v2/app/preferences')[1].get('up_limit'))
|
||||||
|
elif mode == 'peer':
|
||||||
|
print('addPeers', req('POST', '/api/v2/torrents/addPeers', urllib.parse.urlencode({'hashes': 'all', 'peers': sys.argv[3]}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
|
||||||
|
print('reannounce', req('POST', '/api/v2/torrents/reannounce', urllib.parse.urlencode({'hashes': 'all'}).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
|
||||||
|
elif mode == 'post':
|
||||||
|
print(sys.argv[3], req('POST', sys.argv[3], urllib.parse.urlencode(json.loads(sys.argv[4])).encode(), {'Content-Type': 'application/x-www-form-urlencoded'}))
|
||||||
|
elif mode == 'sample':
|
||||||
|
dur, step = int(sys.argv[3]), int(sys.argv[4])
|
||||||
|
t0 = time.time(); prev = None
|
||||||
|
while time.time() - t0 <= dur:
|
||||||
|
ti = req('GET', '/api/v2/transfer/info')[1]
|
||||||
|
tor = req('GET', '/api/v2/torrents/info')[1]
|
||||||
|
etx, ecpu = engine_stats()
|
||||||
|
row = {'t': round(time.time() - t0, 1), 'qb_up_speed': ti['up_info_speed'], 'qb_up_data': ti['up_info_data'],
|
||||||
|
'qb_tor_uploaded': sum(x.get('uploaded', 0) for x in tor), 'proc_tx': netdev(), 'engine_tx': etx, 'cli': cli_stats()}
|
||||||
|
if prev:
|
||||||
|
dt = row['t'] - prev['t']
|
||||||
|
row['rate_qb_data_Mbit'] = round((row['qb_up_data'] - prev['qb_up_data']) * 8 / dt / 1e6, 2)
|
||||||
|
row['rate_proc_Mbit'] = round((row['proc_tx'] - prev['proc_tx']) * 8 / dt / 1e6, 2)
|
||||||
|
row['rate_engine_Mbit'] = round((row['engine_tx'] - prev['engine_tx']) * 8 / dt / 1e6, 2)
|
||||||
|
print(json.dumps(row), flush=True); prev = row
|
||||||
|
time.sleep(step)
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
import sqlite3,sys,collections
|
||||||
|
def runs(r, pred):
|
||||||
|
best=cur=0; bt=None; prev=None
|
||||||
|
for t,v in r:
|
||||||
|
if pred(v) and prev is not None and t-prev<=180: cur+=1
|
||||||
|
elif pred(v): cur=1
|
||||||
|
else: cur=0
|
||||||
|
if cur>best: best=cur; bt=t
|
||||||
|
prev=t
|
||||||
|
return best,bt
|
||||||
|
import datetime
|
||||||
|
for box in sys.argv[1:]:
|
||||||
|
db=sqlite3.connect(f'file:{box}/metrics.db?mode=ro',uri=True)
|
||||||
|
nc=None
|
||||||
|
rows=collections.defaultdict(list)
|
||||||
|
for ts,name,cpu,tx in db.execute('select ts,container_name,cpu_percent,net_tx_bytes from container_metrics order by ts'):
|
||||||
|
rows[name].append((ts,cpu,tx))
|
||||||
|
print('==',box.rsplit('/',1)[1])
|
||||||
|
out=[]
|
||||||
|
for name,r in rows.items():
|
||||||
|
cpu=[(t,c) for t,c,_ in r]
|
||||||
|
tx=[]
|
||||||
|
for (t0,_,a),(t1,_,b) in zip(r,r[1:]):
|
||||||
|
if a is None or b is None or b<a or not 0<t1-t0<=180: continue
|
||||||
|
tx.append((t1,(b-a)*8/(t1-t0)/1e6))
|
||||||
|
c150,ct=runs(cpu,lambda v:v>=150); c80,_=runs(cpu,lambda v:v>=80)
|
||||||
|
u5,ut=runs(tx,lambda v:v>=5); u1,_=runs(tx,lambda v:v>=1)
|
||||||
|
if max(c80,u1)>=3: out.append((name,c80,c150,ct,u1,u5,ut))
|
||||||
|
print(f'{"container":28} cpu>=80% cpu>=150% (at) tx>=1Mbit tx>=5Mbit (at) [longest run, minutes]')
|
||||||
|
for name,c80,c150,ct,u1,u5,ut in sorted(out,key=lambda x:-max(x[1],x[4])):
|
||||||
|
f=lambda t: datetime.datetime.utcfromtimestamp(t).strftime('%m-%d %H:%M') if t else '-'
|
||||||
|
print(f'{name[:28]:28} {c80:8} {c150:9} ({f(ct)}) {u1:9} {u5:9} ({f(ut)})')
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# Minimal BitTorrent HTTP tracker for a closed scratch test (no internet). Compact peer lists.
|
||||||
|
import http.server, urllib.parse, socket, struct, time
|
||||||
|
peers = {} # info_hash -> {(ip,port): last_seen}
|
||||||
|
def benc(o):
|
||||||
|
if isinstance(o, int): return b"i%de" % o
|
||||||
|
if isinstance(o, bytes): return b"%d:%s" % (len(o), o)
|
||||||
|
if isinstance(o, str): return benc(o.encode())
|
||||||
|
if isinstance(o, dict): return b"d" + b"".join(benc(k) + benc(v) for k, v in sorted(o.items())) + b"e"
|
||||||
|
class H(http.server.BaseHTTPRequestHandler):
|
||||||
|
def do_GET(self):
|
||||||
|
q = urllib.parse.parse_qs(urllib.parse.urlsplit(self.path).query.encode('latin-1').decode('latin-1'), encoding='latin-1')
|
||||||
|
ih = q.get('info_hash', [''])[0]; port = int(q.get('port', ['0'])[0]); ip = self.client_address[0]
|
||||||
|
d = peers.setdefault(ih, {}); now = time.time()
|
||||||
|
if q.get('event', [''])[0] == 'stopped': d.pop((ip, port), None)
|
||||||
|
else: d[(ip, port)] = now
|
||||||
|
lst = b"".join(socket.inet_aton(i) + struct.pack(">H", p) for (i, p), t in d.items() if now - t < 600 and (i, p) != (ip, port))
|
||||||
|
body = benc({"interval": 30, "peers": lst})
|
||||||
|
print(time.strftime('%H:%M:%S'), 'announce', ip, port, q.get('event', [''])[0], 'peers', len(lst) // 6, flush=True)
|
||||||
|
self.send_response(200); self.send_header('Content-Length', str(len(body))); self.end_headers(); self.wfile.write(body)
|
||||||
|
def log_message(self, *a): pass
|
||||||
|
http.server.ThreadingHTTPServer(('0.0.0.0', 6969), H).serve_forever()
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
# Runs INSIDE guest 9202. Transmission's own RPC numbers vs the container's exact tx bytes. Password read at use, never printed.
|
||||||
|
import json, subprocess, sys, time, urllib.request, base64
|
||||||
|
def sh(*a): return subprocess.run(a, capture_output=True, text=True).stdout.strip()
|
||||||
|
C = 'transmission'
|
||||||
|
IP = sh('docker', 'inspect', '-f', '{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}', C).split()[0]
|
||||||
|
PID = sh('docker', 'inspect', '-f', '{{.State.Pid}}', C)
|
||||||
|
PW = sh('docker', 'exec', C, 'printenv', 'PASS')
|
||||||
|
AUTH = 'Basic ' + base64.b64encode(('admin:' + PW).encode()).decode()
|
||||||
|
SID = ''
|
||||||
|
def rpc(method, args=None):
|
||||||
|
global SID
|
||||||
|
for _ in range(2):
|
||||||
|
r = urllib.request.Request(f'http://{IP}:9091/transmission/rpc', data=json.dumps({'method': method, 'arguments': args or {}}).encode(),
|
||||||
|
headers={'Authorization': AUTH, 'Host': 'transmission:9091', 'X-Transmission-Session-Id': SID, 'Content-Type': 'application/json'}, method='POST')
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(r, timeout=20) as resp: return json.loads(resp.read())
|
||||||
|
except urllib.error.HTTPError as e:
|
||||||
|
if e.code == 409: SID = e.headers.get('X-Transmission-Session-Id'); continue
|
||||||
|
return {'http': e.code}
|
||||||
|
def tx():
|
||||||
|
for line in open(f'/proc/{PID}/net/dev'):
|
||||||
|
if line.strip().startswith('eth0:'): return int(line.split(':', 1)[1].split()[8])
|
||||||
|
mode = sys.argv[1]
|
||||||
|
if mode == 'info':
|
||||||
|
g = rpc('session-get')['arguments']
|
||||||
|
print('version', g.get('version'), {k: g.get(k) for k in ('speed-limit-up', 'speed-limit-up-enabled', 'alt-speed-up', 'alt-speed-enabled', 'alt-speed-time-enabled', 'units')})
|
||||||
|
print('session-stats', rpc('session-stats')['arguments'])
|
||||||
|
elif mode == 'add':
|
||||||
|
meta = base64.b64encode(open('/root/uwtest.torrent', 'rb').read()).decode()
|
||||||
|
print(rpc('torrent-add', {'metainfo': meta, 'download-dir': '/mnt/felhom-drives/scratch_hdd/userdata/media/transmission', 'paused': False}))
|
||||||
|
time.sleep(5); rpc('torrent-verify', {}); time.sleep(20)
|
||||||
|
print(rpc('torrent-get', {'fields': ['name', 'status', 'percentDone', 'error', 'errorString']}))
|
||||||
|
elif mode == 'limit':
|
||||||
|
print(rpc('session-set', {'speed-limit-up': int(sys.argv[2]), 'speed-limit-up-enabled': True}))
|
||||||
|
g = rpc('session-get')['arguments']; print({k: g.get(k) for k in ('speed-limit-up', 'speed-limit-up-enabled')})
|
||||||
|
elif mode == 'reannounce':
|
||||||
|
print(rpc('torrent-reannounce', {}))
|
||||||
|
elif mode == 'sample':
|
||||||
|
dur, step = int(sys.argv[2]), int(sys.argv[3]); t0 = time.time(); prev = None
|
||||||
|
while time.time() - t0 <= dur:
|
||||||
|
st = rpc('session-stats')['arguments']; tg = rpc('torrent-get', {'fields': ['uploadedEver', 'rateUpload']})['arguments']['torrents']
|
||||||
|
row = {'t': round(time.time() - t0, 1), 'tr_upspeed': st['uploadSpeed'], 'tr_cur_up': st['current-stats']['uploadedBytes'], 'tr_tor_up': sum(x['uploadedEver'] for x in tg), 'tx': tx()}
|
||||||
|
if prev:
|
||||||
|
dt = row['t'] - prev['t']
|
||||||
|
row['rate_tr_Mbit'] = round((row['tr_cur_up'] - prev['tr_cur_up']) * 8 / dt / 1e6, 2)
|
||||||
|
row['rate_container_Mbit'] = round((row['tx'] - prev['tx']) * 8 / dt / 1e6, 2)
|
||||||
|
print(json.dumps(row), flush=True); prev = row; time.sleep(step)
|
||||||
@@ -0,0 +1,236 @@
|
|||||||
|
# Catalog sweep 2026-10-24 — every catalog app on a fresh box, the way a family uses it
|
||||||
|
|
||||||
|
Unattended validation sweep (Saturday 12:30Z → Sunday ~00:00Z). Brief: the operator's „every catalog app, one by one, on
|
||||||
|
a fresh box" prompt. Venue: demo-hp (Tier 0). Method: endpoint-level — every step is the request the dashboard or the
|
||||||
|
app's own page sends (no browser click-through except the screenshot harness; no `docker exec` to make state).
|
||||||
|
|
||||||
|
## Safest readings taken (the brief allows no question)
|
||||||
|
|
||||||
|
1. **ISO 1.30.0, not 1.36.0.** The brief said „1.36.0 or the current one"; the current one on felhom.eu/letoltes was
|
||||||
|
1.30.0 (sha256 checked against the published `.sha256`). See `_box/README.md`.
|
||||||
|
2. **12 GiB for the VM, not 16.** demo-hp had 14 GiB free beside 9201, 9202 and Tester 1; 12 is the brief's minimum.
|
||||||
|
The guest itself is 7.7 GiB (the golden's size); the box's own memory admission decided what could run.
|
||||||
|
3. **„Remove keeping data → install again with use my kept data" is a drive-folder question.** By design (`07` §6.5)
|
||||||
|
the question is asked only for an app that keeps a folder on a drive (`HDD_PATH`). For the 40+ apps that keep their
|
||||||
|
data in Docker volumes, „keep my data" keeps the recovery unit; the reinstall starts empty, and the item comes back by
|
||||||
|
the unit restore („Visszaállítás"). So step 5 + 6 ran as: keep-remove → reinstall → read (empty, expected) → unit
|
||||||
|
restore → read. That is a unit restore on **every** app with a seed, not only every fifth.
|
||||||
|
4. **„Mentés most" is the local copy only** (`07` §8, by design); the second-drive and off-site copies run in the night.
|
||||||
|
The sweep did not wait for a night, so „the right tiers ran" is checked as „the local unit holds the app".
|
||||||
|
5. **Categories:** the brief names „Apps & catalog" for bigger issues; a row whose harm is in backup/restore or storage is
|
||||||
|
filed in that category instead (`register_shape_gate` has one category per row, and the owner reads it there).
|
||||||
|
6. **Sev:** the brief's scale (data at risk P1/P2, a broken app P2, a confusing step P3). R-1269, R-1271 are P2 by that rule (the data is not deleted in either; the household loses access to it).
|
||||||
|
7. **The sweep box reads the LIVE catalog** (main `5e196e1`, then `227a544` — rules only, no template moved). The drill
|
||||||
|
catalog was used only for the three apps whose fix needed proving; the box was pointed back after (`_drill-repoint.txt`).
|
||||||
|
8. **Harness fixes live in the sweep's copy** (`tools/`), not in the catalog's scripts; porting them is R-1277.
|
||||||
|
|
||||||
|
## The box (Part 0)
|
||||||
|
|
||||||
|
`_box/README.md`: VM **9360** `sweep1024-box` (6 cores, 12 GiB, disks 256 + 200 + 150 GB on `nvme-scratch`), ISO 1.30.0,
|
||||||
|
customer Tester4 (hub host `Tester4-aa9466`), guest 9201, domain `tester4.test` (no DNS: the browser LXC maps the
|
||||||
|
names). Bind 12:42:18Z, day-0 SUCCESS 12:45:40Z, claim, recovery code escrowed (stored out-of-band), drives `hdd_1`
|
||||||
|
(210 GB) and `hdd_2` (157 GB) plus the installer's internal place `belso`. Versions at the start: controller 0.342.0,
|
||||||
|
agent 0.164.0, hub 0.169.0. Browser: LXC **9361** `sweep1024-browser` (Chromium + Playwright), 60 dashboard screens
|
||||||
|
(15 pages × phone/computer × HU/EN) in `_box/screens-dashboard/`.
|
||||||
|
|
||||||
|
## Method per app (Part 1)
|
||||||
|
|
||||||
|
`tools/sweep.py <app>` — one app at a time, small first, heavy last (`tools/order.txt`); the next starts after the last
|
||||||
|
is removed with its data. Each `<app>/` holds `record.json` (the measured facts), `walk.log`, the page texts
|
||||||
|
(`page-{install,guide,overview}-{hu,en}.txt`) and `screens/` (phone + computer, HU + EN, install page and the app after
|
||||||
|
first login). Steps: install through `POST /api/stacks/<app>/deploy` (timed to the front door answering); a stranger
|
||||||
|
request without the family cookie (must be refused where the template gates); one item through the app's own API (the
|
||||||
|
catalog's fixture where one exists, a new one in `tools/sweep_fixtures.py` for seven more); „Mentés most"; remove
|
||||||
|
keeping data; reinstall (the kept-data question answered „use" where it is asked); read; unit restore; read; remove with
|
||||||
|
data; leftovers checked (containers, volumes, `app.yaml`, `appdata`, `kept`, the subdomain → 404). Memory: the peak of
|
||||||
|
each container's **anon** memory from its cgroup (`memory.stat`; the file cache is not counted — project rule) against
|
||||||
|
its compose limit; a `*` in the table means only `docker stats` was read for that app (it includes the cache). The anon reading joined mid-sweep, so 22 apps carry it and 40 carry the `docker stats` upper bound; of those 40 none passed 79 % of its limit (Tandoor 1009/1280) and none restarted. Above 80 % on anon: only OpenGist (F14).
|
||||||
|
|
||||||
|
## Results
|
||||||
|
|
||||||
|
<!-- TABLE START -->
|
||||||
|
| App | Verdict | Peak memory / limit (MiB) | Install → answering (s) | What we saw | Row / fix |
|
||||||
|
|---|---|---|---|---|---|
|
||||||
|
| actualbudget | **works** | 49* / 256 | 30 | the seeded item back after the restore | |
|
||||||
|
| adventurelog | **works with a problem** | 381 / 896 | 207 | works; but the keep-backups → reinstall → restore path breaks it (DB password from the new install over the old data) — reproduced in its re-run | R-1269, R-1270 |
|
||||||
|
| audiobookshelf | **works** | 56* / 512 | 41 | the seeded item back after the restore | drive names: R-1275 |
|
||||||
|
| bentopdf | **works** | 3* / 384 | 30 | stateless (nothing to keep); installs, answers, removes clean | |
|
||||||
|
| bookstack | **works** | 167* / 768 | 71 | the seeded item back after the restore | |
|
||||||
|
| calcom | **works** | 807 / 1792 | 278 | the seeded item back after the restore; 2 start-up messages of the image itself | |
|
||||||
|
| calibre-web | **works** | 211 / 768 | 76 | the book (its own upload) is listed again after the keep-data reinstall; the fixture's download check gets 401 (its sign-in for /download, not the app); the admin name is the box's generated one (re-run 2) | R-1277 |
|
||||||
|
| claper | **works** | 267* / 640 | 86 | the seeded item back after the restore | |
|
||||||
|
| code-server | **works** | 81 / 1024 | 46 | first install refused by the registry's rate limit, told as „exit code 1”; the re-run installed and worked (no seed: the editor has no API) | R-1273 |
|
||||||
|
| crafty-controller | **works** | 79* / 2048 | 76 | the seeded item back after the restore | |
|
||||||
|
| dawarich | **works** | 835* / 2688 | 136 | the seeded item back after the restore (every-5th: restore also before the reinstall) | |
|
||||||
|
| docmost | **works** | 528* / 1152 | 126 | the seeded item back after the restore | |
|
||||||
|
| emby | **works** | 95 / 2048 | 41 | installs, answers, removes clean; no seed (no fixture for its wizard) | |
|
||||||
|
| ghost | **works** | 167* / 512 | 121 | the seeded item back after the restore | |
|
||||||
|
| gitea | **works** | 112 / 512 | 25 | the seeded item back after the restore | |
|
||||||
|
| glance | **works** | 7* / 128 | 15 | family gate holds a stranger; config-only (nothing to keep); the app's own headings are English | |
|
||||||
|
| gokapi | **works** | 6 / 128 | 15 | installs, own login, removes clean; no seed (API key only in its UI) | |
|
||||||
|
| grafana | **works** | 383* / 768 | 61 | the seeded item back after the restore | |
|
||||||
|
| gramps-web | **works** | 347* / 1024 | 206 | installs, answers, removes clean; no seed fixture | |
|
||||||
|
| grimmory | **works with a problem** | 594 / 1408 | 96 | first admin, library and an EPUB upload work; setup gate then family gate work as designed; the read-back through the family gate stayed inconclusive (the harness's gate logic, R-1277); remove-with-data leaves the empty parent folder | R-1276, R-1277 |
|
||||||
|
| grocy | **works** | 34* / 384 | 36 | the seeded item back after the restore | |
|
||||||
|
| home-assistant | **works** | 311 / 1024 | 161 | the seeded item back after the restore | |
|
||||||
|
| homebox | **works** | 205 / 256 | 20 | the seeded item back after the restore | |
|
||||||
|
| homepage | **works with a problem** | 91* / 256 | 25 | no gate at all: a stranger sees the box's CPU/RAM/disk (and later the family's links); still the upstream sample page | R-1272 |
|
||||||
|
| immich | **works with a problem** | 1845 / 6016 | 121 | album (new fixture) in; „use my kept data” loaded the rehearsal's future-dated database and Immich never answered again (404, 20 min) | R-1271 |
|
||||||
|
| jellyfin | **works** | 110 / 2048 | 41 | first user (new fixture, its wizard API) back after the restore | |
|
||||||
|
| karakeep | **works** | 649* / 2816 | 107 | the seeded item back after the restore | |
|
||||||
|
| kimai | **works** | 303* / 640 | 111 | own login; a Hungarian browser lands on /hu/ (curl without a language gets /en/) | |
|
||||||
|
| komga | **works** | 473* / 768 | 46 | the seeded item back after the restore | drive names: R-1275 |
|
||||||
|
| lubelogger | **works** | 60* / 512 | 25 | the seeded item back after the restore | |
|
||||||
|
| mealie | **works** | 322 / 1000 | 72 | the seeded item back after the restore (the fixture raced after_install on the first run) | |
|
||||||
|
| metube | **works** | 90* / 768 | 61 | family gate holds a stranger; download queue entry back | |
|
||||||
|
| n8n | **works with a problem** | 346 / 1536 | 151 | the owner back after the restore; on the live template it tries to send usage data out ([Rudder] lines, 6 in the live re-run) — the drill fix: the variable in the running container, 0 error lines | drill a601950 (morning list) |
|
||||||
|
| navidrome | **works** | 23* / 256 | 25 | the seeded item back after the restore | drive names: R-1275 |
|
||||||
|
| nextcloud | **works with a problem** | 298 / 1664 | 122 | user in; „use my kept data” loaded the rehearsal's future-dated database beside today's files — the user was not there | R-1271 |
|
||||||
|
| nzbget | **works** | 6* / 512 | 25 | the seeded item back after the restore; „no news server” until the family adds one (the guide says so) | |
|
||||||
|
| onlyoffice | **works** | 464 / 2048 | 71 | document server (no household data to keep); installs, answers, removes clean (every-5th restore) | |
|
||||||
|
| opengist | **works with a problem** | 125 / 128 | 15 | the gist owner back after the restore; but 128M is too small: one sign-up + sign-in hit the limit 443 times and swapped 98 MiB (memprobe-128M.txt); at 256M (drill) anon 216 MiB | drill 405077f (morning list) |
|
||||||
|
| outline | **works** | 327 / 1152 | 151 | the seeded item back after the restore | |
|
||||||
|
| paperless-ngx | **works with a problem** | 669* / 1664 | 146 | account in; „use my kept data” loaded the rehearsal's future-dated database — the account was not there | R-1271, R-1276 |
|
||||||
|
| papra | **works** | 320 / 768 | 76 | the seeded item back after the restore (Papra 26.6 needs an Origin header — a browser sends it) | |
|
||||||
|
| plant-it | **not installed** | – | – | lifecycle: abandoned — not offered for a new install (by design) | |
|
||||||
|
| plex | **works** | 223 / 2048 | 30 | installs, answers, removes clean; no seed (a server is claimed with a plex.tv account) | |
|
||||||
|
| privatebin | **works** | 31* / 128 | 15 | the seeded item back after the restore; D22 put-back PASS | |
|
||||||
|
| prowlarr | **works** | 114* / 512 | 21 | the seeded item back after the restore | |
|
||||||
|
| qbittorrent | **works** | 77* / 1024 | 15 | the seeded item back after the restore | |
|
||||||
|
| radarr | **works** | 100* / 512 | 41 | the seeded item back after the restore | |
|
||||||
|
| radicale | **works** | 33* / 128 | 25 | the seeded item back after the restore | |
|
||||||
|
| rallly | **works** | 346 / 1024 | 71 | installs, answers, removes clean; no seed: Rallly 4.15 closed password sign-up (login by e-mail code); the guide's polls need no account | R-1277 |
|
||||||
|
| recipe-importer | **works** | 67* / 128 | 25 | stateless helper; installs, answers, removes clean (every-5th restore) | |
|
||||||
|
| romm | **works with a problem** | 748* / 1536 | 116 | item back through „use my kept data” (its own copy); remove-with-data left the empty parent folder | R-1276 |
|
||||||
|
| sabnzbd | **works** | 69* / 1024 | 25 | the seeded item back after the restore | |
|
||||||
|
| seerr | **works** | 166 / 384 | 92 | installs, answers (307 to its setup), removes clean; no seed (needs a media server) | |
|
||||||
|
| sonarr | **works** | 80* / 512 | 41 | the seeded item back after the restore | |
|
||||||
|
| sparkyfitness | **works** | 417* / 1792 | 91 | the seeded item back after the restore | |
|
||||||
|
| tandoor | **works with a problem** | 1052* / 1536 | 177 | works; but the keep-backups → reinstall → restore path breaks it (DB password from the new install over the old data), and the restore still said success — reproduced on its own | R-1269, R-1270 |
|
||||||
|
| termix | **works** | 111* / 256 | 61 | installs, answers, removes clean (every-5th restore); no seed fixture | |
|
||||||
|
| transmission | **works** | 6* / 512 | 20 | the seeded item back after the restore | |
|
||||||
|
| uptime-kuma | **works** | 120* / 384 | 76 | the seeded item back after the restore | |
|
||||||
|
| vaultwarden | **works** | 37 / 256 | 25 | installs in 25 s, its own login page, sign-up closed (404); the box admin's invite works (run-live-2), but 1.37.4 has no route for the fixture's invited sign-up, so no item was seeded | R-1277 |
|
||||||
|
| vikunja | **works** | 37* / 256 | 10 | the seeded item back after the restore; D22 put-back PASS | |
|
||||||
|
| wanderer | **not testable on this box** | 140* / 1024 | 605 | not testable on this box: its web part calls its database at the PUBLIC name, which tester4.test cannot resolve (known, R-739) — unhealthy, 404 | R-739 |
|
||||||
|
| wger | **not installed** | – | – | lifecycle: hidden — not offered for a new install (by design) | |
|
||||||
|
| wishlist | **works** | 200* / 512 | 71 | the seeded item back after the restore | |
|
||||||
|
| zipline | **works** | 233* / 768 | 81 | the seeded item back after the restore | |
|
||||||
|
|
||||||
|
works: 52, works with a problem: 10, not installed: 2, not testable on this box: 1 (of 65)
|
||||||
|
<!-- TABLE END -->
|
||||||
|
|
||||||
|
## Findings — each a row, a fix or NOT-A-FINDING
|
||||||
|
|
||||||
|
| # | What | Disposition |
|
||||||
|
|---|---|---|
|
||||||
|
| F10 | A reinstall's own capture rewrites the unit that „remove — keep backups" kept; restoring it pairs the NEW install's DB password with the OLD data — the app never starts, the item is gone (Tandoor ×2, AdventureLog) | R-1269 (P2) |
|
||||||
|
| F11 | The unit restore reports `ok: true` „…az alkalmazás újraindult" while the controller logged „health check failed … did not reach running state" | R-1270 (P2) |
|
||||||
|
| F12 | „Use my kept data" loaded the rehearsal box's future-dated off-site database (2026-10-28) for Paperless-ngx, Nextcloud and Immich — another drive's install, which `07` §6.5 says is refused; Immich then never answered | R-1271 (P2) |
|
||||||
|
| F1 | Homepage has no gate: a stranger sees the box's CPU/RAM/disk (Glance, same class, is gated) | R-1272 (operator decision) |
|
||||||
|
| F7 | An install refused by the registry's rate limit (`toomanyrequests`) tells the household only „exit code 1" | R-1273 |
|
||||||
|
| F8 | The restore refusal says „turn on remote backup, or connect a second drive" when both exist but have not copied yet | R-1274 |
|
||||||
|
| F2 | English pages show the drive names in Hungarian („Belső tárhely", „Külső tárhely (hdd_1)") | R-1275 |
|
||||||
|
| F9 | „Remove with data" leaves the empty parent folder `appdata/<app>` (Grimmory, RomM, Paperless-ngx) | R-1276 |
|
||||||
|
| — | The catalog's box fixtures drifted from the apps (OpenGist, Calibre-Web, Mealie, Papra, Rallly; seven apps had none) | R-1277 |
|
||||||
|
| F13 | n8n sends usage data (`[Rudder]` lines; no `N8N_DIAGNOSTICS_ENABLED=false`) | **fixed in the drill** `a601950`, proven below; morning list |
|
||||||
|
| F14 | OpenGist at 125.8 of 128 MiB anon after one sign-up + sign-in; at 128M it hits the limit 443 times and swaps 98 MiB | **fixed in the drill** `405077f` + `492a17f` (256M), proven below; morning list |
|
||||||
|
| — | 52 anonymous Docker volumes stayed after every app was removed with data (PrivateBin, OnlyOffice, RomM, Immich: the images' own `VOLUME` paths; 4–72 KB, no household data). The sweep's own leftover check filters by the app's label and missed them; found in the end-state read (`_box/anon-volumes.txt`) | known: note on R-1267 (widened to OnlyOffice and RomM) |
|
||||||
|
| — | Wanderer unhealthy, 404: its web part calls its database by the public name, which a box without DNS cannot resolve | known: note on R-739 |
|
||||||
|
| F3 | The install page's default place is the internal place `belso`, not the 210 GB drive | NOT-A-FINDING: by design — `16` §5 registers the internal place as the default drive |
|
||||||
|
| F4 | Guide labels in English (LubeLogger „Settings", NZBGet „Settings / News-Servers") | NOT-A-FINDING: they are the app's own on-screen labels, and the Hungarian catalog copy is frozen by the copy-i18n gate (measured on the drill) |
|
||||||
|
| F5 | Kimai opens in English | NOT-A-FINDING: a Hungarian browser lands on `/hu/`; only a request with no language gets `/en/` |
|
||||||
|
| F6 | AdventureLog backend 95 % of its limit | NOT-A-FINDING: that was `docker stats` with the cache; on anon 288.7/384 MiB (75 %), = the ladder's 73 % |
|
||||||
|
| — | Database error lines (SparkyFitness `database "sparky" does not exist`, Dawarich `database "dawarich" already exists`; SparkyFitness `Invalid password` = the harness's wrong-password control) | NOT-A-FINDING: the images' own first-start and restore-time checks; no restart, the item read back after the restore |
|
||||||
|
| — | Radicale redirects to an IP | NOT-A-FINDING: a curl artefact (no Host on the redirect probe); the browser lands right |
|
||||||
|
| — | NZBGet / Transmission screenshots show the browser's basic-auth prompt | NOT-A-FINDING: the app's own login, by design |
|
||||||
|
| — | code-server's first install failed (`toomanyrequests`) | the failed install was removed through the product (`code-server.failed-remove.txt`); the re-run worked; the message is R-1273 |
|
||||||
|
|
||||||
|
## F10 proof — the kept unit is rewritten by the reinstall (`_unitprobe/tandoor/`)
|
||||||
|
|
||||||
|
`tools/unitprobe.py tandoor`: install, one account, „Mentés most" (unit 19:45:14–20), remove keeping data and
|
||||||
|
backups, install again (a volume app: no kept-data question), wait for the controller's next capture, „Visszaállítás".
|
||||||
|
|
||||||
|
- 19:50:32 `Recovery unit captured for tandoor` — the unit's `app.yaml` checksum `0f366c97…` → `4438c1c5…`,
|
||||||
|
`created_at` 19:45:26 → 19:50:32; the data files unchanged (`unit-1/2/3*.txt`).
|
||||||
|
- The restore: `ok: true`, „A(z) tandoor: 3 adatkötet és az adatbázis visszaállítva — az alkalmazás újraindult."
|
||||||
|
- After: state `starting`, the account not there, `password authentication failed` ×42 (app) and ×22 (postgres).
|
||||||
|
- The comment at `felhom-controller/controller/internal/backup/recovery_unit.go:101-103` says the refresh never rewrites
|
||||||
|
the definition away from its data; after a remove + reinstall it does.
|
||||||
|
|
||||||
|
Seen three times: the sweep's own Tandoor cycle (17:27:41 reinstall → 17:30:34 capture → 17:30:47 restore), this probe,
|
||||||
|
and AdventureLog's re-run (20:03:18 → 20:05:32 → restore). AdventureLog's first run worked only because no capture fell
|
||||||
|
between its reinstall and its restore.
|
||||||
|
|
||||||
|
## D22 put-back — PASS on two apps (`_putback/`)
|
||||||
|
|
||||||
|
`tools/putback.py <app>`: the largest volume tar of the app's unit truncated to half, then „Visszaállítás". PrivateBin
|
||||||
|
and Vikunja: the restore failed honestly („Nem változott semmi…"), the pre-restore data was put back, the app ran, and the
|
||||||
|
seeded item read back through the front door.
|
||||||
|
|
||||||
|
## Drill fixes, proven on the sweep box pointed at the drill (Part 2)
|
||||||
|
|
||||||
|
Drill catalog `app-catalog-drill` (never the live catalog): `742a354` (merge of live main), then:
|
||||||
|
|
||||||
|
- `a601950` **n8n** — `N8N_DIAGNOSTICS_ENABLED=false` in the compose (usage data stays home).
|
||||||
|
- `405077f` + `492a17f` **OpenGist** — memory 128M → 256M (compose + `.felhom.yml` `mem_limit`), the comment carries the
|
||||||
|
measured need.
|
||||||
|
|
||||||
|
**How the box was pointed at the drill.** The drill repo on Gitea is private (R-992) and the box holds no catalog
|
||||||
|
credential, so the box reads a local mirror: a bare repository in the controller's data volume, loaded from a git bundle,
|
||||||
|
`repo_url: file:///opt/docker/felhom-controller/drill-catalog.git` (the method of
|
||||||
|
`audits/burndown-3-2026-10-17/tools/repoint.py`; loading the mirror is a `docker exec git fetch` into the controller —
|
||||||
|
catalog plumbing, not app state). Only `repo_url` changes; `tools/repoint.py live` puts the saved `controller.yaml` back
|
||||||
|
and checks it byte for byte (`RESTORED-IDENTICAL`, twice in `_drill-repoint.txt`). The box ended on the live catalog
|
||||||
|
`227a544`.
|
||||||
|
|
||||||
|
**What went wrong first, and is kept as evidence.** The first drill pass (chain 3, 21:18Z) set `repo_url` to the Gitea
|
||||||
|
URL in the belief that the repo was public. The box's clone failed (`fatal: could not read Username`), the controller kept
|
||||||
|
the templates it had, and n8n, OpenGist and Vaultwarden ran on the LIVE templates. The leftover `[Rudder]` lines in that
|
||||||
|
n8n run showed it. Those three runs are kept as `run-live-2/` (valid live re-runs). The second pass (chain 4) refused to
|
||||||
|
install anything until the box's catalog cache read the drill's commit and the changed lines were in the box's copy of
|
||||||
|
both templates: `drill main fd23571, box cache fd23571: DRILL-CONFIRMED`. (The drill also carries a six-line Plex
|
||||||
|
ladder entry from an earlier session's test; Plex was not installed in this pass.)
|
||||||
|
|
||||||
|
**n8n proof.** The running container carried `N8N_DIAGNOSTICS_ENABLED=false` (read at 21:37:58, that line only); error
|
||||||
|
lines 0 against 6 `[Rudder]` lines in the live re-run just before it (the control); the owner account read back after the
|
||||||
|
unit restore.
|
||||||
|
|
||||||
|
**OpenGist proof — and a wrong turn.** At 256M (drill) OpenGist took 215.8 MiB anon. I first read that as „it grows into
|
||||||
|
its limit" and reverted the fix (`c420053`); an automated security review flagged the revert as a lowered resource cap.
|
||||||
|
A measurement settled it (`tools/memprobe-guest.sh`, `opengist/memprobe-128M.txt`): on the live 128M template one
|
||||||
|
sign-up + sign-in hit `memory.max` 443 times and pushed 98 MiB into swap (anon 125 MiB, 0 `oom_kill`, 0 restarts). The
|
||||||
|
real need is ~220 MiB; the fix went back in (`405077f`). Where the box has no swap room left, the same sign-up would end in a kill (not measured here).
|
||||||
|
|
||||||
|
Drill gates: `catalog_gates.py --fast opengist n8n` from the drill clone — all OK except `security-scan`, which looks for
|
||||||
|
a `felhom.eu` clone beside the drill and found none; the shared scanner run by hand on the drill clone: 0 hits.
|
||||||
|
|
||||||
|
## Teardown
|
||||||
|
|
||||||
|
Three layers, 2026-10-25 ~00:05Z:
|
||||||
|
|
||||||
|
- **Machine (demo-hp):** VM **9360** `sweep1024-box` and LXC **9361** `sweep1024-browser` — both created by this session —
|
||||||
|
stopped and destroyed with `--purge`; no disk of either is left on `nvme-scratch`. Untouched: Tester 1 (VM 341), 9201,
|
||||||
|
9202, 9401, the thin pool, and the 1.30.0 ISO in `local:iso` (there since 2026-10-22). Before the destroy, the box's
|
||||||
|
controller log, the agent journal, its end state and its anonymous-volume list were copied here (`_box/`).
|
||||||
|
- **Host (DooPlex):** the session's secret files (`/run/user/1000/sweep-sc`: the dashboard password copy, cookies, CSRF
|
||||||
|
tokens) shredded. The drill catalog keeps its commits (`a601950`, `405077f`, `492a17f`, and the reverted turn
|
||||||
|
`c420053`) for the morning list; nothing went to the live catalog.
|
||||||
|
- **Hub:** host **Tester4-aa9466** stays for the operator to delete (CC holds no delete). Until then it reads as a box
|
||||||
|
that stopped reporting.
|
||||||
|
- **ep0:** listed, nothing removed. The host's PBS link to ep0 (`felhom-pbs`) was set up at day-0 (`pbsdr: converged`,
|
||||||
|
14:44:48 CEST, `_box/agent-journal.log.gz`); no whole-box copy went to it (weekly cadence — only the local copy ran,
|
||||||
|
14:51). The app copies the box sent off-site went to the Tester4 restic repository on the storage box (unlocked with
|
||||||
|
the rehearsal's recovery code; a new code escrowed — stored out-of-band), not to ep0.
|
||||||
|
|
||||||
|
## Evidence map
|
||||||
|
|
||||||
|
- `_box/` — Part 0 (VM, ISO, bind, day-0, claim, recovery, drives, dashboard screens).
|
||||||
|
- `<app>/` — one per app; re-run apps keep the earlier run in `first-run/`, `run-2/` or `run-drill-pre/`.
|
||||||
|
- `<app>.console.txt`, `progress.log`, `batch.out`, `chain2.out`, `chain3.out` — the runner's own output.
|
||||||
|
- `_unitprobe/tandoor/` — F10. `_putback/{privatebin,vikunja}/` — D22.
|
||||||
|
- `tools/` — the harness (sweep copies of the catalog's `box_walk.py` and `upgrade_fixtures_box.py`, marked SWEEP COPY;
|
||||||
|
`sweep.py`, `sweep_fixtures.py`, `unitprobe.py`, `putback.py`, `repoint.py`, `table.py`, the rehearsal's VM and browser
|
||||||
|
helpers). Secrets were read by name through `sec` and never written here; the evidence was scanned for every known
|
||||||
|
secret value and for token shapes (one app session token found and blanked: Actual Budget's), with a positive control.
|
||||||
|
- Screenshots were reduced with `pngquant` (lossy palette) to keep the repository small.
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
# Part 0 — the sweep box (2026-10-24, from 12:33Z)
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| VM | **9360 `sweep1024-box`** on demo-hp: 6 cores (`cpu host`), **12 GiB** (demo-hp had 14 GiB available of 29; 9201, 9202 and Tester 1 untouched — the brief's minimum, not its 16 GiB aim), 256 GB system disk + 200 GB + 150 GB data disks, all raw on `nvme-scratch` (the dir storage), never the thin pool (`vm-create.txt`) |
|
||||||
|
| ISO | `felhom-installer-1.30.0-pve9.2-1.iso` — the current one on felhom.eu/letoltes; sha256 `590d7b9b…74d5cc1` = the published `.sha256` |
|
||||||
|
| Installer | graphical, Hungarian keyboard kept, driven by `qm sendkey` (`tools/vmctl.sh`), screens in `install/`. Install pressed **12:37:33Z**; pairing code on the console **12:41** (blacked out in `install/11-console.png`) |
|
||||||
|
| Bind | Tester4's old host deleted by the operator (12:32, asked by CC); the hub mailed a new bind link 12:31:57Z; bound through the real page in Chromium 12:42:18Z, first try (`screens/b-bind-*`) |
|
||||||
|
| Day-0 | 8/8 steps, **SUCCESS 12:45:40Z** — host `Tester4-aa9466`, guest 9201, golden imported and verified; Install → done **8 min 7 s** |
|
||||||
|
| Claim | setup-code mail 12:43:14Z („Új beállító kód — újratelepült a szervered", right for a customer who had a box before); claimed 302 → `/` (`claim.txt`) |
|
||||||
|
| Recovery | the box found Tester4's sealed off-site package (2026-10-20 14:16) and asked for the old code first (`/recovery`): the 2026-10-20 code, typed as it was written down (ten words joined by hyphens), unlocked it — controller „the offsite repository key was recovered and placed" 12:47:41Z. Then the dashboard asked for a NEW code: the ceremony in Chromium (words 4 and 9 asked), escrowed; „Távoli mentés: Aktív" (`screens/box-escrow-*`, the code blacked out) |
|
||||||
|
| Drives | `/dev/sdb` → `hdd_1` (210 GB, the apps' data), `/dev/sdc` → `hdd_2` (157 GB), both through `POST /api/storage/init` (`drive-init.txt`); the installer's internal place `belso` (59 GB) |
|
||||||
|
| Versions | controller **0.342.0**, agent **0.164.0**, hub **0.169.0**; the box syncs the LIVE catalog (`app-catalog-felhom.eu` main, `5e196e1` at the start; it re-syncs every 15 min) |
|
||||||
|
| Guest | 9201 at 192.168.0.157, 7.7 GiB RAM (the golden's size), domain `tester4.test` (no DNS — the browser LXC maps the names; TLS not verified) |
|
||||||
|
| Browser | LXC **9361 `sweep1024-browser`** (2 cores, 2 GB, nvme-scratch): Chromium 156.0.8078.47 + Playwright 1.52.0; the household's session stored in the LXC (`/root/work/state-dash.json`) |
|
||||||
|
| Dashboard | every menu page at phone and computer size, HU and EN (`screens-dashboard/`, 60): no sideways scroll reported; a signed-in household reads its OWN language setting whatever the browser asks — by design (`langFor`), the page texts were read with `?lang=` |
|
||||||
|
|
||||||
|
Secrets used (names only, `sec`): SWEEP_2026_10_24_ROOTPW (the typed root password — day-0 replaced it with the vaulted
|
||||||
|
one; DooPlex's key was added before the bind), SWEEP_2026_10_24_BIND_TOKEN, _PAIRING_CODE, _SETUP_CODE, _DASH_PW,
|
||||||
|
_RECOVERY_CODE, and the existing REHEARSAL_2026_10_20_PASSPHRASE / _RECOVERY_CODE.
|
||||||
Binary file not shown.
@@ -0,0 +1,52 @@
|
|||||||
|
2026-10-24T17:15:18Z 28 assets,config,library,
|
||||||
|
2026-10-24T12:51:57Z 24 client_body,fastcgi,proxy,
|
||||||
|
2026-10-24T13:08:14Z 24 client_body,fastcgi,proxy,
|
||||||
|
2026-10-24T19:54:35Z 4
|
||||||
|
2026-10-24T17:15:18Z 4
|
||||||
|
2026-10-24T12:51:49Z 24 client_body,fastcgi,proxy,
|
||||||
|
2026-10-24T19:54:18Z 4
|
||||||
|
2026-10-24T13:06:43Z 24 client_body,fastcgi,proxy,
|
||||||
|
2026-10-24T13:08:05Z 72 lock,nginx,nginx.pid,
|
||||||
|
2026-10-24T19:06:36Z 36 documentserver,documentserver-example,
|
||||||
|
2026-10-24T13:06:43Z 4
|
||||||
|
2026-10-24T13:08:05Z 24 client_body,fastcgi,proxy,
|
||||||
|
2026-10-24T13:06:43Z 72 lock,nginx,nginx.pid,
|
||||||
|
2026-10-24T19:54:18Z 24 client_body,fastcgi,proxy,
|
||||||
|
2026-10-24T17:18:02Z 4
|
||||||
|
2026-10-24T13:07:11Z 24 client_body,fastcgi,proxy,
|
||||||
|
2026-10-24T12:50:39Z 72 lock,nginx,nginx.pid,
|
||||||
|
2026-10-24T13:07:11Z 72 lock,nginx,nginx.pid,
|
||||||
|
2026-10-24T19:09:38Z 36 documentserver,documentserver-example,
|
||||||
|
2026-10-24T17:20:41Z 28 assets,config,library,
|
||||||
|
2026-10-24T19:54:35Z 24 client_body,fastcgi,proxy,
|
||||||
|
2026-10-24T12:50:39Z 24 client_body,fastcgi,proxy,
|
||||||
|
2026-10-24T12:51:57Z 72 lock,nginx,nginx.pid,
|
||||||
|
2026-10-24T19:07:30Z 4
|
||||||
|
2026-10-24T12:50:57Z 24 client_body,fastcgi,proxy,
|
||||||
|
2026-10-24T19:21:09Z 4
|
||||||
|
2026-10-24T19:09:38Z 4
|
||||||
|
2026-10-24T12:50:57Z 72 lock,nginx,nginx.pid,
|
||||||
|
2026-10-24T19:20:01Z 4
|
||||||
|
2026-10-24T19:55:03Z 4
|
||||||
|
2026-10-24T12:51:57Z 4
|
||||||
|
2026-10-24T19:54:18Z 72 lock,nginx,nginx.pid,
|
||||||
|
2026-10-24T17:18:02Z 28 assets,config,library,
|
||||||
|
2026-10-24T19:24:43Z 4
|
||||||
|
2026-10-24T12:50:39Z 4
|
||||||
|
2026-10-24T13:08:05Z 4
|
||||||
|
2026-10-24T12:51:49Z 4
|
||||||
|
2026-10-24T19:07:30Z 32 documentserver,documentserver-example,
|
||||||
|
2026-10-24T19:06:36Z 4
|
||||||
|
2026-10-24T19:07:39Z 4
|
||||||
|
2026-10-24T13:08:14Z 4
|
||||||
|
2026-10-24T17:17:24Z 4
|
||||||
|
2026-10-24T12:50:57Z 4
|
||||||
|
2026-10-24T17:17:24Z 28 assets,config,library,
|
||||||
|
2026-10-24T19:55:03Z 24 client_body,fastcgi,proxy,
|
||||||
|
2026-10-24T19:55:03Z 72 lock,nginx,nginx.pid,
|
||||||
|
2026-10-24T13:08:14Z 72 lock,nginx,nginx.pid,
|
||||||
|
2026-10-24T19:07:39Z 36 documentserver,documentserver-example,
|
||||||
|
2026-10-24T19:54:35Z 72 lock,nginx,nginx.pid,
|
||||||
|
2026-10-24T17:20:41Z 4
|
||||||
|
2026-10-24T12:51:49Z 72 lock,nginx,nginx.pid,
|
||||||
|
2026-10-24T13:07:11Z 4
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
GET /claim HTTP/2 200
|
||||||
|
csrf_len 64 cookie_present yes
|
||||||
|
POST /claim HTTP/2 302 location: /
|
||||||
Binary file not shown.
@@ -0,0 +1,11 @@
|
|||||||
|
candidates 200 {"data": {"vmid": 9201, "initialize": [{"device": "/dev/sdb", "size_bytes": 214748364800, "model": "QEMU HARDDISK", "data_bearing": false, "mountable": false}, {"device": "/dev/sdc", "size_bytes": 161061273600, "model": "QEMU HARDDISK", "data_bearing": false, "mountable": false}], "attach": []}, "ok": true}
|
||||||
|
12:50:04Z init /dev/sdb hdd_1 200 {"data": {"phase": "formatting", "started": true}, "ok": true}
|
||||||
|
status 200 {"device": "/dev/sdb", "durable_id": "", "error": "", "opsign": "", "phase": "done", "reason": "", "started_at": "2026-10-24T12:50:04.496400956Z", "updated_at": "2026-10-24T12:50:06.691612961Z", "where": "/mnt/felhom-drives/hdd_1"}
|
||||||
|
12:50:09Z init /dev/sdc hdd_2 200 {"data": {"phase": "formatting", "started": true}, "ok": true}
|
||||||
|
status 200 {"device": "/dev/sdc", "durable_id": "", "error": "", "opsign": "", "phase": "done", "reason": "", "started_at": "2026-10-24T12:50:09.550638661Z", "updated_at": "2026-10-24T12:50:11.762283786Z", "where": "/mnt/felhom-drives/hdd_2"}
|
||||||
|
9dcbe7bf-098d-4d99-9c43-f39fe2b20f42 disk user-data attached 157 GB /mnt/felhom-drives/hdd_2
|
||||||
|
belso internal user-data attached 59 GB /mnt/felhom-drives/belso
|
||||||
|
e8b4f4a0-7aca-4829-9220-b394948b9368 disk user-data attached 210 GB /mnt/felhom-drives/hdd_1
|
||||||
|
local system system attached 77 GB None backup_target
|
||||||
|
local-lvm system system attached 165 GB None
|
||||||
|
felhom-pbs network backup attached 0 GB None
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
filebrowser gtstef/filebrowser:1.5.6-stable Up 9 hours (healthy)
|
||||||
|
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.342.0 Up 6 minutes (healthy)
|
||||||
|
traefik traefik:v3.7.13 Up 9 hours
|
||||||
|
54
|
||||||
|
Filesystem Size Used Avail Use% Mounted on
|
||||||
|
/dev/sdb 196G 268M 186G 1% /mnt/felhom-drives/hdd_1
|
||||||
|
/dev/sdc 147G 2.2M 140G 1% /mnt/felhom-drives/hdd_2
|
||||||
|
/dev/mapper/pve-vm--9201--disk--0 32G 1.2G 29G 4% /
|
||||||
Binary file not shown.
|
After Width: | Height: | Size: 58 KiB |
Loaded 100 of 2967 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user