D63 live: R-1256 H4 + H5 failed stop proven on Tester 1 (test build); D62 proven live; R-1340 filed (hold mail tells the option-C story for any cause); 07 §6.3 [FACT]; evidence audits/d63-2026-10-29/
gates / gates (push) Successful in 1m45s
gates / gates (push) Successful in 1m45s
This commit is contained in:
admin committed
1 parent
bf69bb91ac
commit
41f8f67015
20 files changed
+1964
-1
No files matched your search
@@ -846,6 +846,19 @@ under the backup manager's lock. One `[WARN]` log line and one operator-only eve
|
||||
restore date, size; hub `19afefab`) per delete. Controller `e3fce09`, tests `TestD62_*`. **D63** (`09` §3 decision 282):
|
||||
H4 and H5's failed stop ship only with a live proof, staged with a fault switch that exists only in a `-tags faultinject`
|
||||
test build.
|
||||
**`[FACT]` 2026-10-29, Tester 1, test build `0.347.0-rc.1-d63fault` (`audits/d63-2026-10-29/`):** (H4, 08:10 UTC) a
|
||||
throwaway Radicale ran the true 3.8.0 (digest `0d60d544…`) while its newest off-site snapshot held 3.8.1's data; the full
|
||||
restore said „it comes back at the snapshot's version", the fault failed the definition write after the compose was
|
||||
written, the log then read „the live definition is written back before the start" and only then „Starting stack"; the app
|
||||
ran 3.8.0 again, no hold, and an event made before the snapshot and one made after it both read back. (H5, 08:18 UTC) a
|
||||
created-only container held the data volume („Failed to remove volume … (not written into)"); the restore's own stop
|
||||
passed, the put-back's stop failed — „nothing is put back; the copy stays at …; holding the app"; the app page showed the
|
||||
hold sentence; on the restore page the held copy's „Törlés most" was off with „Nem törölhető: az alkalmazás le van
|
||||
állítva, és ez a másolat az út vissza." (D62, 08:15 UTC) a copy kept because a second restore began inside the first one's
|
||||
stable-start window was listed with „visszaállítás: 2026. 10. 29. 09:11 · 37,3 KB · automatikusan törlődik: 2026. 11.
|
||||
28."; a wrong typed name → 400, the right one → 200, the copy gone, one `[WARN]` line; the event got HTTP 400 from hub
|
||||
0.174.1, which does not know the type yet (hub 0.175.0 does). The release image carries no fault switch (`grep -c "FAULT
|
||||
INJECTED"` = 0 on `0.347.0-rc.1`, 2 on the test image).
|
||||
**`[FACT]` 2026-10-29 night, scratch 9202 (`audits/night-2026-10-28/r1256-restore/h5-live/`):** vikunja's first volume
|
||||
held by a created-only container, the household's restore pressed through the page: „Failed to remove volume
|
||||
vikunja_vikunja_data before the restore … (not written into)", the restore failed, the put-back poured the live copy of
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
# D63 + D62 live proofs — Tester 1, 2026-10-29 (attended)
|
||||
|
||||
Test build `0.347.0-rc.1-d63fault` = controller `e3fce09` built with `-tags faultinject` on top of the local release
|
||||
candidate `0.347.0-rc.1` (never pushed; removed from the box after). The release image carries no fault switch
|
||||
(`grep -c "FAULT INJECTED"`: 0 on `0.347.0-rc.1`, 2 on the test image). Driver: `d63_live.py` (box_walk helpers).
|
||||
|
||||
| File | What it shows |
|
||||
|---|---|
|
||||
| `run-setup.log`, `run-setup2.log` | first setup tries. A hand-edited compose at "3.8.0" kept 3.8.1's digest, and the backup refused the app (R-645: it ran a version other than its pin) — so no unit. Both reverted. These runs pushed one `backup_run_failures` event (Radicale not protected off-site) — caused by this setup. |
|
||||
| `run-setup3.log` | the real setup: Radicale backed up at its catalog 3.8.1 (local unit + off-site snapshot `78002a51`). |
|
||||
| `run-pull.log` | the off-site snapshot pulled to the restore scratch (the wizard's two steps). |
|
||||
| `run-h4.log`, `run-h4b.log`, `run-h4c.log` | failed seeds: 401 from Radicale. Cause: the harness, not the app — the generated login lives only in the deploying process, a new process sent an empty password. NOT 3.8.0 (an early guess, withdrawn). |
|
||||
| `run-h4d.log` | **H4 proven**: live 3.8.0 (true digest `0d60d544…`), snapshot 3.8.1; fault after the compose write; the live definition written back before the start; app on 3.8.0, no hold, T1 + T2 read back. |
|
||||
| `run-d62.log` | first D62 try: the script typed the wrong name (a parsing bug in the script) → 400; its "copy gone" line came from a finder that could not see `/mnt/sys_drive` (wrong). |
|
||||
| `run-d62b.log` | **D62 proven**: the list line with the date and „automatikusan törlődik", the warning text, wrong name 400, right name 200, the copy gone, the log line; the hub event 400 (hub 0.174.1 does not know the type). |
|
||||
| `run-h5.log` | **H5's failed stop proven**: busy volume not written into; the put-back's stop failed; nothing put back; copy kept; app held; the held copy's button off. Its operator mail text is wrong for this case → R-1340. |
|
||||
| `run-teardown.log`, `tester1-after.txt` | the app removed; controller back on 0.346.1; test image removed; 15 anonymous volumes (R-1267's PrivateBin class, made by this session's five backup runs) removed by name — 268 volumes, as before. The Radicale **restore hold record stays** (clearing a hold is the operator's). |
|
||||
| `controller-full.log` | the controller's whole log 07:50 → teardown. |
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1 @@
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault Up 50 seconds (healthy)
|
||||
@@ -0,0 +1,260 @@
|
||||
#!/usr/bin/env python3
|
||||
"""d63_live.py <phase> — D63 live proofs on Tester 1 (2026-10-29, attended), test build with `-tags faultinject`.
|
||||
|
||||
Phases:
|
||||
setup deploy a throwaway Radicale, seed T1 through its own front door, run it at 3.8.0 (the stack folder's
|
||||
compose edited, the household's Restart), switch it into the off-site set and press „Mentés most" for
|
||||
off-site, then put the compose back at 3.8.1 and Restart — so the newest off-site snapshot holds 3.8.0's
|
||||
data under 3.8.1's live definition.
|
||||
pull pull the newest off-site snapshot of radicale to the restore scratch (the wizard's two steps).
|
||||
h4 seed T2; arm defwrite-after-compose "0 1"; press the full restore. Want: the fault, then the live definition
|
||||
written back BEFORE the start; the app runs 3.8.1; no hold; T1 and T2 read back.
|
||||
d62 a full restore with no fault; the copy is listed on the restore page with its date; a wrong name is refused;
|
||||
the right name deletes it; it is gone.
|
||||
h5 a created-only container holds radicale's data volume (the restore cannot replace it → the put-back);
|
||||
arm stop-radicale "1 1" (the restore's own stop passes, the put-back's stop fails). Want: nothing is put
|
||||
back, the copy stays, the app is held with the D20 sentence; the page shows the held copy's button OFF.
|
||||
teardown the blocker and the fault files go; the app is removed through the product.
|
||||
State (passwords) stays in the scratchpad, never printed."""
|
||||
import json, os, re, subprocess, sys, time
|
||||
|
||||
sys.path.insert(0, "/mnt/5_hdd/felhom.eu/git/app-catalog-felhom.eu/scripts")
|
||||
SCR = "/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0b76c4d7-9b39-4e4e-869d-d0d2c6006331/scratchpad"
|
||||
os.environ.setdefault("SC", f"{SCR}/rt-tester1")
|
||||
os.environ.setdefault("TARGET", "tester-1")
|
||||
import box_walk as w
|
||||
import upgrade_fixtures_box as fb
|
||||
|
||||
EV = f"{SCR}/d63/ev"
|
||||
os.makedirs(EV, exist_ok=True)
|
||||
STATE = f"{SCR}/d63/.state.json"
|
||||
APP, SUB = "radicale", "radicale"
|
||||
fx = fb.FIXTURES[APP]
|
||||
FAULTS = "/var/lib/docker/volumes/felhom-controller-data/_data/faults"
|
||||
COMPOSE = f"/opt/docker/stacks/{APP}/docker-compose.yml"
|
||||
OLD, NEW = "ghcr.io/kozea/radicale:3.8.0", "ghcr.io/kozea/radicale:3.8.1"
|
||||
OLD_FULL = "ghcr.io/kozea/radicale:3.8.0@sha256:0d60d544710bda0fa83bfdea685db731b6cbf74e12d7f22b4997e9cff3a50468"
|
||||
NEW_FULL = "ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c"
|
||||
|
||||
|
||||
def state():
|
||||
return json.load(open(STATE)) if os.path.exists(STATE) else {}
|
||||
|
||||
|
||||
def save(st):
|
||||
json.dump(st, open(STATE, "w"))
|
||||
os.chmod(STATE, 0o600)
|
||||
|
||||
|
||||
def form(path, fields, accept_json=False):
|
||||
"""A dashboard form post with the session and its CSRF token — what the page's button sends."""
|
||||
sess = open(f"{w.SC}/sess{os.getpid()}.txt").read().strip()
|
||||
csrf = open(f"{w.SC}/csrf{os.getpid()}.txt").read().strip()
|
||||
args = ["curl", "-sk", "-D", "-", "-H", w.HOSTHDR, "-H", f"Cookie: {sess}", "-H", f"X-CSRF-Token: {csrf}", "-X", "POST",
|
||||
"--data-urlencode", f"_csrf={csrf}"]
|
||||
if accept_json:
|
||||
args += ["-H", "Accept: application/json"]
|
||||
for k, v in fields.items():
|
||||
args += ["--data-urlencode", f"{k}={v}"]
|
||||
bodyf = f"{SCR}/d63/.body"
|
||||
r = w.sh(args + ["-o", bodyf, f"{w.BASE}{path}"], timeout=180)
|
||||
out = r.stdout or ""
|
||||
head = out.split("\n")[0].strip()
|
||||
loc = [l.strip() for l in out.split("\n") if l.lower().startswith("location:")]
|
||||
body = open(bodyf).read() if os.path.exists(bodyf) else ""
|
||||
return head, loc[:1], body
|
||||
|
||||
|
||||
def wait_op(label, cap=1800):
|
||||
t0, last = time.time(), None
|
||||
while time.time() - t0 < cap:
|
||||
code, d = w.ctl("GET", "/api/backup/restore-status")
|
||||
dd = d.get("data") or {}
|
||||
cur = (dd.get("running"), dd.get("op"), dd.get("ok"), (dd.get("message") or "")[:300])
|
||||
if cur != last:
|
||||
w.say(f" +{round(time.time()-t0):>5}s {label}: {cur}")
|
||||
last = cur
|
||||
if not dd.get("running") and time.time() - t0 > 5:
|
||||
return dd
|
||||
time.sleep(3)
|
||||
return {}
|
||||
|
||||
|
||||
def ctl_log(since, pat):
|
||||
return w.guest(f"docker logs --since {since} felhom-controller 2>&1 | grep -E '{pat}' | tail -60")
|
||||
|
||||
|
||||
def now():
|
||||
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())
|
||||
|
||||
|
||||
def look():
|
||||
st = w.stack(APP)
|
||||
w.say(f" app: state={st.get('state')} hold={st.get('hold_reason')!r} deployed={st.get('deployed')}")
|
||||
w.say(" compose image: " + w.guest(f"grep -E '^ +image:' {COMPOSE}").strip())
|
||||
w.say(" container: " + w.guest(f"docker ps -a --filter name=^/{APP}$ --format '{{{{.Image}}}} {{{{.Status}}}}'").strip())
|
||||
w.say(" copies: " + (w.guest("find /mnt/*/felhom-data/backups/restore-undo /opt/docker/felhom-data/backups/restore-undo -mindepth 2 -maxdepth 2 -type d 2>/dev/null | head").strip() or "(none)"))
|
||||
return st
|
||||
|
||||
|
||||
def set_image(img, label, full=False):
|
||||
if full:
|
||||
w.guest(f"sed -i -E 's#^( +image: )ghcr.io/kozea/radicale:[^ ]+$#\\1{img}#' {COMPOSE}")
|
||||
img = img.split("@")[0]
|
||||
else:
|
||||
w.guest(f"sed -i -E 's#^( +image: )ghcr.io/kozea/radicale:[0-9.]+#\\1{img}#' {COMPOSE}")
|
||||
c, d = w.ctl("POST", f"/api/stacks/{APP}/restart")
|
||||
w.say(f" [{label}] compose → {img}; Restart → {c} {str(d)[:120]}")
|
||||
for _ in range(40):
|
||||
time.sleep(3)
|
||||
if img in w.guest(f"docker ps --filter name=^/{APP}$ --format '{{{{.Image}}}}'"):
|
||||
break
|
||||
look()
|
||||
|
||||
|
||||
w.login()
|
||||
phase = sys.argv[1]
|
||||
st = state()
|
||||
if st.get("t1"): # the deploy's generated login lives only in the deploying process; the seed reads w.GENERATED
|
||||
w.GENERATED[APP] = {"RADICALE_USER": st["t1"]["user"], "RADICALE_PASSWORD": st["t1"]["pw"]}
|
||||
t_start = now()
|
||||
w.say(f"=== phase {phase} at {t_start}")
|
||||
w.say("controller image: " + w.guest("cat /etc/felhom-controller-image; docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'").strip())
|
||||
|
||||
if phase == "setup":
|
||||
assert w.deploy(APP, SUB), "deploy failed"
|
||||
t1 = fx.seed(w, SUB, w.say)
|
||||
assert t1, "seed T1 failed"
|
||||
st["t1"] = t1
|
||||
save(st)
|
||||
set_image(OLD, "setup")
|
||||
w.say(f" T1 readback at 3.8.0: {fx.verify(w, SUB, t1, w.say)}")
|
||||
w.say(" off-site toggle: " + str(form("/backup/offbox/toggle", {"app": APP, "enabled": "on"})[:2]))
|
||||
w.say(" off-site run: " + str(form("/backup/offbox/run", {})[:2]))
|
||||
t0 = time.time()
|
||||
while time.time() - t0 < 3600:
|
||||
time.sleep(15)
|
||||
if "manual off-box backup" in ctl_log(t_start, "off-box backup|offbox.*(completed|finished|failed)") or \
|
||||
re.search(r"(?i)off-?box (run|backup) (completed|finished|done)", ctl_log(t_start, "offbox|off-box")):
|
||||
break
|
||||
w.say(" off-site run log: " + ctl_log(t_start, "offbox|off-box|radicale").strip()[-3000:])
|
||||
set_image(NEW, "setup")
|
||||
w.say(f" T1 readback at 3.8.1: {fx.verify(w, SUB, t1, w.say)}")
|
||||
|
||||
elif phase in ("setup2", "setup3"):
|
||||
# The off-site run pushes recovery units; a fresh app has none until a backup run made one ("skipped (no unit)").
|
||||
# The household's „Mentés most" (whole box, the only backup press there is — R-648) makes radicale's unit at 3.8.0.
|
||||
look()
|
||||
c, d = w.ctl("POST", "/api/backup/run")
|
||||
w.say(f" local backup press → {c} {str(d)[:200]}")
|
||||
t0 = time.time()
|
||||
while time.time() - t0 < 2400:
|
||||
time.sleep(10)
|
||||
lg = ctl_log(t_start, r"App-data backup completed|\[backup\].*(failed|FAILED)|backup run (finished|complete)")
|
||||
if "App-data backup completed" in lg:
|
||||
break
|
||||
w.say(" local backup log: " + ctl_log(t_start, r"App-data backup completed|radicale").strip()[-2500:])
|
||||
w.say(" off-site run: " + str(form("/backup/offbox/run", {})[:2]))
|
||||
t1 = now()
|
||||
t0 = time.time()
|
||||
while time.time() - t0 < 2400:
|
||||
time.sleep(10)
|
||||
if re.search(r"\[offbox\] backup (OK|failed)", ctl_log(t1, r"\[offbox\] backup")):
|
||||
break
|
||||
w.say(" off-site run log: " + ctl_log(t1, r"\[offbox\] backup|radicale").strip()[-2500:])
|
||||
if phase == "setup2":
|
||||
set_image(NEW, "setup2")
|
||||
w.say(f" T1 readback: {fx.verify(w, SUB, st['t1'], w.say)}")
|
||||
w.say(" radicale's off-site snapshots: " + w.guest("ls -la /mnt/sys_drive/felhom-data/backups/primary/radicale 2>&1 | head -8"))
|
||||
|
||||
elif phase == "pull":
|
||||
w.say(" prepare: " + str(form("/backup/offbox/restore", {"app": APP, "mode": "full"})[:2]))
|
||||
w.say(" pull: " + str(form("/backup/offbox/restore", {"app": APP, "mode": "full", "confirm": "1"})[:2]))
|
||||
wait_op("pull")
|
||||
|
||||
elif phase == "h4":
|
||||
# the live definition moves to the TRUE 3.8.0 (its own digest) right before the restore — within one catalog-sync
|
||||
# tick — so the snapshot (3.8.1) and the live definition differ and the restore writes the snapshot's definition
|
||||
set_image(NEW_FULL, "h4", full=True) # T2 is seeded on 3.8.1 (09:05-09:08: the 401s were the harness's empty password in a new process, not 3.8.0)
|
||||
t2 = None
|
||||
for _ in range(6):
|
||||
time.sleep(10)
|
||||
t2 = fx.seed(w, SUB, w.say)
|
||||
if t2:
|
||||
break
|
||||
assert t2, "seed T2 failed"
|
||||
st["t2"] = t2
|
||||
save(st)
|
||||
set_image(OLD_FULL, "h4", full=True)
|
||||
time.sleep(15)
|
||||
look()
|
||||
w.guest(f"mkdir -p {FAULTS} && echo '0 1' > {FAULTS}/defwrite-after-compose && cat {FAULTS}/defwrite-after-compose")
|
||||
w.say(" armed: defwrite-after-compose = 0 1")
|
||||
w.say(" restore: " + str(form("/backup/offbox/reconstitute", {"app": APP, "confirm": "1"})[:2]))
|
||||
rec = wait_op("h4 restore")
|
||||
w.say(f" restore record: {json.dumps(rec)[:1200]}")
|
||||
w.say(" controller log:\n" + ctl_log(t_start, "faultinject|radicale|R-1256|definition"))
|
||||
look()
|
||||
w.say(f" T1 readback: {fx.verify(w, SUB, st['t1'], w.say)}")
|
||||
w.say(f" T2 readback: {fx.verify(w, SUB, t2, w.say)}")
|
||||
w.say(" fault file left: " + (w.guest(f"ls {FAULTS} 2>/dev/null").strip() or "(none)"))
|
||||
|
||||
elif phase == "d62":
|
||||
look()
|
||||
w.say(" restore (no fault): " + str(form("/backup/offbox/reconstitute", {"app": APP, "confirm": "1"})[:2]))
|
||||
rec = wait_op("d62 restore")
|
||||
# at once: the stable-start check releases the copy about a minute after a good restore (R-1270)
|
||||
page = w.page("/backups/restore")
|
||||
t_page = now()
|
||||
rows = re.findall(r'<details class="kept-delete" data-restore-copy="([^"]+)">(.*?)</details>', page, re.S)
|
||||
w.say(f" page at {t_page}: {len(rows)} deletable restore copy row(s); restore ok={(rec.get('last') or {}).get('ok')}")
|
||||
m = re.search(r'data-restore-copies="true">(.*?)</div>\s*<script>', page, re.S)
|
||||
if m:
|
||||
w.say(" list text: " + re.sub(r"\s+", " ", re.sub(r"<[^>]+>", " ", m.group(1))).strip()[:800])
|
||||
if rows:
|
||||
app_stamp = rows[0][0]
|
||||
app, stamp = app_stamp.split("/", 1)
|
||||
names = re.findall(r'<span class="app-row-name">([^<]+)</span>', page[:page.find('data-restore-copy="' + app_stamp)])
|
||||
typed = names[-1].strip() if names else app
|
||||
h, l, b = form("/backup/restore-copy/delete", {"app": app, "stamp": stamp, "confirm": typed + "x"}, accept_json=True)
|
||||
w.say(f" wrong name ({typed + 'x'!r}) → {h} {b.strip()[:300]}")
|
||||
h, l, b = form("/backup/restore-copy/delete", {"app": app, "stamp": stamp, "confirm": typed}, accept_json=True)
|
||||
w.say(f" right name ({typed!r}) → {h} {b.strip()[:300]}")
|
||||
look()
|
||||
page2 = w.page("/backups/restore")
|
||||
w.say(f" page after: rows={page2.count('data-restore-copy=')}")
|
||||
w.say(" controller log: " + ctl_log(t_start, "restore copy DELETED|restore-copy delete|restore_copy_deleted|Event push|released"))
|
||||
else:
|
||||
w.say(" NO deletable copy listed — the D62 page proof did not happen")
|
||||
rec2 = wait_op("settle", cap=5)
|
||||
w.say(f" T1 readback: {fx.verify(w, SUB, st['t1'], w.say)}")
|
||||
|
||||
elif phase == "h5":
|
||||
look()
|
||||
vols = w.guest(f"docker volume ls --format '{{{{.Name}}}}' | grep -E '^{APP}' | sort").split()
|
||||
w.say(f" volumes: {vols}")
|
||||
datav = vols[0]
|
||||
w.say(" blocker: " + w.guest(f"docker rm -f d63-blocker >/dev/null 2>&1; docker create --name d63-blocker -v {datav}:/x alpine true"))
|
||||
w.say(" premise (rm -f with the blocker): " + w.guest(f"docker volume rm -f {datav} 2>&1; echo rc=$?").strip())
|
||||
w.guest(f"mkdir -p {FAULTS} && echo '1 1' > {FAULTS}/stop-{APP}")
|
||||
w.say(f" armed: stop-{APP} = 1 1")
|
||||
w.say(" restore: " + str(form("/backup/offbox/reconstitute", {"app": APP, "confirm": "1"})[:2]))
|
||||
rec = wait_op("h5 restore")
|
||||
w.say(f" restore record: {json.dumps(rec)[:1500]}")
|
||||
w.say(" controller log:\n" + ctl_log(t_start, "faultinject|radicale|R-1256|put-back|put back|Failed to remove|hold"))
|
||||
st2 = look()
|
||||
app_page = w.page(f"/apps/{APP}")
|
||||
held = re.search(r'data-held="true">(.*?)</div>', app_page, re.S)
|
||||
w.say(" app page hold sentence: " + (re.sub(r"<[^>]+>", " ", held.group(1)).strip()[:600] if held else "(none)"))
|
||||
page = w.page("/backups/restore")
|
||||
w.say(f" restore page: button off={'data-restore-copy-off' in page}, needed line="
|
||||
f"{re.findall(r'data-restore-copy-needed=.true.>([^<]+)<', page)[:2]}, deletable rows={page.count('data-restore-copy=')}")
|
||||
w.say(" fault file left: " + (w.guest(f"ls {FAULTS} 2>/dev/null").strip() or "(none)"))
|
||||
|
||||
elif phase == "teardown":
|
||||
w.guest(f"docker rm -f d63-blocker >/dev/null 2>&1; rm -rf {FAULTS}")
|
||||
look()
|
||||
w.remove(APP)
|
||||
look()
|
||||
|
||||
open(os.path.join(EV, f"{phase}.log"), "a").write("\n".join(w.LOG) + "\n")
|
||||
@@ -0,0 +1,24 @@
|
||||
09:10:56 === phase d62 at 2026-10-29T08:10:56Z
|
||||
09:10:59 controller image: gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault Up 18 minutes (healthy)
|
||||
09:10:59 app: state=running hold=None deployed=True
|
||||
09:11:03 compose image: image: ghcr.io/kozea/radicale:3.8.0@sha256:0d60d544710bda0fa83bfdea685db731b6cbf74e12d7f22b4997e9cff3a50468
|
||||
09:11:06 container: ghcr.io/kozea/radicale:3.8.0 Up 45 seconds (healthy)
|
||||
09:11:09 copies: (none)
|
||||
09:11:09 restore (no fault): ('HTTP/2 302', ['location: /backups/restore/app?name=radicale&flash=flash.offbox.full_restore_started'])
|
||||
09:11:09 + 0s d62 restore: (True, 'offbox-reconstitute', None, '')
|
||||
09:12:12 + 64s d62 restore: (False, 'offbox-reconstitute', None, '')
|
||||
09:12:12 restore record: {"running": false, "op": "offbox-reconstitute", "stack": "radicale", "started_at": "2026-10-29T08:11:09.437135355Z", "last": {"op": "offbox-reconstitute", "stack": "radicale", "ok": true, "message": "A(z) radicale: 0 f\u00e1jl \u00e9s 1 adatk\u00f6tet vissza\u00e1ll\u00edtva (ment\u00e9s: 2026. 10. 29. 09:02) \u2014 az alkalmaz\u00e1s \u00fajraindult. Ennek az alkalmaz\u00e1snak nincs adatb\u00e1zisa.", "finished_at": "2026-10-29T08:12:11.98337803Z"}, "last_recent": true}
|
||||
09:12:13 app: state=running hold=None deployed=True
|
||||
09:12:16 compose image: image: ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c
|
||||
09:12:19 container: ghcr.io/kozea/radicale:3.8.1 Up About a minute (healthy)
|
||||
09:12:22 copies: (none)
|
||||
09:12:58 page: 1 deletable restore copy row(s)
|
||||
09:12:58 list text: Radicale visszaállítás: 2026. 10. 29. 09:11 · 37,3 KB · automatikusan törlődik: 2026. 11. 28. Törlés most Ez a(z) Radicale adatainak másolata a 2026. 10. 29-i visszaállítás előttről. Ha törlöd, a visszaállítás többé nem vonható vissza, és ez a másolat nem hozható vissza. A törléshez írd be: Radicale Törlés most
|
||||
09:12:58 warning shown: Ez a(z) Radicale adatainak másolata a 2026. 10. 29-i visszaállítás előttről. Ha törlöd, a visszaállítás többé nem vonható vissza, és ez a másolat nem hozható vissza.
|
||||
09:12:58 wrong name → HTTP/2 400
|
||||
09:12:58 right name ('sonarr') → HTTP/2 400
|
||||
09:13:02 copy after: (gone)
|
||||
09:13:05 controller log: 2026/10/29 08:12:58 [WARN] [web] restore-copy delete REFUSED for /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081111Z: the typed confirmation did not match
|
||||
2026/10/29 08:12:58 [WARN] [web] restore-copy delete REFUSED for /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081111Z: the typed confirmation did not match
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
09:13:34 === phase d62 at 2026-10-29T08:13:34Z
|
||||
09:13:37 controller image: gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault Up 20 minutes (healthy)
|
||||
09:13:37 app: state=running hold=None deployed=True
|
||||
09:13:40 compose image: image: ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c
|
||||
09:13:43 container: ghcr.io/kozea/radicale:3.8.1 Up 2 minutes (healthy)
|
||||
09:13:46 copies: /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081111Z
|
||||
09:13:46 restore (no fault): ('HTTP/2 302', ['location: /backups/restore/app?name=radicale&flash=flash.offbox.full_restore_started'])
|
||||
09:13:46 + 0s d62 restore: (True, 'offbox-reconstitute', None, '')
|
||||
09:14:50 + 64s d62 restore: (False, 'offbox-reconstitute', None, '')
|
||||
09:15:32 page at 2026-10-29T08:15:32Z: 2 deletable restore copy row(s); restore ok=True
|
||||
09:15:32 list text: Radicale visszaállítás: 2026. 10. 29. 09:11 · 37,3 KB · automatikusan törlődik: 2026. 11. 28. Törlés most Ez a(z) Radicale adatainak másolata a 2026. 10. 29-i visszaállítás előttről. Ha törlöd, a visszaállítás többé nem vonható vissza, és ez a másolat nem hozható vissza. A törléshez írd be: Radicale Törlés most Radicale visszaállítás: 2026. 10. 29. 09:13 · 29,3 KB · automatikusan törlődik: 2026. 11. 28. Törlés most Ez a(z) Radicale adatainak másolata a 2026. 10. 29-i visszaállítás előttről. Ha törlöd, a visszaállítás többé nem vonható vissza, és ez a másolat nem hozható vissza. A törléshez írd be: Radicale Törlés most
|
||||
09:15:32 wrong name ('Radicalex') → HTTP/2 400 {"error":"A beírt név nem egyezik (Radicale), ezért nem töröltünk semmit.","ok":false,"redirect":""}
|
||||
09:15:32 right name ('Radicale') → HTTP/2 200 {"error":"","ok":true,"redirect":"/backups/restore?fa=Radicale\u0026flash=flash.restore_copy.deleted"}
|
||||
09:15:32 app: state=running hold=None deployed=True
|
||||
09:15:35 compose image: image: ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c
|
||||
09:15:38 container: ghcr.io/kozea/radicale:3.8.1 Up About a minute (healthy)
|
||||
09:15:41 copies: /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081348Z
|
||||
09:16:38 page after: rows=1
|
||||
09:16:41 controller log: 2026/10/29 08:15:32 [WARN] [web] restore-copy delete REFUSED for /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081111Z: the typed confirmation did not match
|
||||
2026/10/29 08:15:32 [WARN] [web] restore copy DELETED by hand (D62): app=radicale restore_at=2026-10-29T08:11:11Z size=38243 path=/mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081111Z who=the household (dashboard sign-in) from 192.168.0.180
|
||||
2026/10/29 08:15:38 [WARN] Event push failed after 3 attempts (restore_copy_deleted/info): HTTP 400
|
||||
|
||||
09:16:41 + 0s settle: (False, 'offbox-reconstitute', None, '')
|
||||
09:16:50 radicale: readback http=200 found=True (controls: no auth 401, wrong 401, absent 404)
|
||||
09:16:50 T1 readback: True
|
||||
@@ -0,0 +1,14 @@
|
||||
09:04:12 === phase h4 at 2026-10-29T08:04:12Z
|
||||
09:04:15 controller image: gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault Up 11 minutes (healthy)
|
||||
09:04:27 [h4] compose → ghcr.io/kozea/radicale:3.8.0; Restart → 200 {'ok': True, 'data': {'state': 'starting'}, 'message': 'Stack radicale restart requested — state now: starting'}
|
||||
09:04:34 app: state=starting hold=None deployed=True
|
||||
09:04:37 compose image: image: ghcr.io/kozea/radicale:3.8.0@sha256:0d60d544710bda0fa83bfdea685db731b6cbf74e12d7f22b4997e9cff3a50468
|
||||
09:04:40 container: ghcr.io/kozea/radicale:3.8.0 Up 12 seconds (healthy)
|
||||
09:04:43 copies: (none)
|
||||
09:04:45 radicale: MKCALENDAR /csalad/felhom-a718ed00/ http=401
|
||||
Traceback (most recent call last):
|
||||
File "/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0b76c4d7-9b39-4e4e-869d-d0d2c6006331/scratchpad/d63/d63_live.py", line 176, in <module>
|
||||
assert t2, "seed T2 failed"
|
||||
^^
|
||||
AssertionError: seed T2 failed
|
||||
@@ -0,0 +1,19 @@
|
||||
09:05:09 === phase h4 at 2026-10-29T08:05:09Z
|
||||
09:05:12 controller image: gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault Up 12 minutes (healthy)
|
||||
09:05:17 [h4] compose → ghcr.io/kozea/radicale:3.8.0; Restart → 200 {'ok': True, 'data': {'state': 'starting'}, 'message': 'Stack radicale restart requested — state now: starting'}
|
||||
09:05:23 app: state=starting hold=None deployed=True
|
||||
09:05:26 compose image: image: ghcr.io/kozea/radicale:3.8.0@sha256:0d60d544710bda0fa83bfdea685db731b6cbf74e12d7f22b4997e9cff3a50468
|
||||
09:05:29 container: ghcr.io/kozea/radicale:3.8.0 Up 12 seconds (healthy)
|
||||
09:05:33 copies: (none)
|
||||
09:05:44 radicale: MKCALENDAR /csalad/felhom-5424c15e/ http=401
|
||||
09:05:56 radicale: MKCALENDAR /csalad/felhom-e4dc5c62/ http=401
|
||||
09:06:07 radicale: MKCALENDAR /csalad/felhom-561ce5bf/ http=401
|
||||
09:06:19 radicale: MKCALENDAR /csalad/felhom-f49d45b0/ http=401
|
||||
09:06:30 radicale: MKCALENDAR /csalad/felhom-258756e5/ http=401
|
||||
09:06:41 radicale: MKCALENDAR /csalad/felhom-dc019255/ http=401
|
||||
Traceback (most recent call last):
|
||||
File "/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0b76c4d7-9b39-4e4e-869d-d0d2c6006331/scratchpad/d63/d63_live.py", line 181, in <module>
|
||||
assert t2, "seed T2 failed"
|
||||
^^
|
||||
AssertionError: seed T2 failed
|
||||
@@ -0,0 +1,19 @@
|
||||
09:07:01 === phase h4 at 2026-10-29T08:07:01Z
|
||||
09:07:04 controller image: gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault Up 14 minutes (healthy)
|
||||
09:07:08 [h4] compose → ghcr.io/kozea/radicale:3.8.1; Restart → 200 {'ok': True, 'data': {'state': 'starting'}, 'message': 'Stack radicale restart requested — state now: starting'}
|
||||
09:07:14 app: state=starting hold=None deployed=True
|
||||
09:07:17 compose image: image: ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c
|
||||
09:07:21 container: ghcr.io/kozea/radicale:3.8.1 Up 11 seconds (healthy)
|
||||
09:07:24 copies: (none)
|
||||
09:07:35 radicale: MKCALENDAR /csalad/felhom-4d229933/ http=401
|
||||
09:07:46 radicale: MKCALENDAR /csalad/felhom-091d495a/ http=401
|
||||
09:07:57 radicale: MKCALENDAR /csalad/felhom-a07bdb98/ http=401
|
||||
09:08:08 radicale: MKCALENDAR /csalad/felhom-f69aaadf/ http=401
|
||||
09:08:20 radicale: MKCALENDAR /csalad/felhom-576f2139/ http=401
|
||||
09:08:32 radicale: MKCALENDAR /csalad/felhom-a26a7f40/ http=401
|
||||
Traceback (most recent call last):
|
||||
File "/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0b76c4d7-9b39-4e4e-869d-d0d2c6006331/scratchpad/d63/d63_live.py", line 182, in <module>
|
||||
assert t2, "seed T2 failed"
|
||||
^^
|
||||
AssertionError: seed T2 failed
|
||||
@@ -0,0 +1,71 @@
|
||||
09:08:54 === phase h4 at 2026-10-29T08:08:54Z
|
||||
09:08:58 controller image: gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault Up 15 minutes (healthy)
|
||||
09:09:02 [h4] compose → ghcr.io/kozea/radicale:3.8.1; Restart → 200 {'ok': True, 'data': {'state': 'starting'}, 'message': 'Stack radicale restart requested — state now: starting'}
|
||||
09:09:08 app: state=starting hold=None deployed=True
|
||||
09:09:11 compose image: image: ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c
|
||||
09:09:14 container: ghcr.io/kozea/radicale:3.8.1 Up 11 seconds (healthy)
|
||||
09:09:17 copies: (none)
|
||||
09:09:28 radicale: MKCALENDAR /csalad/felhom-749ed888/ http=201
|
||||
09:09:28 radicale: PUT event http=201
|
||||
09:09:33 [h4] compose → ghcr.io/kozea/radicale:3.8.0; Restart → 200 {'ok': True, 'data': {'state': 'starting'}, 'message': 'Stack radicale restart requested — state now: starting'}
|
||||
09:09:39 app: state=running hold=None deployed=True
|
||||
09:09:42 compose image: image: ghcr.io/kozea/radicale:3.8.0@sha256:0d60d544710bda0fa83bfdea685db731b6cbf74e12d7f22b4997e9cff3a50468
|
||||
09:09:45 container: ghcr.io/kozea/radicale:3.8.0 Up 11 seconds (healthy)
|
||||
09:09:48 copies: (none)
|
||||
09:10:03 app: state=running hold=None deployed=True
|
||||
09:10:07 compose image: image: ghcr.io/kozea/radicale:3.8.0@sha256:0d60d544710bda0fa83bfdea685db731b6cbf74e12d7f22b4997e9cff3a50468
|
||||
09:10:10 container: ghcr.io/kozea/radicale:3.8.0 Up 36 seconds (healthy)
|
||||
09:10:13 copies: (none)
|
||||
09:10:16 armed: defwrite-after-compose = 0 1
|
||||
09:10:16 restore: ('HTTP/2 302', ['location: /backups/restore/app?name=radicale&flash=flash.offbox.full_restore_started'])
|
||||
09:10:16 + 0s h4 restore: (True, 'offbox-reconstitute', None, '')
|
||||
09:10:22 + 6s h4 restore: (False, 'offbox-reconstitute', None, '')
|
||||
09:10:22 restore record: {"running": false, "op": "offbox-reconstitute", "stack": "radicale", "started_at": "2026-10-29T08:10:16.87791986Z", "last": {"op": "offbox-reconstitute", "stack": "radicale", "ok": false, "message": "A teljes vissza\u00e1ll\u00edt\u00e1s sikertelen: restoring radicale: writing the snapshot's definition failed: FAULT INJECTED at defwrite-after-compose (D63 test build)", "finished_at": "2026-10-29T08:10:20.32238992Z"}, "last_recent": true}
|
||||
09:10:26 controller log:
|
||||
2026/10/29 08:09:01 [INFO] [api] restart requested for stack: radicale
|
||||
2026/10/29 08:09:01 [INFO] [stacks] Restarting stack: radicale
|
||||
2026/10/29 08:09:02 [INFO] [stacks] Stack radicale: started again: radicale 2026-10-29T08:07:08.330110078Z → 2026-10-29T08:09:01.852817156Z
|
||||
2026/10/29 08:09:02 [INFO] [stacks] Stack radicale restarted successfully (took 1.1s)
|
||||
2026/10/29 08:09:05 [INFO] [stacks] Stack radicale post-start status:
|
||||
2026/10/29 08:09:05 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 3 seconds (health: starting)
|
||||
2026/10/29 08:09:32 [INFO] [api] restart requested for stack: radicale
|
||||
2026/10/29 08:09:32 [INFO] [stacks] Restarting stack: radicale
|
||||
2026/10/29 08:09:33 [INFO] [stacks] Stack radicale: started again: radicale 2026-10-29T08:09:01.852817156Z → 2026-10-29T08:09:32.950147579Z
|
||||
2026/10/29 08:09:33 [INFO] [stacks] Stack radicale restarted successfully (took 0.8s)
|
||||
2026/10/29 08:09:33 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 0 encrypted, 1 sensitive fields
|
||||
2026/10/29 08:09:33 [INFO] [stacks] installed-images radicale: recorded 1 service(s) (radicale=ghcr.io/kozea/radicale:3.8.0 (sha256:0d60d544710b…))
|
||||
2026/10/29 08:09:33 [INFO] [stacks] SaveAppConfig: saved config for radicale
|
||||
2026/10/29 08:09:36 [INFO] [stacks] Stack radicale post-start status:
|
||||
2026/10/29 08:09:36 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.0@sha256:0d60d544710bda0fa83bfdea685db731b6cbf74e12d7f22b4997e9cff3a50468 running Up 3 seconds (health: starting)
|
||||
2026/10/29 08:10:18 [INFO] [offbox] radicale: snapshot 78002a51 holds data of [ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c] (written 2026-10-29T08:03:01Z); the app runs [ghcr.io/kozea/radicale:3.8.0@sha256:0d60d544710bda0fa83bfdea685db731b6cbf74e12d7f22b4997e9cff3a50468] — it comes back at the snapshot's version
|
||||
2026/10/29 08:10:19 [INFO] [stacks] Stopping stack: radicale
|
||||
2026/10/29 08:10:19 [INFO] [stacks] Stack radicale stopped successfully (took 0.3s)
|
||||
2026/10/29 08:10:19 [INFO] [offbox] radicale: the live volumes are copied before the restore (1 copied, 0 new from the snapshot) in 332ms — /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081019Z (R-893)
|
||||
2026/10/29 08:10:19 [INFO] [backup] radicale: the restore copy is recorded — a power cut from here on puts the app back on start (R-1195)
|
||||
2026/10/29 08:10:19 [ERROR] [faultinject] defwrite-after-compose: FAULT INJECTED (fail left 0)
|
||||
2026/10/29 08:10:19 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 1 encrypted, 1 sensitive fields
|
||||
2026/10/29 08:10:19 [INFO] [stacks] Redeploying radicale from recovery unit with 4 env vars
|
||||
2026/10/29 08:10:19 [INFO] [stacks] SaveAppConfig: saved config for radicale
|
||||
2026/10/29 08:10:19 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 0 encrypted, 1 sensitive fields
|
||||
2026/10/29 08:10:19 [INFO] [stacks] SaveAppConfig: saved config for radicale
|
||||
2026/10/29 08:10:19 [INFO] [stacks] pin radicale: radicale=ghcr.io/kozea/radicale:3.8.0
|
||||
2026/10/29 08:10:19 [INFO] [offbox] radicale: the definition write failed (FAULT INJECTED at defwrite-after-compose (D63 test build)); the live definition is written back before the start
|
||||
2026/10/29 08:10:19 [INFO] [stacks] Starting stack: radicale
|
||||
2026/10/29 08:10:20 [INFO] [stacks] Stack radicale started successfully (took 0.3s)
|
||||
2026/10/29 08:10:20 [DEBUG] [stacks] SaveAppConfig: saving /opt/docker/stacks/radicale — 4 env vars, 0 encrypted, 1 sensitive fields
|
||||
2026/10/29 08:10:20 [INFO] [stacks] installed-images radicale: recorded 1 service(s) (radicale=ghcr.io/kozea/radicale:3.8.0 (sha256:0d60d544710b…))
|
||||
2026/10/29 08:10:20 [INFO] [stacks] SaveAppConfig: saved config for radicale
|
||||
2026/10/29 08:10:20 [ERROR] [web] off-box reconstitute radicale (async): restoring radicale: writing the snapshot's definition failed: FAULT INJECTED at defwrite-after-compose (D63 test build)
|
||||
2026/10/29 08:10:23 [INFO] [stacks] Stack radicale post-start status:
|
||||
2026/10/29 08:10:23 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.0@sha256:0d60d544710bda0fa83bfdea685db731b6cbf74e12d7f22b4997e9cff3a50468 running Up 3 seconds (health: starting)
|
||||
|
||||
09:10:26 app: state=starting hold=None deployed=True
|
||||
09:10:29 compose image: image: ghcr.io/kozea/radicale:3.8.0@sha256:0d60d544710bda0fa83bfdea685db731b6cbf74e12d7f22b4997e9cff3a50468
|
||||
09:10:33 container: ghcr.io/kozea/radicale:3.8.0 Up 11 seconds (healthy)
|
||||
09:10:36 copies: (none)
|
||||
09:10:38 radicale: readback http=200 found=True (controls: no auth 401, wrong 401, absent 404)
|
||||
09:10:38 T1 readback: True
|
||||
09:10:42 radicale: readback http=200 found=True (controls: no auth 401, wrong 401, absent 404)
|
||||
09:10:42 T2 readback: True
|
||||
09:10:45 fault file left: (none)
|
||||
@@ -0,0 +1,42 @@
|
||||
09:18:10 === phase h5 at 2026-10-29T08:18:10Z
|
||||
09:18:13 controller image: gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault Up 25 minutes (healthy)
|
||||
09:18:13 app: state=running hold=None deployed=True
|
||||
09:18:16 compose image: image: ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c
|
||||
09:18:19 container: ghcr.io/kozea/radicale:3.8.1 Up 4 minutes (healthy)
|
||||
09:18:23 copies: (none)
|
||||
09:18:26 volumes: ['radicale_radicale_data']
|
||||
09:18:29 blocker: bcea23576117066b1fbd5e4f1c0ce85c9ce3784f26e361ea1c8454c3f0bb24b7
|
||||
|
||||
09:18:32 premise (rm -f with the blocker): Error response from daemon: remove radicale_radicale_data: volume is in use - [68fba54d540a68f2b3b45988d919d05a8b974388f4accd6825129acfec7f4e6d, bcea23576117066b1fbd5e4f1c0ce85c9ce3784f26e361ea1c8454c3f0bb24b7]
|
||||
rc=1
|
||||
09:18:35 armed: stop-radicale = 1 1
|
||||
09:18:35 restore: ('HTTP/2 302', ['location: /backups/restore/app?name=radicale&flash=flash.offbox.full_restore_started'])
|
||||
09:18:35 + 0s h5 restore: (True, 'offbox-reconstitute', None, '')
|
||||
09:18:41 + 6s h5 restore: (False, 'offbox-reconstitute', None, '')
|
||||
09:18:41 restore record: {"running": false, "op": "offbox-reconstitute", "stack": "radicale", "started_at": "2026-10-29T08:18:35.733794285Z", "last": {"op": "offbox-reconstitute", "stack": "radicale", "ok": false, "message": "A teljes vissza\u00e1ll\u00edt\u00e1s sikertelen: a(z) radicale vissza\u00e1ll\u00edt\u00e1sa f\u00e9l\u00faton megszakadt. Az alkalmaz\u00e1s egyes f\u00e1jljai vagy adatai m\u00e1r a ment\u00e9sb\u0151l sz\u00e1rmaznak, m\u00e1sok nem, ez\u00e9rt nem illenek \u00f6ssze. Az alkalmaz\u00e1st biztons\u00e1gi okb\u00f3l LE\u00c1LL\u00cdTVA hagytuk. Ehhez seg\u00edts\u00e9g kell: vedd fel vel\u00fcnk a kapcsolatot", "finished_at": "2026-10-29T08:18:38.916515438Z"}, "last_recent": true}
|
||||
09:18:44 controller log:
|
||||
2026/10/29 08:18:38 [WARN] [faultinject] stop-radicale: passed (skip left 0, fail left 1)
|
||||
2026/10/29 08:18:38 [INFO] [stacks] Stopping stack: radicale
|
||||
2026/10/29 08:18:38 [INFO] [stacks] Stack radicale stopped successfully (took 0.3s)
|
||||
2026/10/29 08:18:38 [INFO] [offbox] radicale: the live volumes are copied before the restore (1 copied, 0 new from the snapshot) in 345ms — /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081838Z (R-893)
|
||||
2026/10/29 08:18:38 [INFO] [backup] radicale: the restore copy is recorded — a power cut from here on puts the app back on start (R-1195)
|
||||
2026/10/29 08:18:38 [INFO] [backup] Restoring Docker volume radicale_radicale_data for radicale
|
||||
2026/10/29 08:18:38 [ERROR] [backup] Failed to remove volume radicale_radicale_data before the restore: Error response from daemon: remove radicale_radicale_data: volume is in use - [bcea23576117066b1fbd5e4f1c0ce85c9ce3784f26e361ea1c8454c3f0bb24b7] — exit status 1 (not written into)
|
||||
2026/10/29 08:18:38 [ERROR] [offbox] radicale: stopping the app before the put-back FAILED: FAULT INJECTED at stop-radicale (D63 test build) — nothing is put back; the copy stays at /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081838Z; holding the app
|
||||
2026/10/29 08:18:38 [WARN] [settings] restore hold SET for radicale — the app stays stopped until it is cleared
|
||||
2026/10/29 08:18:38 [ERROR] [faultinject] stop-radicale: FAULT INJECTED (fail left 0)
|
||||
2026/10/29 08:18:38 [INFO] [stacks] Stopping stack: radicale
|
||||
2026/10/29 08:18:38 [INFO] [stacks] Stack radicale stopped successfully (took 0.1s)
|
||||
2026/10/29 08:18:38 [ERROR] [offbox] radicale: the restore stopped half-way after files/volumes/version had moved — HOLDING the app stopped for support (R-893); error was: failed to restore 1 volume(s): [radicale_radicale_data]
|
||||
2026/10/29 08:18:38 [WARN] [offbox] radicale: the restore's volume copy is KEPT for support at /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081838Z
|
||||
2026/10/29 08:18:38 [ERROR] [web] off-box reconstitute radicale (async): a(z) radicale visszaállítása félúton megszakadt. Az alkalmazás egyes fájljai vagy adatai már a mentésből származnak, mások nem, ezért nem illenek össze. Az alkalmazást biztonsági okból LEÁLLÍTVA hagytuk. Ehhez segítség kell: vedd fel velünk a kapcsolatot
|
||||
2026/10/29 08:18:39 [INFO] Event pushed: backup_run_failures (error) [hu-only] — App "radicale" is HELD STOPPED for support: its off-site database restore failed. The database was rolled back to the pre-restore copy and the live definition written back, but its files and named volumes are the snapshot's — a mixed state (R-893). The app will not start from any path until the hold is cleared. Replay error: failed to restore 1 volume(s): [radicale_radicale_data]
|
||||
|
||||
09:18:44 app: state=stopped hold='a(z) radicale visszaállítása 2026-10-29 08:18-kor félúton megszakadt: az adatai részben a mentésből származnak, részben nem. Az alkalmazás biztonsági okból leállítva marad. Ehhez segítség kell: vedd fel velünk a kapcsolatot' deployed=True
|
||||
09:18:47 compose image: image: ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c
|
||||
09:18:51 container:
|
||||
09:18:54 copies: /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081838Z
|
||||
09:18:54 app page hold sentence: a(z) radicale visszaállítása 2026-10-29 08:18-kor félúton megszakadt: az adatai részben a mentésből származnak, részben nem. Az alkalmazás biztonsági okból leállítva marad. Ehhez segítség kell: vedd fel velünk a kapcsolatot Mentések
|
||||
09:19:02 restore page: button off=True, needed line=['Nem törölhető: az alkalmazás le van állítva, és ez a másolat az út vissza. Az ügyfélszolgálat dönt róla.'], deletable rows=0
|
||||
09:19:05 fault file left: (none)
|
||||
@@ -0,0 +1,7 @@
|
||||
09:03:51 === phase pull at 2026-10-29T08:03:51Z
|
||||
09:03:54 controller image: gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault Up 10 minutes (healthy)
|
||||
09:03:57 prepare: ('HTTP/2 302', ['location: /backups/restore/app?name=radicale&full_prep=radicale&full_size=31.3+KB'])
|
||||
09:03:57 pull: ('HTTP/2 302', ['location: /backups/restore/app?name=radicale&flash=flash.offbox.restore_started'])
|
||||
09:03:57 + 0s pull: (True, 'offbox-restore', None, '')
|
||||
09:04:06 + 9s pull: (False, 'offbox-restore', None, '')
|
||||
@@ -0,0 +1,17 @@
|
||||
08:53:55 === phase setup at 2026-10-29T07:53:55Z
|
||||
08:53:58 controller image: gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault Up 59 seconds (healthy)
|
||||
08:53:58 [1] required fields filled beyond DOMAIN/SUBDOMAIN: ['RADICALE_USER', 'RADICALE_PASSWORD']
|
||||
08:53:58 [1] deploy -> 202 {'ok': True, 'message': 'Telepítés elindítva – az állapot a kártyán követhető'}
|
||||
08:54:23 [1] deployed, controller state=running, pinned={'radicale': 'ghcr.io/kozea/radicale:3.8.1'}
|
||||
08:54:24 radicale: MKCALENDAR /csalad/felhom-e10daea4/ http=201
|
||||
08:54:24 radicale: PUT event http=201
|
||||
08:54:29 [setup] compose → ghcr.io/kozea/radicale:3.8.0; Restart → 200 {'ok': True, 'data': {'state': 'starting'}, 'message': 'Stack radicale restart requested — state now: starting'}
|
||||
08:54:35 app: state=starting hold=None deployed=True
|
||||
08:54:38 compose image: image: ghcr.io/kozea/radicale:3.8.0@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c
|
||||
08:54:41 container: ghcr.io/kozea/radicale:3.8.0 Up 11 seconds (healthy)
|
||||
08:54:44 copies: (none)
|
||||
08:54:48 radicale: readback http=200 found=True (controls: no auth 401, wrong 401, absent 404)
|
||||
08:54:48 T1 readback at 3.8.0: True
|
||||
08:54:48 off-site toggle: ('HTTP/2 302', ['location: /backups/remote?flash=flash.offbox.app_setting_updated'])
|
||||
08:54:48 off-site run: ('HTTP/2 302', ['location: /backups/remote?flash=flash.offbox.run_started'])
|
||||
@@ -0,0 +1,28 @@
|
||||
08:58:30 === phase setup2 at 2026-10-29T07:58:30Z
|
||||
08:58:34 controller image: gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault Up 5 minutes (healthy)
|
||||
08:58:34 app: state=running hold=None deployed=True
|
||||
08:58:37 compose image: image: ghcr.io/kozea/radicale:3.8.0@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c
|
||||
08:58:40 container: ghcr.io/kozea/radicale:3.8.0 Up 4 minutes (healthy)
|
||||
08:58:43 copies: (none)
|
||||
08:58:43 local backup press → 200 {'ok': True, 'message': 'Mentés elindítva'}
|
||||
08:59:28 local backup log: 2026/10/29 07:59:16 [WARN] [backup] Skipping volume dump for radicale — it is not running its pinned version (radicale runs ghcr.io/kozea/radicale:3.8.0, pinned ghcr.io/kozea/radicale:3.8.1); the last good backup is kept (R-645)
|
||||
2026/10/29 07:59:16 [INFO] [backup] App-data backup completed: 2 databases (532.5 KB total), 3 volume dump(s) (33.493s)
|
||||
2026/10/29 07:59:16 [WARN] [backup] Recovery unit NOT captured for radicale — it is not running its pinned version (radicale runs ghcr.io/kozea/radicale:3.8.0, pinned ghcr.io/kozea/radicale:3.8.1); the last good unit is kept (R-645)
|
||||
2026/10/29 07:59:16 [ERROR] [backup] radicale is RUNNING but its newest local copy (/mnt/sys_drive/felhom-data/backups/primary/radicale) holds NO database dump and NO volume tar — it is not protected (Part D)
|
||||
08:59:28 off-site run: ('HTTP/2 302', ['location: /backups/remote?flash=flash.offbox.run_started'])
|
||||
09:00:41 off-site run log: 2026/10/29 07:59:28 [INFO] [offbox] backup run started (10 app(s) toggled)
|
||||
2026/10/29 08:00:07 [WARN] [backup] Skipping volume dump for radicale — it is not running its pinned version (radicale runs ghcr.io/kozea/radicale:3.8.0, pinned ghcr.io/kozea/radicale:3.8.1); the last good backup is kept (R-645)
|
||||
2026/10/29 08:00:07 [WARN] [backup] Recovery unit NOT captured for radicale — it is not running its pinned version (radicale runs ghcr.io/kozea/radicale:3.8.0, pinned ghcr.io/kozea/radicale:3.8.1); the last good unit is kept (R-645)
|
||||
2026/10/29 08:00:07 [ERROR] [backup] radicale is RUNNING but its newest local copy (/mnt/sys_drive/felhom-data/backups/primary/radicale) holds NO database dump and NO volume tar — it is not protected (Part D)
|
||||
2026/10/29 08:00:13 [WARN] [offbox] radicale: no recovery unit found on any connected drive — skipping
|
||||
2026/10/29 08:00:25 [INFO] [offbox] backup OK: 4 app(s) backed up, 6 skipped (no unit), 77 snapshot(s), 53s
|
||||
2026/10/29 08:00:25 [INFO] Event pushed: backup_run_failures (error) [hu-only] — Offsite backup INCOMPLETE: 1 app(s) had a directory they declare MANDATORY missing from the snapshot — radicale ((a teljes alkalmazás — nincs helyi mentési egysége)). The run itself succeeded and what it captured is real, but these apps are NOT fully protected off-site. Check that the declared path exists on disk and that the app's data root resolves where the backup looks (R-203).
|
||||
09:00:45 [setup2] compose → ghcr.io/kozea/radicale:3.8.1; Restart → 200 {'ok': True, 'data': {'state': 'starting'}, 'message': 'Stack radicale restart requested — state now: starting'}
|
||||
09:00:51 app: state=starting hold=None deployed=True
|
||||
09:00:54 compose image: image: ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c
|
||||
09:00:58 container: ghcr.io/kozea/radicale:3.8.1 Up 12 seconds (healthy)
|
||||
09:01:01 copies: (none)
|
||||
09:01:05 radicale: readback http=200 found=True (controls: no auth 401, wrong 401, absent 404)
|
||||
09:01:05 T1 readback at 3.8.1: True
|
||||
rc=0
|
||||
@@ -0,0 +1,50 @@
|
||||
09:01:24 === phase setup3 at 2026-10-29T08:01:24Z
|
||||
09:01:27 controller image: gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault Up 8 minutes (healthy)
|
||||
09:01:27 app: state=running hold=None deployed=True
|
||||
09:01:30 compose image: image: ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c
|
||||
09:01:33 container: ghcr.io/kozea/radicale:3.8.1 Up 48 seconds (healthy)
|
||||
09:01:37 copies: (none)
|
||||
09:01:37 local backup press → 200 {'ok': True, 'message': 'Mentés elindítva'}
|
||||
09:02:21 local backup log: 2026/10/29 08:02:11 [INFO] [backup] Stopping radicale for safe volume dump
|
||||
2026/10/29 08:02:11 [INFO] [stacks] Stopping stack: radicale
|
||||
2026/10/29 08:02:11 [INFO] [stacks] Stack radicale stopped successfully (took 0.4s)
|
||||
2026/10/29 08:02:11 [INFO] [backup] Volume dump: radicale/radicale_radicale_data → 12.5 KB
|
||||
2026/10/29 08:02:11 [INFO] [backup] Restarting radicale after volume dump
|
||||
2026/10/29 08:02:11 [INFO] [stacks] Starting stack: radicale
|
||||
2026/10/29 08:02:12 [INFO] [stacks] Stack radicale started successfully (took 0.4s)
|
||||
2026/10/29 08:02:12 [INFO] [backup] App-data backup completed: 2 databases (532.8 KB total), 4 volume dump(s) (35.916s)
|
||||
2026/10/29 08:02:13 [INFO] [backup] Recovery unit captured for radicale → /mnt/sys_drive/felhom-data/backups/primary/radicale (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1)
|
||||
2026/10/29 08:02:16 [INFO] [stacks] Stack radicale post-start status:
|
||||
2026/10/29 08:02:16 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 3 seconds (health: starting)
|
||||
09:02:21 off-site run: ('HTTP/2 302', ['location: /backups/remote?flash=flash.offbox.run_started'])
|
||||
09:03:36 off-site run log: 2026/10/29 08:02:21 [INFO] [offbox] backup run started (10 app(s) toggled)
|
||||
2026/10/29 08:02:59 [INFO] [backup] Stopping radicale for safe volume dump
|
||||
2026/10/29 08:02:59 [INFO] [stacks] Stopping stack: radicale
|
||||
2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml
|
||||
2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml
|
||||
2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml
|
||||
2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml
|
||||
2026/10/29 08:03:00 [INFO] [stacks] ParseComposeHDDMounts: found 0 HDD mounts for /opt/docker/stacks/radicale/docker-compose.yml
|
||||
2026/10/29 08:03:00 [INFO] [backup] Recovery unit captured for radicale → /mnt/sys_drive/felhom-data/backups/primary/radicale (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1)
|
||||
2026/10/29 08:03:00 [INFO] [stacks] Stack radicale stopped successfully (took 0.8s)
|
||||
2026/10/29 08:03:01 [INFO] [backup] Volume dump: radicale/radicale_radicale_data → 12.5 KB
|
||||
2026/10/29 08:03:01 [INFO] [backup] Restarting radicale after volume dump
|
||||
2026/10/29 08:03:01 [INFO] [stacks] Starting stack: radicale
|
||||
2026/10/29 08:03:02 [INFO] [stacks] Stack radicale started successfully (took 0.9s)
|
||||
2026/10/29 08:03:03 [INFO] [backup] Recovery unit captured for radicale → /mnt/sys_drive/felhom-data/backups/primary/radicale (images=1, secrets-referenced=1, data_keys=0, portable-carried=0/0, withheld=1)
|
||||
2026/10/29 08:03:06 [INFO] [stacks] Stack radicale post-start status:
|
||||
2026/10/29 08:03:06 [INFO] [stacks] radicale ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c running Up 4 seconds (health: starting)
|
||||
2026/10/29 08:03:17 [INFO] [offbox] backed up radicale (/mnt/sys_drive/felhom-data/backups/primary/radicale, 0 mandatory path(s))
|
||||
2026/10/29 08:03:23 [INFO] [offbox] backup OK: 5 app(s) backed up, 5 skipped (no unit), 82 snapshot(s), 58s
|
||||
09:03:39 radicale: readback http=200 found=True (controls: no auth 401, wrong 401, absent 404)
|
||||
09:03:39 T1 readback: True
|
||||
09:03:42 radicale's off-site snapshots: total 24
|
||||
drwxr-xr-x 4 root root 4096 Oct 29 08:03 .
|
||||
drwxr-xr-x 6 root root 4096 Oct 29 08:02 ..
|
||||
drwxr-xr-x 2 root root 4096 Oct 29 08:03 compose
|
||||
-rw-r--r-- 1 root root 391 Oct 29 08:03 data-stamps.json
|
||||
-rw-r--r-- 1 root root 1997 Oct 29 08:03 manifest.json
|
||||
drwxr-xr-x 2 root root 4096 Oct 29 08:03 volume-dumps
|
||||
|
||||
rc=0
|
||||
@@ -0,0 +1,14 @@
|
||||
09:19:40 === phase teardown at 2026-10-29T08:19:40Z
|
||||
09:19:43 controller image: gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.347.0-rc.1-d63fault Up 26 minutes (healthy)
|
||||
09:19:46 app: state=stopped hold='a(z) radicale visszaállítása 2026-10-29 08:18-kor félúton megszakadt: az adatai részben a mentésből származnak, részben nem. Az alkalmazás biztonsági okból leállítva marad. Ehhez segítség kell: vedd fel velünk a kapcsolatot' deployed=True
|
||||
09:19:49 compose image: image: ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c
|
||||
09:19:53 container:
|
||||
09:19:56 copies: /mnt/sys_drive/felhom-data/backups/restore-undo/radicale/20261029T081838Z
|
||||
09:19:56 [X] stop -> 200 {'ok': True, 'message': 'Stack radicale stop completed'}
|
||||
09:20:28 [X] remove (with drive data) -> 200 {'ok': True, 'data': {'removed': 'radicale', 'volumes_removed': ['radicale_radicale_data'], 'hdd_paths_removed': [], 'hdd_paths_preserved': [], 'hdd_note': 'Az
|
||||
09:20:37 [X] after remove: deployed=False leftovers=''
|
||||
09:20:37 app: state=not_deployed hold=None deployed=False
|
||||
09:20:40 compose image: image: ghcr.io/kozea/radicale:3.8.1@sha256:54d9406cd30f9e206a9dd6d61dfac48da26b9afc37200e14f23beb8fafa7d11c
|
||||
09:20:43 container:
|
||||
09:20:46 copies: (none)
|
||||
@@ -0,0 +1,17 @@
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.346.1 Up 50 seconds (healthy)
|
||||
Deleted: sha256:31eb2ef55bea2a1b8796d3e9b6e744550a38057b04ce7d195d29a2d92e118f9e
|
||||
bookstack lscr.io/linuxserver/bookstack:26.09.2
|
||||
bookstack-db mariadb:12.3
|
||||
cloudflared cloudflare/cloudflared:2026.10.4
|
||||
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.346.1
|
||||
filebrowser gtstef/filebrowser:1.5.8-stable
|
||||
paperless-postgres postgres:18-alpine
|
||||
paperless-redis redis:7-alpine
|
||||
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15
|
||||
privatebin privatebin/pdo:2.1.0
|
||||
traefik traefik:v3.7.16
|
||||
283
|
||||
12
|
||||
271
|
||||
3
|
||||
268
|
||||
@@ -0,0 +1,12 @@
|
||||
gitea.dooplex.hu/admin/felhom-controller:0.346.1
|
||||
bookstack lscr.io/linuxserver/bookstack:26.09.2
|
||||
bookstack-db mariadb:12.3
|
||||
cloudflared cloudflare/cloudflared:2026.10.4
|
||||
felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.346.1
|
||||
filebrowser gtstef/filebrowser:1.5.8-stable
|
||||
paperless-postgres postgres:18-alpine
|
||||
paperless-redis redis:7-alpine
|
||||
paperless-webserver ghcr.io/paperless-ngx/paperless-ngx:2.20.15
|
||||
privatebin privatebin/pdo:2.1.0
|
||||
traefik traefik:v3.7.16
|
||||
268
|
||||
@@ -154,6 +154,7 @@ stopping line that lies.
|
||||
| **R-1337** | Apps & catalog | P3 | **Calibre-Web (v4.0.8, the newest) loads a bad Brotli decoder: its `cps.py` maps Python Brotli 1.2.0 (fixed 1.2.1), Pillow's libbrotlidec 1.2.0 and Debian's libbrotlidec 1.1.0** — read from the process's maps on demo-hp 9201 (`audits/runtime-fixes-2026-10-28/calibre-web-brotli.txt`). Exposure: the Python decoder is urllib3's, for replies to the app's own outbound calls (book metadata), not for what a visitor sends — the Paperless shape (R-1331). No fixed release exists. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) | upstream: an image with Brotli ≥ 1.2.1 | take that release as a lane step; Debian's libbrotli1 fixed version is not known (R-1327's table has no Debian line) | CC |
|
||||
| **R-1338** | Apps & catalog | P3 | **Wanderer's Meilisearch fills whatever memory limit it gets: 92–100 % anon of 256 MB (bench and 9202), still 88.8 % of 384 MB on the bench (2026-10-28)** — it sizes its indexing memory from the memory it sees (its default: 2/3 of the available RAM), so a higher limit alone does not end the `memory_tight` mark of Wanderer's step. No kill in any run. `audits/dockerhub-bigsteps-2026-10-28/wanderer/` | OPEN — **2026-10-28 night: MEASURED — it is the measurement, not Wanderer, and the cap is not needed** (`audits/night-2026-10-28/r1338-meili/`; `09` §3 decision 22 refined). Meilisearch frees with MADV_FREE: the given-back pages stay in `anon` but sit on the inactive FILE list (`file` ~1 MB, `inactive_file` ~290 MB) and the kernel takes them first — 83–96 % `anon` on the bench (no swap) AND on 9202, with and without the cap, 0 kills; it uses ~100 MB. The evening's 17.7 % was the container after the swap, not the cap. **Fixed in the tools:** the mark reads the live anon (catalog `scripts/memfigure.py`, the larger of two estimates, errs high; `upgrade-test.py`, `box_walk.py` soak; `test_memory_live.py` red-proved). Re-run: bench Wanderer v0.20.0 → v0.21.2 `proven`, **no marks** (search 83.3 % anon, 27.7 % live); 9202 95.2 % anon, 25.1 % live. **Prepared, NOT pushed** (the night's catalog fence): the tool change + Wanderer's entry without `memory_tight` (`audits/night-2026-10-28/catalog-morning/`; the catalog gates passed in a scratch clone). **Earlier, 2026-10-28 evening: `MEILI_MAX_INDEXING_MEMORY` = 128 MiB (a third of 384 MB) measured, NOT pushed** (`audits/gitea-wanderer-2026-10-28/wanderer-meili/`): on the bench, under the harness's load, the search container peaked at 17.7 % (84 MB; 88.8 % without the cap, same load); on 9202 the cap was in the container (read back) and it still held 95.7 % anon (363 MB steady over 6 min, no kill). The box result decides: not shown to help, so not pushed. Also: the catalog's ladder has no entry type for a re-measure of a definition change at the SAME digests (`ladder.py`: a re-test whose digest equals its digest_from tests nothing new), so the v0.21.2 entry's `memory_tight` mark cannot be replaced without an image move. | the catalog push (the morning list) | push the prepared catalog change (the live figure + Wanderer's entry re-measured, `memory_tight` off — an existing entry's memory fields edited at the SAME digests, which the gates accept); then close | CC |
|
||||
| **R-1339** | Storage & devices | P4 | **At every start the controller logs „FileBrowser: drive …/userdata/<app> not mounted — skipping userdata skeleton" for per-app folders on a drive that IS mounted** (scratch 9202, 2026-10-29 03:35 UTC: nine lines, `scratch_hdd` a live mountpoint, mp8). `skipFileBrowserPath` (`controller/internal/web/handlers.go:3766`) asks whether the folder ITSELF is a mountpoint, and a folder under a mounted drive never is. No data is lost: the file browser shows the folders through the drive's own mount (`/srv/scratch_hdd`). The cost is a false „not mounted" in the log of a box whose drive works, which misleads a diagnosis. Fix: ask about the drive the folder lives on, with a test that a sub-folder of a mounted drive is not skipped. Evidence: `audits/night-2026-10-28/part5-claims/selfheal/controller-after-reboot.log`. | OPEN | — | the fix above, with its red-first test | CC |
|
||||
| **R-1340** | Backup & restore | P3 | **The operator's „HELD STOPPED" mail for a mixed restore says the DATABASE restore failed and was rolled back, and that the files and volumes are the snapshot's — whatever the cause.** Measured 2026-10-29 08:18 UTC on Tester 1 (D63's H5 live proof, test build): Radicale (no database) — the volume could not be removed (not written into), the put-back's stop failed, nothing moved; the mail read „its off-site database restore failed. The database was rolled back to the pre-restore copy and the live definition written back, but its files and named volumes are the snapshot's". The text (`controller/cmd/controller/main.go:1555`) predates the D20/D22 put-back, when the only mixed hold was R-893 option C. Support reads a wrong account of what is on the box. The household's sentence is conservative and stays. Fix: the operator text names the cause it was given (the replay error, the put-back step that failed, whether the copy is kept and where) instead of the option-C story; a test per cause. Evidence: `audits/d63-2026-10-29/run-h5.log`. | OPEN | — | the fix above, red first | CC |
|
||||
|
||||
|
||||
## App updates — 10 rows (P2 1, P3 9)
|
||||
@@ -187,7 +188,7 @@ stopping line that lies.
|
||||
| **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** **2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair. | — | — | CC |
|
||||
| **R-933** | Backup & restore | P3 | **Mirror the images of every catalog app we support into our own registry, before scale.** Operator, 2026-10-12 (S8, `09` §3 decision 203): *„Will need to mirror the apps we support later, just to be sure."* Today a restore re-pulls each image from its maker; a version the maker deleted cannot start (the accepted limit, `07` §6.6, R-698 closed). Size it first: the supported catalog's images (every ladder version a box can hold) × their size; then where the copy lives (the DooPlex registry is Tier 2, precious — the copy must not make DooPlex a single point of failure for every household's restore) and how a restore falls back to it. | **OPEN** — NEW 2026-10-12 | the size measurement | Measure the total size of the supported catalog's images; then a one-page design (where, how a restore falls back, cost) | CC (measure + design), operator (the go) |
|
||||
| **R-1255** | Backup & restore | P3 | **Escrow edge cases (review round 2, HUNT-3 H3/H4):** the R-1176 carry moves the newest kept package even when it holds no repository password (`hub/internal/store/escrow_carry.go:100-110` checks `identity_blob` only), while the right one stays under the deleted host (`store.go:3533-3534`); two set-asides on the same UTC day reuse a name and the purge can delete a still-stored copy's package (`hub/internal/offsitekeys/offsitekeys.go:378-385`; `store.go:4295-4300`). Static. | OPEN — **fixed on main 2026-10-28 night, not released** (felhom.eu `1ee6d5a5`, CI 2472; store tests red first; `03` §8 [DESIGN] 2026-10-28): the carry takes the newest retained row with an identity AND a repository-password hash (the newest row only when none has a hash — a legacy package, as before); a set-aside is `<repo>.orphaned-<YYYYMMDDTHHMMSSZ>`; a retained row is purged only when it was tagged no later than its path's deletion (`repo_tagged_at`, new column). No live proof: a hub deploy is not a night's. | the hub release (attended) | release it (the morning list), read the store back after one carry and one abandonment, then close | CC |
|
||||
| **R-1256** | Backup & restore | P3 | **Restore edge cases (review round 2, HUNT-1 H4/H5/H6/H8):** (H4) an off-site restore whose definition write fails AFTER the compose was written restarts the app on the snapshot's compose (`internal/backup/offbox_reconstitute.go:979-986`; `cmd/controller/main.go:3318-3331`); (H5) a put-back whose stop fails goes on and can merge (`restore_crash_putback.go:155-158`; `restore.go:218` ignores `volume rm -f`), then removes the copy; (H6) a second-drive mirror can start between the web refusal and the restore's own record (`web/restore_wizard.go:226`; `opstatus.go:56-66`), and the install-time kept-data load checks only RestoreStatus/IsRunning (`api/kept_install.go:89-92`); (H8) an unused restore copy is never cleaned up. Static. **2026-10-28 night:** all four fixed, each test red first (controller `342a394`, CI 2469 green; `07` §6.3 [DESIGN] 2026-10-28 — an unused copy goes after 14 days, decided by CC unattended). Live on scratch 9202 2026-10-29: H5's busy volume (not written into, the live data put back, no hold; `audits/night-2026-10-28/r1256-restore/h5-live/`) and H8 (the 03:29 sweep removed the two old copies and kept the young one; `…/h8-live/`). H4 and H5's failed stop: tests only; H6: tests only (a live press cannot tell old from new). | OPEN — fixed on main, not released | the controller release (morning list) | the release; then an attended live proof of H4 and H5's failed stop, or revert those two branches of `342a394` | CC |
|
||||
| **R-1256** | Backup & restore | P3 | **Restore edge cases (review round 2, HUNT-1 H4/H5/H6/H8):** (H4) an off-site restore whose definition write fails AFTER the compose was written restarts the app on the snapshot's compose (`internal/backup/offbox_reconstitute.go:979-986`; `cmd/controller/main.go:3318-3331`); (H5) a put-back whose stop fails goes on and can merge (`restore_crash_putback.go:155-158`; `restore.go:218` ignores `volume rm -f`), then removes the copy; (H6) a second-drive mirror can start between the web refusal and the restore's own record (`web/restore_wizard.go:226`; `opstatus.go:56-66`), and the install-time kept-data load checks only RestoreStatus/IsRunning (`api/kept_install.go:89-92`); (H8) an unused restore copy is never cleaned up. Static. **2026-10-28 night:** all four fixed, each test red first (controller `342a394`, CI 2469 green; `07` §6.3 [DESIGN] 2026-10-28 — an unused copy goes after 14 days, decided by CC unattended). Live on scratch 9202 2026-10-29: H5's busy volume (not written into, the live data put back, no hold; `audits/night-2026-10-28/r1256-restore/h5-live/`) and H8 (the 03:29 sweep removed the two old copies and kept the young one; `…/h8-live/`). H4 and H5's failed stop: tests only; H6: tests only (a live press cannot tell old from new). | OPEN — fixed on main (controller `342a394`), not released. **All four proven live:** H5 busy volume + H8 sweep (9202, night 2026-10-28); **H4 and H5's failed stop on Tester 1, 2026-10-29 08:10 / 08:18 UTC** (D63, a `-tags faultinject` test build; `audits/d63-2026-10-29/`). D62 set the sweep age to 30 days (`e3fce09`). Closes with the controller release | the controller release (morning list) | the release; then an attended live proof of H4 and H5's failed stop, or revert those two branches of `342a394` | CC |
|
||||
| **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC |
|
||||
| **R-1274** | Backup & restore | P3 | **The restore refusal tells the household to „turn on remote backup, or connect a second drive” when the box has both and their first copy has not run yet.** Catalog sweep 2026-10-24, Grimmory and Calibre-Web: „Mentés most” makes the local copy only (`07` §8, by design), so the drive files had no copy yet; the refusal's sentence is `restore.refuse.files.none` (`felhom-controller/controller/internal/web/handlers.go:1835`) whatever the box has. The Hungarian sentence is frozen by `i18n_go_parity`, so the fix is a NEW key („the first copy is made in tonight's backup”) chosen when a tier is set up but holds nothing yet. | OPEN | — | The new key + the branch + a test | CC |
|
||||
| **R-91** | Backup & restore | P4 | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk **Checked from source 2026-10-05 (burn-down round 2):** Gate is long past (row waits on demo-felhom's first post-migration PBS backup, migration 2026-07-27). Last positive record of the copy: audits/CAMPAIGN-9-restore-proof-2026-07-28.md:759 'ep0 : /srv/pbs-felhom rollback copy intact (13G)'; CONTEXT.md:3666 still says it is 13 G of dead weight awaiting R-91. No later record of deletion found (grep srv/pbs-felhom across felhom.eu). Deleting is on ep0 (protected) and needs an operator word. | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC |
|
||||
|
||||
Reference in new issue
Block a user