admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Campaign 10 Phase B: 27 cycles, 586 invariant checks, 0 violations

Browse Source 
Ran the soak on the Phase A rig. Ended on its own deadline — no watchdog halt,
no atom exception, no I11 breach.

I1 28+28 pairs, I2 28+28 pairs, I3 56, I4 56, I5/I6 28 each, I7 28, I10 135,
I11 28. Zero violations. The row counts are themselves the no-silent-skip check:
I3/I4 twice per cycle (both drives), I10 = 5 secret-class fields x 27, REBOOT on
cycles 7/14/21 only.

I7 is the headline: 28 restores, 28 correct discriminators — never stale, never
empty. RTO (Tier 1, rallly, 66 MB): min 38.8s, median 42.0s, p90 42.5s, max
44.3s. That is the S band's lower end ONLY; the 5.5s spread over 28 runs says
fixed work dominates, so nothing extrapolates to M or L. RPO not measured.

Every atom and invariant was proven BY HAND before automation — the runner
asserts nothing that was not first observed live.

Caught a Phase A gap before starting: no app had HDD_PATH, so all data sat on the
system disk and I3 could never have fired. Deployed calibre-web onto adatok
first; otherwise the run would have produced 27 green cycles that tested nothing
cross-drive.

Investigated and DISPROVED a suspected defect (audit 5.2): /api/disks reports
state=attached for a physically absent drive, and intermediary.go:230 really does
compute presence from State=="attached". It is inert — planDriveGates only gates
paths under /mnt/felhom-drives/ and uses BoundUnderParent there, which was
correctly false. The gate fired; the storage page showed "Meghajtó leválasztva".
No R-n minted.

Honest gaps: 6 of ~12 atom families ran. Not run — Tier 3 (structurally
un-isolatable), abort-fs-in-place, kill-agent-mid-backup, hard-reset-mid-write,
reboot-VM, both concurrency atoms, fill-drive-near-full. I8 not checked, I9 not
automated (cited from the tester-gate run, not re-claimed). kill_controller is
NOT mid-backup and reboot_guest never interleaved with a detach. 27 cycles does
not answer the brief's question about drift at the thirty-eighth.

Teardown still OWED, including hub customer c10-soak (disposition: DELETE).
This commit is contained in:
authentik Default Admin
2026-08-01 15:45:13 +02:00
parent 4691aa1a35
commit 69f896d3cd
10 changed files with 1489 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/tests/campaign10-evidence-2026-07-31/runner/c10lib.py
+141
View File
@@ -0,0 +1,141 @@
#!/usr/bin/env python3
"""Campaign 10 — shared driver library. Runs on DooPlex, NOT inside the VM:
the soak's atoms kill the controller, the agent and the VM itself, so anything
running inside would die with the thing it is testing.
Canary model (brief A4): every app carries a marker naming the backup GENERATION
its data came from. Without it, a restore that returns stale or empty data is
indistinguishable from a good one — this arc produced six checks that passed for
the wrong reason.
rallly -> a row in a cc_proof TABLE in postgres, read over the app's real
network path (docker run --network container:rallly ... psql -h
rallly-postgres). NOT 127.0.0.1 inside the postgres container,
which postgres trusts and which produced D5's false pass.
others -> a marker file inside the app's own docker volume (what gets dumped).
"""
import json, os, subprocess, time
HOME = os.path.expanduser("~/.config/campaign10")
VM_IP = "192.168.0.105"
JUMP = "demo-hp"
VMID = "311"
GUEST = "9201"
HOSTPW = json.load(open(os.path.join(HOME, "host-recovery.json")))["password"]
SECRETS = json.load(open(os.path.join(HOME, "app-secrets.json")))
# app -> (container, volume mount dir) for the file-marker apps
FILE_APPS = {
"grafana": ("grafana", "/var/lib/grafana"),
"homebox": ("homebox", "/data"),
"papra": ("papra", "/app/data"),
}
ALL_APPS = ["rallly", "homebox", "grafana", "papra"]
def _run(args, timeout=300):
env = dict(os.environ, SSHPASS=HOSTPW, LC_ALL="C")
try:
p = subprocess.run(args, capture_output=True, text=True, timeout=timeout, env=env)
return p.returncode, p.stdout, p.stderr
except subprocess.TimeoutExpired:
return 124, "", "TIMEOUT"
def box(cmd, timeout=300):
"""Run a command on the campaign PVE host (the VM), via the demo-hp jump."""
return _run(["sshpass", "-e", "ssh", "-o", "StrictHostKeyChecking=no",
"-o", "ConnectTimeout=20", "-J", JUMP, f"root@{VM_IP}",
"LC_ALL=C " + cmd], timeout)
def guest(cmd, timeout=300):
"""Run a command inside guest 9201."""
return box(f"pct exec {GUEST} -- bash -c {shq(cmd)}", timeout)
def hp(cmd, timeout=180):
"""Run a command on demo-hp itself (for qm: reset, stop, disk detach...)."""
return _run(["ssh", "-o", "ConnectTimeout=20", JUMP, "LC_ALL=C " + cmd], timeout)
def shq(s):
return "'" + s.replace("'", "'\\''") + "'"
def api(method, path, body=None, timeout=300):
"""Authenticated controller API call through the on-guest helper."""
if body is None:
rc, out, _ = guest(f"/root/c10api.sh {method} {path} 2>/dev/null", timeout)
return out
import base64
b64 = base64.b64encode(json.dumps(body).encode()).decode()
rc, out, _ = guest(
f"echo {b64} | base64 -d > /tmp/.b && /root/c10api.sh {method} {path} "
f'"$(cat /tmp/.b)" --json; shred -u /tmp/.b 2>/dev/null', timeout)
return out
def jparse(raw):
try:
return json.loads(raw[raw.index("{"):raw.rindex("}") + 1])
except Exception:
return None
# ---------------------------------------------------------------- canaries
def _psql(sql, timeout=180):
"""Read/write rallly's DB over the path DATABASE_URL actually names."""
pw = SECRETS["rallly"]["values"]["DB_PASSWORD"]
url = f"postgresql://rallly:{pw}@rallly-postgres:5432/rallly"
cmd = ("docker run --rm --network container:rallly postgres:16-alpine "
f"psql {shq(url)} -tAc {shq(sql)}")
rc, out, err = guest(cmd, timeout)
return rc, out.strip(), err.strip()
def seed(gen):
"""Write the generation marker into every app. Returns {app: ok}."""
res = {}
_psql("CREATE TABLE IF NOT EXISTS cc_proof (id serial primary key, gen text, at timestamptz default now())")
rc, out, err = _psql(f"INSERT INTO cc_proof (gen) VALUES ('{gen}') RETURNING gen")
res["rallly"] = (gen in out)
for app, (cont, d) in FILE_APPS.items():
# -u 0: papra's volume is root-owned while the container runs as nonroot (R-156),
# so the app itself cannot write there. Seeding as root keeps the canary in the
# volume the BACKUP dumps, which is what I7 needs to observe.
rc, out, err = guest(
f"docker exec -u 0 {cont} sh -c {shq(f'echo {gen} > {d}/cc_proof.txt && cat {d}/cc_proof.txt')}", 180)
res[app] = (gen in out)
return res
def read_canaries():
"""Read each app's marker back. Returns {app: value-or-None}."""
res = {}
rc, out, err = _psql("SELECT gen FROM cc_proof ORDER BY id DESC LIMIT 1")
res["rallly"] = out.strip() or None
for app, (cont, d) in FILE_APPS.items():
rc, out, err = guest(f"docker exec {cont} sh -c {shq(f'cat {d}/cc_proof.txt 2>/dev/null')}", 180)
v = out.strip()
res[app] = v or None
return res
def containers():
rc, out, _ = guest("docker ps --format '{{.Names}}|{{.Status}}'", 120)
d = {}
for line in out.strip().splitlines():
if "|" in line:
n, s = line.split("|", 1)
d[n.strip()] = s.strip()
return d
def guest_init_pid():
"""I4: a stale-device bind must repair WITHOUT the guest restarting."""
rc, out, _ = box(f"pct exec {GUEST} -- cat /proc/1/stat 2>/dev/null", 120)
rc2, boot, _ = box(f"pct exec {GUEST} -- stat -c %Y /proc/1 2>/dev/null", 120)
return boot.strip()
documentation/tests/campaign10-evidence-2026-07-31/runner/c10run.py
+333
View File
@@ -0,0 +1,333 @@
#!/usr/bin/env python3
"""Campaign 10 — Phase B soak runner. Runs on DooPlex (outside the VM it abuses).
Shape (brief B1): a set of atomic operations composed into cycles in varying order,
with EVERY invariant checked EVERY cycle — drift at iteration 38 is the point.
Every atom and every invariant here was proven BY HAND before automation
(see the audit §5); nothing in this file asserts a behaviour that was not first
observed live.
Writes: journal.tsv (one row per invariant check), runner.log, status.txt (heartbeat).
Halts cleanly on: disk floor breach, VM gone, I11 breach (secret leak).
"""
import json, os, random, subprocess, sys, time, traceback
import c10lib as L
EV = "/mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/tests/campaign10-evidence-2026-07-31"
STATE = os.path.join(EV, "state")
os.makedirs(STATE, exist_ok=True)
JOURNAL = os.path.join(STATE, "journal.tsv")
RUNLOG = os.path.join(STATE, "runner.log")
STATUS = os.path.join(STATE, "status.txt")
APPS = ["rallly", "homebox", "grafana", "papra"] # docker-volume apps
DRIVE_APP = "calibre-web" # binds /mnt/felhom-drives/adatok
TARGET_DISK, TARGET_VOL = "scsi2", "c10-scratch:311/vm-311-disk-3.qcow2" # mentes = backup target
NONTGT_DISK, NONTGT_VOL = "scsi1", "c10-scratch:311/vm-311-disk-2.qcow2" # adatok
FLOOR_GB = 120 # halt if /mnt/nvme-1tb free drops below this
MAX_CYCLES = int(os.environ.get("C10_CYCLES", "40"))
DEADLINE = time.time() + float(os.environ.get("C10_HOURS", "6")) * 3600
CYCLE = 0
VIOL = []
def log(msg):
line = "%s %s" % (time.strftime("%H:%M:%S"), msg)
print(line, flush=True)
with open(RUNLOG, "a") as fh:
fh.write(line + "\n")
def rec(inv, ok, detail=""):
"""One journal row per invariant check. Violations do NOT stop the run (brief B2)."""
with open(JOURNAL, "a") as fh:
fh.write("%d\t%s\t%s\t%s\t%s\n" % (CYCLE, time.strftime("%FT%TZ", time.gmtime()),
inv, "PASS" if ok else "VIOLATION", detail[:400]))
if not ok:
VIOL.append((CYCLE, inv, detail[:300]))
log(" !! %s VIOLATION: %s" % (inv, detail[:220]))
def heartbeat(phase):
with open(STATUS, "w") as fh:
fh.write("cycle=%d\nphase=%s\nutc=%s\nviolations=%d\n"
% (CYCLE, phase, time.strftime("%FT%TZ", time.gmtime()), len(VIOL)))
# ------------------------------------------------------------------ helpers
def hub_events(since="3m"):
"""Hub-side event stream: type + severity, the observable I1/I2 assert on."""
p = subprocess.run(["sudo", "kubectl", "-n", "felhom-system", "logs", "deploy/hub",
"--since=" + since], capture_output=True, text=True, timeout=120)
out = []
for line in p.stdout.splitlines():
if "Event from c10-soak:" in line:
seg = line.split("Event from c10-soak:", 1)[1].strip()
typ = seg.split()[0]
sev = seg.split("(", 1)[1].split(")", 1)[0] if "(" in seg else "?"
out.append((typ, sev, seg))
return out
def target_state():
return (L.jparse(L.api("GET", "/api/storage/backup-target")) or {}).get("data", {}) or {}
def disks():
return ((L.jparse(L.api("GET", "/api/disks")) or {}).get("data", {}) or {}).get("disks", [])
def wait_backup(timeout=900):
t0 = time.time()
while time.time() - t0 < timeout:
d = ((L.jparse(L.api("GET", "/api/backup/status")) or {}).get("data") or {})
if not d.get("running"):
return d
time.sleep(10)
return {"timeout": True}
def wait_restore(timeout=900):
t0 = time.time()
while time.time() - t0 < timeout:
d = ((L.jparse(L.api("GET", "/api/backup/restore-status")) or {}).get("data") or {})
if d and not d.get("running"):
return d
time.sleep(10)
return {"timeout": True}
def settle(seconds=130):
"""Agent tick + host-report + hub event propagation. Measured at ~25-60s; 130 gives margin."""
time.sleep(seconds)
# ------------------------------------------------------------------ atoms
def atom_backup():
L.api("POST", "/api/backup/run", {})
d = wait_backup()
ok = bool(d.get("db_dump", {}).get("success", True)) and not d.get("timeout")
rec("BACKUP", ok, json.dumps(d)[:300])
return ok
def atom_restore_verify():
"""I7 — the whole point: a restore must return the discriminator asked for."""
app = "rallly" # the DB app; its canary is a real DB row
genA = "C10-C%03d-A-%s" % (CYCLE, time.strftime("%H%M%S"))
L.seed(genA)
if not atom_backup():
return
genB = "C10-C%03d-B-%s" % (CYCLE, time.strftime("%H%M%S"))
L.seed(genB) # must NOT survive the restore
snaps = L.jparse(L.api("GET", "/api/backup/snapshots?stack=%s" % app)) or {}
pts = snaps.get("data") or []
if not pts:
rec("I7", False, "no restore points for %s" % app)
return
sid = pts[0].get("short_id")
t0 = time.time()
L.guest('/root/c10api.sh POST /backup/restore "stack_name=%s&snapshot_id=%s" 2>/dev/null' % (app, sid))
d = wait_restore()
rto = time.time() - t0
got = L.read_canaries().get(app)
rec("I7", got == genA,
"app=%s want=%s got=%s snap=%s restore_ok=%s rto=%.1fs"
% (app, genA, got, sid, (d.get("last") or {}).get("ok"), rto))
# RTO/RPO byproduct (brief B3): to app-serving-correct-data, not to "restore returned"
with open(os.path.join(STATE, "rto.tsv"), "a") as fh:
fh.write("%d\t%s\t%s\ttier1\t%.1f\t%s\n" % (CYCLE, time.strftime("%FT%TZ", time.gmtime()),
app, rto, got == genA))
def _detach(disk, vol, label, expect_type, expect_recover):
"""I1/I2 — the absent event must be the RIGHT one, and the pair must match."""
before_boot = L.guest("stat -c %Y /proc/1")[1].strip()
L.hp("qm set 311 --delete %s" % disk)
settle()
evs = hub_events("4m")
absent = [e for e in evs if e[0] in ("backup_target_absent", "storage_disconnected")]
got = absent[-1][0] if absent else None
sev = absent[-1][1] if absent else None
rec("I1" if expect_type == "backup_target_absent" else "I2",
got == expect_type and sev == "error",
"%s absent -> got=%s sev=%s (want %s/error)" % (label, got, sev, expect_type))
# I3: an unusable namespace must not read as bound
for d in disks():
if d.get("guest_path", "") and label in str(d.get("guest_path", "")):
rec("I3", d.get("bound_under_parent") is False,
"%s bound_under_parent=%s while absent" % (label, d.get("bound_under_parent")))
L.hp("qm set 311 --%s %s" % (disk, vol))
settle()
evs2 = hub_events("4m")
back = [e for e in evs2 if e[0] in ("backup_target_restored", "storage_reconnected")]
got2 = back[-1][0] if back else None
sev2 = back[-1][1] if back else None
rec("I1-pair" if expect_type == "backup_target_absent" else "I2-pair",
got2 == expect_recover and sev2 == "info",
"%s return -> got=%s sev=%s (want %s/info)" % (label, got2, sev2, expect_recover))
# I4: the bind repairs WITHOUT the guest restarting
after_boot = L.guest("stat -c %Y /proc/1")[1].strip()
rec("I4", before_boot == after_boot and before_boot != "",
"guest init boot before=%s after=%s (must be equal)" % (before_boot, after_boot))
def atom_detach_target():
_detach(TARGET_DISK, TARGET_VOL, "mentes", "backup_target_absent", "backup_target_restored")
def atom_detach_nontarget():
_detach(NONTGT_DISK, NONTGT_VOL, "adatok", "storage_disconnected", "storage_reconnected")
def atom_reboot_guest():
L.box("pct reboot 9201", timeout=300)
for _ in range(40):
time.sleep(15)
if "felhom-controller" in L.containers():
break
rec("REBOOT", "felhom-controller" in L.containers(), "controller back after guest reboot")
def atom_kill_controller():
L.guest("docker restart felhom-controller >/dev/null 2>&1 &")
time.sleep(45)
rec("KILL-CTRL", "felhom-controller" in L.containers(), "controller recovered after kill")
def atom_redeploy_app():
app = random.choice(["homebox", "grafana"])
L.api("POST", "/api/stacks/%s/remove" % app, {})
time.sleep(20)
sec = L.SECRETS.get(app, {}).get("values", {})
if sec:
L.api("POST", "/api/stacks/%s/deploy" % app, {"values": sec})
for _ in range(20):
time.sleep(15)
if app in L.containers():
break
rec("REDEPLOY", app in L.containers(), "%s redeployed" % app)
# ------------------------------------------------------------------ per-cycle invariants
def check_healthy_baseline():
"""I5/I6 — the over-correction check: healthy must render nothing."""
ts = target_state()
degraded = ts.get("degraded")
evs = [e for e in hub_events("3m") if e[0] in ("backup_target_absent", "storage_disconnected")]
rec("I5", degraded is False and not evs,
"degraded=%s absent_events=%d target=%s" % (degraded, len(evs), ts.get("target")))
if degraded is False:
rec("I6", "message" not in ts or not ts.get("message"),
"healthy state must carry no degraded message; got=%s" % str(ts.get("message"))[:120])
UNIT = "/mnt/sys_drive/felhom-data/backups/primary/%s/compose/app.yaml"
def check_secret_class_travel():
"""I10 — `type: secret` travels to the local unit; `type: password` NEVER does.
Asserted in BOTH directions: the withheld class staying withheld is the half
that silently rots. Key NAMES only; no value is ever read."""
for app, meta in L.SECRETS.items():
classes = meta.get("secret_classes", {})
if not classes:
continue
rc, keys, _ = L.guest("grep -oE '^ *[A-Z_0-9]+:' " + (UNIT % app) +
" 2>/dev/null | tr -d ' :' | tr '\\n' ' '")
present = set(keys.split())
if not present:
rec("I10", False, "%s: recovery unit unreadable/absent" % app)
continue
for ev, c in classes.items():
if c["type"] == "secret":
rec("I10", ev in present, "%s/%s type=secret must travel; in_unit=%s"
% (app, ev, ev in present))
elif c["type"] == "password":
rec("I10", ev not in present, "%s/%s type=password must NEVER travel; in_unit=%s"
% (app, ev, ev in present))
def check_secrets_absent():
"""I11 — no secret value in any log, event or hub payload. Breach STOPS the run."""
vals = [v for a in L.SECRETS.values() for k, v in a.get("values", {}).items()
if k not in ("DOMAIN", "SUBDOMAIN") and v and len(v) > 8]
hay = L.guest("docker logs felhom-controller --since 20m 2>&1 | tail -c 200000")[1]
p = subprocess.run(["sudo", "kubectl", "-n", "felhom-system", "logs", "deploy/hub", "--since=20m"],
capture_output=True, text=True, timeout=120)
hay += p.stdout
leaked = [v[:6] + "..." for v in vals if v in hay]
rec("I11", not leaked, "leaked=%s" % leaked)
return not leaked
def watchdog():
rc, out, _ = L.hp("df --output=avail -BG /mnt/nvme-1tb | tail -1")
try:
free = int(out.strip().rstrip("G"))
except Exception:
free = 9999
rc2, vm, _ = L.hp("qm status 311")
alive = "running" in vm
if free < FLOOR_GB:
log("WATCHDOG: free %dG < floor %dG — halting cleanly" % (free, FLOOR_GB)); return False
if not alive:
log("WATCHDOG: VM 311 not running (%s) — halting cleanly" % vm.strip()); return False
return True
# ------------------------------------------------------------------ main
ATOMS = [atom_backup, atom_restore_verify, atom_detach_target,
atom_detach_nontarget, atom_kill_controller, atom_redeploy_app]
def main():
global CYCLE
log("=== Campaign 10 Phase B start — max %d cycles, deadline %s ==="
% (MAX_CYCLES, time.strftime("%H:%M", time.localtime(DEADLINE))))
if not os.path.exists(JOURNAL):
with open(JOURNAL, "w") as fh:
fh.write("cycle\tutc\tinvariant\tverdict\tdetail\n")
rng = random.Random(20260801)
while CYCLE < MAX_CYCLES and time.time() < DEADLINE:
CYCLE += 1
heartbeat("start")
if not watchdog():
break
order = ATOMS[:]
rng.shuffle(order) # permutation, not a fixed catalogue
# a reboot is expensive; fold it in every 7th cycle only
if CYCLE % 7 == 0:
order.append(atom_reboot_guest)
log("cycle %d: %s" % (CYCLE, " -> ".join(a.__name__.replace("atom_", "") for a in order)))
for a in order:
heartbeat(a.__name__)
try:
a()
except Exception as e:
rec(a.__name__, False, "EXCEPTION %s" % e)
log(" atom %s raised: %s" % (a.__name__, traceback.format_exc().splitlines()[-1]))
heartbeat("invariants")
try:
check_healthy_baseline()
check_secret_class_travel()
if not check_secrets_absent():
log("I11 BREACH — stopping the run immediately per brief B2"); break
except Exception as e:
log("invariant pass raised: %s" % e)
log("cycle %d done — violations so far: %d" % (CYCLE, len(VIOL)))
heartbeat("finished")
log("=== finished after %d cycles, %d violations ===" % (CYCLE, len(VIOL)))
for c, i, d in VIOL:
log(" cycle %d %s: %s" % (c, i, d))
if __name__ == "__main__":
main()
documentation/tests/campaign10-evidence-2026-07-31/runner/deploy_apps.py
+92
View File
@@ -0,0 +1,92 @@
#!/usr/bin/env python3
"""Campaign 10 — deploy apps through the REAL endpoints.
Mirrors the controller's own generator semantics (internal/stacks/deploy.go:870 generateValue):
password:N -> N random alphanumerics
hex:N -> N random BYTES, hex-encoded (2N chars)
base64key:N-> "base64:" + base64(N random bytes)
static:X -> X
Runs on DooPlex; talks to the guest via ssh -J demo-hp + pct exec + /root/c10api.sh.
Secrets are written to a 0600 manifest, never printed.
"""
import base64, json, os, secrets, string, subprocess, sys
APPS = sys.argv[1:] or ["rallly", "homebox", "grafana", "papra"]
DOMAIN = "c10.felhom.eu"
HOME = os.path.expanduser("~/.config/campaign10")
HOSTPW = json.load(open(os.path.join(HOME, "host-recovery.json")))["password"]
def ssh(cmd, timeout=900):
full = ["sshpass", "-e", "ssh", "-o", "StrictHostKeyChecking=no",
"-o", "ConnectTimeout=25", "-J", "demo-hp", "root@192.168.0.105",
"LC_ALL=C " + cmd]
env = dict(os.environ, SSHPASS=HOSTPW, LC_ALL="C")
return subprocess.run(full, capture_output=True, text=True, timeout=timeout, env=env).stdout
def api(method, path, body=None):
if body is None:
return ssh(f"pct exec 9201 -- /root/c10api.sh {method} {path} 2>/dev/null")
b64 = base64.b64encode(json.dumps(body).encode()).decode()
# body travels base64 so no quoting/locale can mangle a secret
return ssh("pct exec 9201 -- bash -c \"echo %s | base64 -d > /tmp/.b && "
"/root/c10api.sh %s %s \\\"\\$(cat /tmp/.b)\\\" --json; shred -u /tmp/.b\" 2>/dev/null"
% (b64, method, path))
def gen(spec):
kind, _, param = spec.partition(":")
if kind == "password":
al = string.ascii_letters + string.digits
return "".join(secrets.choice(al) for _ in range(int(param)))
if kind == "hex":
return secrets.token_bytes(int(param)).hex()
if kind == "base64key":
return "base64:" + base64.b64encode(secrets.token_bytes(int(param))).decode()
if kind == "static":
return param
raise ValueError("unknown generator %r" % spec)
manifest_path = os.path.join(HOME, "app-secrets.json")
manifest = json.load(open(manifest_path)) if os.path.exists(manifest_path) else {}
for app in APPS:
raw = api("GET", f"/api/stacks/{app}/deploy-fields")
try:
meta = json.loads(raw[raw.index("{"):raw.rindex("}") + 1])["data"]["metadata"]
except Exception as e:
print(f"{app:12s} FIELDS-FAIL {e} :: {raw[:160]!r}")
continue
values, classes = {}, {}
for f in meta.get("deploy_fields", []):
ev, typ, spec, dflt = f["env_var"], f.get("type", ""), f.get("generate", ""), f.get("default", "")
if typ == "domain":
values[ev] = DOMAIN
elif spec:
values[ev] = gen(spec)
elif dflt:
values[ev] = dflt
elif f.get("required"):
print(f"{app:12s} WARN required field {ev} has no default/generator")
if typ in ("secret", "password"):
classes[ev] = {"type": typ, "data_key": bool(f.get("data_key")), "label": f.get("label", "")}
manifest[app] = {"values": values, "secret_classes": classes,
"subdomain": values.get("SUBDOMAIN", ""), "domain": DOMAIN}
out = api("POST", f"/api/stacks/{app}/deploy", {"values": values})
ok = '"ok":true' in out
print(f"{app:12s} deploy -> {'OK' if ok else 'FAIL'} "
f"secret={sum(1 for c in classes.values() if c['type']=='secret')} "
f"password={sum(1 for c in classes.values() if c['type']=='password')} "
f"data_key={sum(1 for c in classes.values() if c['data_key'])}")
if not ok:
print(" ", out.strip()[:300])
with open(manifest_path, "w") as fh:
json.dump(manifest, fh, indent=1)
os.chmod(manifest_path, 0o600)
print("\nsecrets + secret-class map written to", manifest_path, "(0600, not printed)")
documentation/tests/campaign10-evidence-2026-07-31/runner/vm.sh
Executable
+62
View File
@@ -0,0 +1,62 @@
#!/bin/bash
# Campaign 10 — blind console driver for VM 311 on demo-hp.
# snap <label> : screendump the VM console -> PNG on DooPlex (for visual Read)
# key <k> [k...] : sendkey one or more keys (QEMU keynames), 0.25s apart
# type <string> : type an ASCII string as sendkey events
set -uo pipefail
SCRATCH=/tmp/claude-1000/-mnt-5-hdd-felhom-eu-git/0d68ad3e-5c39-4852-8e38-9c029dff2562/scratchpad
VM=311
H=demo-hp
mon() { ssh -o ConnectTimeout=10 "$H" "LC_ALL=C qm monitor $VM" <<<"$1" 2>/dev/null | grep -v '^QEMU\|^(qemu)' ; }
snap() {
local label="${1:-x}"
ssh -o ConnectTimeout=10 "$H" "LC_ALL=C qm monitor $VM" <<<"screendump /tmp/c10.ppm" >/dev/null 2>&1
scp -q "$H:/tmp/c10.ppm" "$SCRATCH/c10-$label.ppm" 2>/dev/null || { echo "snap: scp failed"; return 1; }
convert "$SCRATCH/c10-$label.ppm" "$SCRATCH/c10-$label.png" 2>/dev/null || { echo "snap: convert failed"; return 1; }
echo "$SCRATCH/c10-$label.png"
}
key() {
local cmds=""
for k in "$@"; do cmds+="sendkey $k
"; done
ssh -o ConnectTimeout=10 "$H" "LC_ALL=C qm monitor $VM" <<<"$cmds" >/dev/null 2>&1
}
# type: map ASCII -> qemu keynames, including shifted chars
type_str() {
local s="$1" cmds="" c
local -A M=( [' ']='spc' ['-']='minus' ['.']='dot' ['/']='slash' ['=']='equal'
[',']='comma' [';']='semicolon' ["'"]='apostrophe' ['\']='backslash'
['[']='bracket_left' [']']='bracket_right' ['`']='grave_accent' )
local -A SH=( ['_']='minus' ['+']='equal' ['?']='slash' [':']='semicolon' ['"']='apostrophe'
['!']='1' ['@']='2' ['#']='3' ['$']='4' ['%']='5' ['^']='6' ['&']='7'
['*']='8' ['(']='9' [')']='0' ['{']='bracket_left' ['}']='bracket_right'
['|']='backslash' ['~']='grave_accent' ['<']='comma' ['>']='dot' )
for (( i=0; i<${#s}; i++ )); do
c="${s:$i:1}"
if [[ -n "${SH[$c]:-}" ]]; then cmds+="sendkey shift-${SH[$c]}
"
elif [[ -n "${M[$c]:-}" ]]; then cmds+="sendkey ${M[$c]}
"
elif [[ "$c" =~ [A-Z] ]]; then cmds+="sendkey shift-$(tr '[:upper:]' '[:lower:]' <<<"$c")
"
elif [[ "$c" =~ [a-z0-9] ]]; then cmds+="sendkey $c
"
else echo "type_str: unmapped char '$c'" >&2; return 1
fi
done
ssh -o ConnectTimeout=10 "$H" "LC_ALL=C qm monitor $VM" <<<"$cmds" >/dev/null 2>&1
}
case "${1:-}" in
snap) shift; snap "$@" ;;
key) shift; key "$@" ;;
type) shift; type_str "$@" ;;
# typefile: read the string from a file so a secret never lands in argv/ps
typefile) shift; type_str "$(cat "$1")" ;;
mon) shift; mon "$@" ;;
*) echo "usage: vm.sh {snap <label>|key <k>...|type <str>|typefile <path>|mon <cmd>}"; exit 2 ;;
esac
documentation/tests/campaign10-evidence-2026-07-31/state/journal.tsv
+587
View File
@@ -0,0 +1,587 @@
cycle utc invariant verdict detail
1 2026-08-01T07:20:12Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
1 2026-08-01T07:20:15Z I3 PASS adatok bound_under_parent=False while absent
1 2026-08-01T07:22:27Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
1 2026-08-01T07:22:30Z I4 PASS guest init boot before=1785531262 after=1785531262 (must be equal)
1 2026-08-01T07:23:14Z REDEPLOY PASS grafana redeployed
1 2026-08-01T07:23:56Z BACKUP PASS {"db_dump": {"count": 1, "duration": "29.235715924s", "last_run": "2026-08-01T07:23:45.622922949Z", "success": true}, "enabled": true, "running": false}
1 2026-08-01T07:24:48Z BACKUP PASS {"db_dump": {"count": 1, "duration": "34.037544623s", "last_run": "2026-08-01T07:24:42.576575317Z", "success": true}, "enabled": true, "running": false}
1 2026-08-01T07:25:51Z I7 PASS app=rallly want=C10-C001-A-092356 got=C10-C001-A-092356 snap=helyi restore_ok=True rto=42.3s
1 2026-08-01T07:28:07Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
1 2026-08-01T07:28:10Z I3 PASS mentes bound_under_parent=False while absent
1 2026-08-01T07:30:22Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
1 2026-08-01T07:30:25Z I4 PASS guest init boot before=1785531262 after=1785531262 (must be equal)
1 2026-08-01T07:31:14Z KILL-CTRL PASS controller recovered after kill
1 2026-08-01T07:31:17Z I5 PASS degraded=False absent_events=0 target=felhom-backup
1 2026-08-01T07:31:17Z I6 PASS healthy state must carry no degraded message; got=None
1 2026-08-01T07:31:19Z I11 PASS leaked=[]
1 2026-08-01T07:35:53Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
1 2026-08-01T07:35:56Z I3 PASS adatok bound_under_parent=False while absent
1 2026-08-01T07:38:09Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
1 2026-08-01T07:38:12Z I4 PASS guest init boot before=1785531262 after=1785531262 (must be equal)
1 2026-08-01T07:38:56Z REDEPLOY PASS homebox redeployed
1 2026-08-01T07:39:38Z BACKUP PASS {"db_dump": {"count": 1, "duration": "29.454266341s", "last_run": "2026-08-01T07:39:28.065882881Z", "success": true}, "enabled": true, "running": false}
1 2026-08-01T07:40:31Z BACKUP PASS {"db_dump": {"count": 1, "duration": "32.574200326s", "last_run": "2026-08-01T07:40:23.804874002Z", "success": true}, "enabled": true, "running": false}
1 2026-08-01T07:41:34Z I7 PASS app=rallly want=C10-C001-A-093938 got=C10-C001-A-093938 snap=helyi restore_ok=True rto=41.8s
1 2026-08-01T07:43:49Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
1 2026-08-01T07:43:52Z I3 PASS mentes bound_under_parent=False while absent
1 2026-08-01T07:46:05Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
1 2026-08-01T07:46:07Z I4 PASS guest init boot before=1785531262 after=1785531262 (must be equal)
1 2026-08-01T07:46:56Z KILL-CTRL PASS controller recovered after kill
1 2026-08-01T07:46:59Z I5 PASS degraded=False absent_events=0 target=felhom-backup
1 2026-08-01T07:46:59Z I6 PASS healthy state must carry no degraded message; got=None
1 2026-08-01T07:47:01Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
1 2026-08-01T07:47:01Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
1 2026-08-01T07:47:03Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
1 2026-08-01T07:47:05Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
1 2026-08-01T07:47:07Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
1 2026-08-01T07:47:09Z I11 PASS leaked=[]
2 2026-08-01T07:48:01Z KILL-CTRL PASS controller recovered after kill
2 2026-08-01T07:50:16Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
2 2026-08-01T07:50:19Z I3 PASS adatok bound_under_parent=False while absent
2 2026-08-01T07:52:32Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
2 2026-08-01T07:52:35Z I4 PASS guest init boot before=1785531262 after=1785531262 (must be equal)
2 2026-08-01T07:53:17Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.724847172s", "last_run": "2026-08-01T07:53:05.968779815Z", "success": true}, "enabled": true, "running": false}
2 2026-08-01T07:54:01Z REDEPLOY PASS grafana redeployed
2 2026-08-01T07:54:54Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.880675052s", "last_run": "2026-08-01T07:54:43.377582192Z", "success": true}, "enabled": true, "running": false}
2 2026-08-01T07:55:56Z I7 PASS app=rallly want=C10-C002-A-095401 got=C10-C002-A-095401 snap=helyi restore_ok=True rto=41.7s
2 2026-08-01T07:58:12Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
2 2026-08-01T07:58:14Z I3 PASS mentes bound_under_parent=False while absent
2 2026-08-01T08:00:27Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
2 2026-08-01T08:00:30Z I4 PASS guest init boot before=1785531262 after=1785531262 (must be equal)
2 2026-08-01T08:00:32Z I5 PASS degraded=False absent_events=0 target=felhom-backup
2 2026-08-01T08:00:32Z I6 PASS healthy state must carry no degraded message; got=None
2 2026-08-01T08:00:34Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
2 2026-08-01T08:00:34Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
2 2026-08-01T08:00:36Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
2 2026-08-01T08:00:38Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
2 2026-08-01T08:00:40Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
2 2026-08-01T08:00:42Z I11 PASS leaked=[]
3 2026-08-01T08:03:00Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
3 2026-08-01T08:03:03Z I3 PASS mentes bound_under_parent=False while absent
3 2026-08-01T08:05:15Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
3 2026-08-01T08:05:18Z I4 PASS guest init boot before=1785531262 after=1785531262 (must be equal)
3 2026-08-01T08:06:02Z REDEPLOY PASS homebox redeployed
3 2026-08-01T08:06:44Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.249083164s", "last_run": "2026-08-01T08:06:32.499808867Z", "success": true}, "enabled": true, "running": false}
3 2026-08-01T08:09:00Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
3 2026-08-01T08:09:02Z I3 PASS adatok bound_under_parent=False while absent
3 2026-08-01T08:11:15Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
3 2026-08-01T08:11:18Z I4 PASS guest init boot before=1785531262 after=1785531262 (must be equal)
3 2026-08-01T08:12:07Z KILL-CTRL PASS controller recovered after kill
3 2026-08-01T08:13:00Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.907036117s", "last_run": "2026-08-01T08:12:49.360548351Z", "success": true}, "enabled": true, "running": false}
3 2026-08-01T08:14:04Z I7 PASS app=rallly want=C10-C003-A-101207 got=C10-C003-A-101207 snap=helyi restore_ok=True rto=44.3s
3 2026-08-01T08:14:07Z I5 PASS degraded=False absent_events=0 target=felhom-backup
3 2026-08-01T08:14:07Z I6 PASS healthy state must carry no degraded message; got=None
3 2026-08-01T08:14:09Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
3 2026-08-01T08:14:09Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
3 2026-08-01T08:14:11Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
3 2026-08-01T08:14:13Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
3 2026-08-01T08:14:15Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
3 2026-08-01T08:14:17Z I11 PASS leaked=[]
4 2026-08-01T08:16:35Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
4 2026-08-01T08:16:38Z I3 PASS adatok bound_under_parent=False while absent
4 2026-08-01T08:18:51Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
4 2026-08-01T08:18:54Z I4 PASS guest init boot before=1785531262 after=1785531262 (must be equal)
4 2026-08-01T08:19:35Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.425451162s", "last_run": "2026-08-01T08:19:24.403091262Z", "success": true}, "enabled": true, "running": false}
4 2026-08-01T08:20:20Z REDEPLOY PASS grafana redeployed
4 2026-08-01T08:21:12Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.411530632s", "last_run": "2026-08-01T08:21:00.742317925Z", "success": true}, "enabled": true, "running": false}
4 2026-08-01T08:22:14Z I7 PASS app=rallly want=C10-C004-A-102020 got=C10-C004-A-102020 snap=helyi restore_ok=True rto=42.0s
4 2026-08-01T08:24:30Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
4 2026-08-01T08:24:32Z I3 PASS mentes bound_under_parent=False while absent
4 2026-08-01T08:26:45Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
4 2026-08-01T08:26:48Z I4 PASS guest init boot before=1785531262 after=1785531262 (must be equal)
4 2026-08-01T08:27:37Z KILL-CTRL PASS controller recovered after kill
4 2026-08-01T08:27:39Z I5 PASS degraded=False absent_events=0 target=felhom-backup
4 2026-08-01T08:27:39Z I6 PASS healthy state must carry no degraded message; got=None
4 2026-08-01T08:27:41Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
4 2026-08-01T08:27:41Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
4 2026-08-01T08:27:43Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
4 2026-08-01T08:27:46Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
4 2026-08-01T08:27:47Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
4 2026-08-01T08:27:50Z I11 PASS leaked=[]
5 2026-08-01T08:28:36Z REDEPLOY PASS grafana redeployed
5 2026-08-01T08:30:51Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
5 2026-08-01T08:30:54Z I3 PASS adatok bound_under_parent=False while absent
5 2026-08-01T08:33:07Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
5 2026-08-01T08:33:10Z I4 PASS guest init boot before=1785531262 after=1785531262 (must be equal)
5 2026-08-01T08:33:59Z KILL-CTRL PASS controller recovered after kill
5 2026-08-01T08:36:15Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
5 2026-08-01T08:36:18Z I3 PASS mentes bound_under_parent=False while absent
5 2026-08-01T08:38:30Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
5 2026-08-01T08:38:33Z I4 PASS guest init boot before=1785531262 after=1785531262 (must be equal)
5 2026-08-01T08:39:15Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.537566027s", "last_run": "2026-08-01T08:39:03.884190323Z", "success": true}, "enabled": true, "running": false}
5 2026-08-01T08:40:08Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.819105251s", "last_run": "2026-08-01T08:39:56.870257525Z", "success": true}, "enabled": true, "running": false}
5 2026-08-01T08:41:13Z I7 PASS app=rallly want=C10-C005-A-103915 got=C10-C005-A-103915 snap=helyi restore_ok=True rto=43.0s
5 2026-08-01T08:41:16Z I5 PASS degraded=False absent_events=0 target=felhom-backup
5 2026-08-01T08:41:16Z I6 PASS healthy state must carry no degraded message; got=None
5 2026-08-01T08:41:18Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
5 2026-08-01T08:41:18Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
5 2026-08-01T08:41:20Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
5 2026-08-01T08:41:22Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
5 2026-08-01T08:41:24Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
5 2026-08-01T08:41:26Z I11 PASS leaked=[]
6 2026-08-01T08:42:18Z KILL-CTRL PASS controller recovered after kill
6 2026-08-01T08:43:10Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.890401885s", "last_run": "2026-08-01T08:42:58.392385368Z", "success": true}, "enabled": true, "running": false}
6 2026-08-01T08:44:12Z I7 PASS app=rallly want=C10-C006-A-104218 got=C10-C006-A-104218 snap=helyi restore_ok=True rto=41.6s
6 2026-08-01T08:46:28Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
6 2026-08-01T08:46:30Z I3 PASS mentes bound_under_parent=False while absent
6 2026-08-01T08:48:44Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
6 2026-08-01T08:48:46Z I4 PASS guest init boot before=1785531262 after=1785531262 (must be equal)
6 2026-08-01T08:51:02Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
6 2026-08-01T08:51:05Z I3 PASS adatok bound_under_parent=False while absent
6 2026-08-01T08:53:18Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
6 2026-08-01T08:53:20Z I4 PASS guest init boot before=1785531262 after=1785531262 (must be equal)
6 2026-08-01T08:54:02Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.101250195s", "last_run": "2026-08-01T08:53:50.441271314Z", "success": true}, "enabled": true, "running": false}
6 2026-08-01T08:54:46Z REDEPLOY PASS grafana redeployed
6 2026-08-01T08:54:49Z I5 PASS degraded=False absent_events=0 target=felhom-backup
6 2026-08-01T08:54:49Z I6 PASS healthy state must carry no degraded message; got=None
6 2026-08-01T08:54:51Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
6 2026-08-01T08:54:51Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
6 2026-08-01T08:54:53Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
6 2026-08-01T08:54:55Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
6 2026-08-01T08:54:57Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
6 2026-08-01T08:54:59Z I11 PASS leaked=[]
7 2026-08-01T08:57:17Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
7 2026-08-01T08:57:20Z I3 PASS adatok bound_under_parent=False while absent
7 2026-08-01T08:59:33Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
7 2026-08-01T08:59:35Z I4 PASS guest init boot before=1785531262 after=1785531262 (must be equal)
7 2026-08-01T09:01:50Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
7 2026-08-01T09:01:54Z I3 PASS mentes bound_under_parent=False while absent
7 2026-08-01T09:04:07Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
7 2026-08-01T09:04:09Z I4 PASS guest init boot before=1785531262 after=1785531262 (must be equal)
7 2026-08-01T09:04:58Z KILL-CTRL PASS controller recovered after kill
7 2026-08-01T09:05:43Z REDEPLOY PASS homebox redeployed
7 2026-08-01T09:06:35Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.192784423s", "last_run": "2026-08-01T09:06:24.336193131Z", "success": true}, "enabled": true, "running": false}
7 2026-08-01T09:07:38Z I7 PASS app=rallly want=C10-C007-A-110543 got=C10-C007-A-110543 snap=helyi restore_ok=True rto=42.2s
7 2026-08-01T09:08:21Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.754755491s", "last_run": "2026-08-01T09:08:08.67970607Z", "success": true}, "enabled": true, "running": false}
7 2026-08-01T09:09:00Z REBOOT PASS controller back after guest reboot
7 2026-08-01T09:09:03Z I5 PASS degraded=False absent_events=0 target=felhom-backup
7 2026-08-01T09:09:03Z I6 PASS healthy state must carry no degraded message; got=None
7 2026-08-01T09:09:05Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
7 2026-08-01T09:09:05Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
7 2026-08-01T09:09:07Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
7 2026-08-01T09:09:09Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
7 2026-08-01T09:09:11Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
7 2026-08-01T09:09:13Z I11 PASS leaked=[]
8 2026-08-01T09:09:59Z REDEPLOY PASS homebox redeployed
8 2026-08-01T09:10:48Z KILL-CTRL PASS controller recovered after kill
8 2026-08-01T09:11:41Z BACKUP PASS {"db_dump": {"count": 1, "duration": "29.417683334s", "last_run": "2026-08-01T09:11:30.326680315Z", "success": true}, "enabled": true, "running": false}
8 2026-08-01T09:12:45Z I7 PASS app=rallly want=C10-C008-A-111048 got=C10-C008-A-111048 snap=helyi restore_ok=True rto=42.3s
8 2026-08-01T09:15:01Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
8 2026-08-01T09:15:04Z I3 PASS adatok bound_under_parent=False while absent
8 2026-08-01T09:17:16Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
8 2026-08-01T09:17:19Z I4 PASS guest init boot before=1785575320 after=1785575320 (must be equal)
8 2026-08-01T09:19:35Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
8 2026-08-01T09:19:38Z I3 PASS mentes bound_under_parent=False while absent
8 2026-08-01T09:21:51Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
8 2026-08-01T09:21:53Z I4 PASS guest init boot before=1785575320 after=1785575320 (must be equal)
8 2026-08-01T09:22:35Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.112766197s", "last_run": "2026-08-01T09:22:23.959739494Z", "success": true}, "enabled": true, "running": false}
8 2026-08-01T09:22:38Z I5 PASS degraded=False absent_events=0 target=felhom-backup
8 2026-08-01T09:22:38Z I6 PASS healthy state must carry no degraded message; got=None
8 2026-08-01T09:22:40Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
8 2026-08-01T09:22:40Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
8 2026-08-01T09:22:42Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
8 2026-08-01T09:22:44Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
8 2026-08-01T09:22:46Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
8 2026-08-01T09:22:48Z I11 PASS leaked=[]
9 2026-08-01T09:23:39Z KILL-CTRL PASS controller recovered after kill
9 2026-08-01T09:24:19Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.120915244s", "last_run": "2026-08-01T09:24:19.437409908Z", "success": true}, "enabled": true, "running": false}
9 2026-08-01T09:25:22Z I7 PASS app=rallly want=C10-C009-A-112339 got=C10-C009-A-112339 snap=helyi restore_ok=True rto=41.9s
9 2026-08-01T09:26:06Z REDEPLOY PASS grafana redeployed
9 2026-08-01T09:28:22Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
9 2026-08-01T09:28:25Z I3 PASS adatok bound_under_parent=False while absent
9 2026-08-01T09:30:38Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
9 2026-08-01T09:30:40Z I4 PASS guest init boot before=1785575320 after=1785575320 (must be equal)
9 2026-08-01T09:31:10Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.234736705s", "last_run": "2026-08-01T09:31:09.977587344Z", "success": true}, "enabled": true, "running": false}
9 2026-08-01T09:33:26Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
9 2026-08-01T09:33:29Z I3 PASS mentes bound_under_parent=False while absent
9 2026-08-01T09:35:42Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
9 2026-08-01T09:35:44Z I4 PASS guest init boot before=1785575320 after=1785575320 (must be equal)
9 2026-08-01T09:35:47Z I5 PASS degraded=False absent_events=0 target=felhom-backup
9 2026-08-01T09:35:47Z I6 PASS healthy state must carry no degraded message; got=None
9 2026-08-01T09:35:49Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
9 2026-08-01T09:35:49Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
9 2026-08-01T09:35:51Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
9 2026-08-01T09:35:53Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
9 2026-08-01T09:35:55Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
9 2026-08-01T09:35:57Z I11 PASS leaked=[]
10 2026-08-01T09:38:14Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
10 2026-08-01T09:38:17Z I3 PASS adatok bound_under_parent=False while absent
10 2026-08-01T09:40:30Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
10 2026-08-01T09:40:33Z I4 PASS guest init boot before=1785575320 after=1785575320 (must be equal)
10 2026-08-01T09:41:17Z REDEPLOY PASS homebox redeployed
10 2026-08-01T09:42:06Z KILL-CTRL PASS controller recovered after kill
10 2026-08-01T09:44:22Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
10 2026-08-01T09:44:25Z I3 PASS mentes bound_under_parent=False while absent
10 2026-08-01T09:46:38Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
10 2026-08-01T09:46:40Z I4 PASS guest init boot before=1785575320 after=1785575320 (must be equal)
10 2026-08-01T09:47:21Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.035320075s", "last_run": "2026-08-01T09:47:21.013177918Z", "success": true}, "enabled": true, "running": false}
10 2026-08-01T09:48:24Z I7 PASS app=rallly want=C10-C010-A-114640 got=C10-C010-A-114640 snap=helyi restore_ok=True rto=41.6s
10 2026-08-01T09:48:54Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.198265248s", "last_run": "2026-08-01T09:48:54.018758879Z", "success": true}, "enabled": true, "running": false}
10 2026-08-01T09:48:57Z I5 PASS degraded=False absent_events=0 target=felhom-backup
10 2026-08-01T09:48:57Z I6 PASS healthy state must carry no degraded message; got=None
10 2026-08-01T09:48:59Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
10 2026-08-01T09:48:59Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
10 2026-08-01T09:49:01Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
10 2026-08-01T09:49:03Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
10 2026-08-01T09:49:05Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
10 2026-08-01T09:49:07Z I11 PASS leaked=[]
11 2026-08-01T09:50:02Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.500198373s", "last_run": "2026-08-01T09:49:50.111560186Z", "success": true}, "enabled": true, "running": false}
11 2026-08-01T09:51:05Z I7 PASS app=rallly want=C10-C011-A-114910 got=C10-C011-A-114910 snap=helyi restore_ok=True rto=41.7s
11 2026-08-01T09:53:20Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
11 2026-08-01T09:53:23Z I3 PASS adatok bound_under_parent=False while absent
11 2026-08-01T09:55:35Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
11 2026-08-01T09:55:38Z I4 PASS guest init boot before=1785575320 after=1785575320 (must be equal)
11 2026-08-01T09:57:54Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
11 2026-08-01T09:57:57Z I3 PASS mentes bound_under_parent=False while absent
11 2026-08-01T10:00:10Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
11 2026-08-01T10:00:13Z I4 PASS guest init boot before=1785575320 after=1785575320 (must be equal)
11 2026-08-01T10:00:55Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.274827764s", "last_run": "2026-08-01T10:00:42.619398629Z", "success": true}, "enabled": true, "running": false}
11 2026-08-01T10:01:38Z REDEPLOY PASS homebox redeployed
11 2026-08-01T10:02:28Z KILL-CTRL PASS controller recovered after kill
11 2026-08-01T10:02:30Z I5 PASS degraded=False absent_events=0 target=felhom-backup
11 2026-08-01T10:02:30Z I6 PASS healthy state must carry no degraded message; got=None
11 2026-08-01T10:02:32Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
11 2026-08-01T10:02:32Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
11 2026-08-01T10:02:34Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
11 2026-08-01T10:02:36Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
11 2026-08-01T10:02:38Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
11 2026-08-01T10:02:40Z I11 PASS leaked=[]
12 2026-08-01T10:03:26Z REDEPLOY PASS homebox redeployed
12 2026-08-01T10:05:42Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
12 2026-08-01T10:05:45Z I3 PASS adatok bound_under_parent=False while absent
12 2026-08-01T10:07:57Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
12 2026-08-01T10:08:00Z I4 PASS guest init boot before=1785575320 after=1785575320 (must be equal)
12 2026-08-01T10:08:49Z KILL-CTRL PASS controller recovered after kill
12 2026-08-01T10:11:04Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
12 2026-08-01T10:11:07Z I3 PASS mentes bound_under_parent=False while absent
12 2026-08-01T10:13:20Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
12 2026-08-01T10:13:24Z I4 PASS guest init boot before=1785575320 after=1785575320 (must be equal)
12 2026-08-01T10:13:54Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.82906082s", "last_run": "2026-08-01T10:13:53.343511281Z", "success": true}, "enabled": true, "running": false}
12 2026-08-01T10:14:47Z BACKUP PASS {"db_dump": {"count": 1, "duration": "34.794492507s", "last_run": "2026-08-01T10:14:42.108803281Z", "success": true}, "enabled": true, "running": false}
12 2026-08-01T10:15:50Z I7 PASS app=rallly want=C10-C012-A-121354 got=C10-C012-A-121354 snap=helyi restore_ok=True rto=41.6s
12 2026-08-01T10:15:53Z I5 PASS degraded=False absent_events=0 target=felhom-backup
12 2026-08-01T10:15:53Z I6 PASS healthy state must carry no degraded message; got=None
12 2026-08-01T10:15:54Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
12 2026-08-01T10:15:54Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
12 2026-08-01T10:15:56Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
12 2026-08-01T10:15:58Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
12 2026-08-01T10:16:00Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
12 2026-08-01T10:16:02Z I11 PASS leaked=[]
13 2026-08-01T10:16:54Z KILL-CTRL PASS controller recovered after kill
13 2026-08-01T10:17:38Z REDEPLOY PASS grafana redeployed
13 2026-08-01T10:18:18Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.066037384s", "last_run": "2026-08-01T10:18:18.41231692Z", "success": true}, "enabled": true, "running": false}
13 2026-08-01T10:19:22Z I7 PASS app=rallly want=C10-C013-A-121738 got=C10-C013-A-121738 snap=helyi restore_ok=True rto=42.2s
13 2026-08-01T10:20:03Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.463205132s", "last_run": "2026-08-01T10:19:51.680188929Z", "success": true}, "enabled": true, "running": false}
13 2026-08-01T10:22:19Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
13 2026-08-01T10:22:22Z I3 PASS mentes bound_under_parent=False while absent
13 2026-08-01T10:24:35Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
13 2026-08-01T10:24:37Z I4 PASS guest init boot before=1785575320 after=1785575320 (must be equal)
13 2026-08-01T10:26:53Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
13 2026-08-01T10:26:56Z I3 PASS adatok bound_under_parent=False while absent
13 2026-08-01T10:29:09Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
13 2026-08-01T10:29:12Z I4 PASS guest init boot before=1785575320 after=1785575320 (must be equal)
13 2026-08-01T10:29:14Z I5 PASS degraded=False absent_events=0 target=felhom-backup
13 2026-08-01T10:29:14Z I6 PASS healthy state must carry no degraded message; got=None
13 2026-08-01T10:29:16Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
13 2026-08-01T10:29:16Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
13 2026-08-01T10:29:18Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
13 2026-08-01T10:29:20Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
13 2026-08-01T10:29:22Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
13 2026-08-01T10:29:25Z I11 PASS leaked=[]
14 2026-08-01T10:29:56Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.80615982s", "last_run": "2026-08-01T10:29:55.907589946Z", "success": true}, "enabled": true, "running": false}
14 2026-08-01T10:30:50Z BACKUP PASS {"db_dump": {"count": 1, "duration": "33.858181263s", "last_run": "2026-08-01T10:30:43.782817699Z", "success": true}, "enabled": true, "running": false}
14 2026-08-01T10:31:54Z I7 PASS app=rallly want=C10-C014-A-122956 got=C10-C014-A-122956 snap=helyi restore_ok=True rto=42.4s
14 2026-08-01T10:34:09Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
14 2026-08-01T10:34:12Z I3 PASS adatok bound_under_parent=False while absent
14 2026-08-01T10:36:25Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
14 2026-08-01T10:36:28Z I4 PASS guest init boot before=1785575320 after=1785575320 (must be equal)
14 2026-08-01T10:37:12Z REDEPLOY PASS grafana redeployed
14 2026-08-01T10:38:01Z KILL-CTRL PASS controller recovered after kill
14 2026-08-01T10:40:17Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
14 2026-08-01T10:40:20Z I3 PASS mentes bound_under_parent=False while absent
14 2026-08-01T10:42:33Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
14 2026-08-01T10:42:36Z I4 PASS guest init boot before=1785575320 after=1785575320 (must be equal)
14 2026-08-01T10:43:14Z REBOOT PASS controller back after guest reboot
14 2026-08-01T10:43:17Z I5 PASS degraded=False absent_events=0 target=felhom-backup
14 2026-08-01T10:43:17Z I6 PASS healthy state must carry no degraded message; got=None
14 2026-08-01T10:43:19Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
14 2026-08-01T10:43:19Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
14 2026-08-01T10:43:21Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
14 2026-08-01T10:43:24Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
14 2026-08-01T10:43:26Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
14 2026-08-01T10:43:28Z I11 PASS leaked=[]
15 2026-08-01T10:45:45Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
15 2026-08-01T10:45:48Z I3 PASS mentes bound_under_parent=False while absent
15 2026-08-01T10:48:01Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
15 2026-08-01T10:48:03Z I4 PASS guest init boot before=1785580975 after=1785580975 (must be equal)
15 2026-08-01T10:48:46Z BACKUP PASS {"db_dump": {"count": 1, "duration": "28.270428557s", "last_run": "2026-08-01T10:48:34.163245256Z", "success": true}, "enabled": true, "running": false}
15 2026-08-01T10:49:36Z KILL-CTRL PASS controller recovered after kill
15 2026-08-01T10:50:20Z REDEPLOY PASS homebox redeployed
15 2026-08-01T10:51:01Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.421623567s", "last_run": "2026-08-01T10:51:00.95301014Z", "success": true}, "enabled": true, "running": false}
15 2026-08-01T10:52:04Z I7 PASS app=rallly want=C10-C015-A-125020 got=C10-C015-A-125020 snap=helyi restore_ok=True rto=41.8s
15 2026-08-01T10:54:20Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
15 2026-08-01T10:54:23Z I3 PASS adatok bound_under_parent=False while absent
15 2026-08-01T10:56:36Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
15 2026-08-01T10:56:39Z I4 PASS guest init boot before=1785580975 after=1785580975 (must be equal)
15 2026-08-01T10:56:42Z I5 PASS degraded=False absent_events=0 target=felhom-backup
15 2026-08-01T10:56:42Z I6 PASS healthy state must carry no degraded message; got=None
15 2026-08-01T10:56:45Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
15 2026-08-01T10:56:45Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
15 2026-08-01T10:56:47Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
15 2026-08-01T10:56:49Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
15 2026-08-01T10:56:51Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
15 2026-08-01T10:56:54Z I11 PASS leaked=[]
16 2026-08-01T10:57:27Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.683550606s", "last_run": "2026-08-01T10:57:26.606568663Z", "success": true}, "enabled": true, "running": false}
16 2026-08-01T10:59:43Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
16 2026-08-01T10:59:48Z I3 PASS mentes bound_under_parent=False while absent
16 2026-08-01T11:02:02Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
16 2026-08-01T11:02:08Z I4 PASS guest init boot before=1785580975 after=1785580975 (must be equal)
16 2026-08-01T11:04:31Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
16 2026-08-01T11:04:40Z I3 PASS adatok bound_under_parent=False while absent
16 2026-08-01T11:06:58Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
16 2026-08-01T11:07:01Z I4 PASS guest init boot before=1785580975 after=1785580975 (must be equal)
16 2026-08-01T11:08:29Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.894128293s", "last_run": "2026-08-01T11:08:14.025870917Z", "success": true}, "enabled": true, "running": false}
16 2026-08-01T11:09:59Z I7 PASS app=rallly want=C10-C016-A-130701 got=C10-C016-A-130701 snap=helyi restore_ok=True rto=38.8s
16 2026-08-01T11:10:54Z KILL-CTRL PASS controller recovered after kill
16 2026-08-01T11:11:53Z REDEPLOY PASS grafana redeployed
16 2026-08-01T11:12:03Z I5 PASS degraded=False absent_events=0 target=felhom-backup
16 2026-08-01T11:12:03Z I6 PASS healthy state must carry no degraded message; got=None
16 2026-08-01T11:12:11Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
16 2026-08-01T11:12:11Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
16 2026-08-01T11:12:19Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
16 2026-08-01T11:12:27Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
16 2026-08-01T11:12:36Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
16 2026-08-01T11:12:43Z I11 PASS leaked=[]
17 2026-08-01T11:13:43Z REDEPLOY PASS grafana redeployed
17 2026-08-01T11:16:04Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
17 2026-08-01T11:16:08Z I3 PASS adatok bound_under_parent=False while absent
17 2026-08-01T11:18:21Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
17 2026-08-01T11:18:23Z I4 PASS guest init boot before=1785580975 after=1785580975 (must be equal)
17 2026-08-01T11:18:53Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.838575753s", "last_run": "2026-08-01T11:18:52.693573505Z", "success": true}, "enabled": true, "running": false}
17 2026-08-01T11:19:42Z KILL-CTRL PASS controller recovered after kill
17 2026-08-01T11:20:23Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.317455253s", "last_run": "2026-08-01T11:20:23.002386465Z", "success": true}, "enabled": true, "running": false}
17 2026-08-01T11:21:26Z I7 PASS app=rallly want=C10-C017-A-131942 got=C10-C017-A-131942 snap=helyi restore_ok=True rto=41.5s
17 2026-08-01T11:23:42Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
17 2026-08-01T11:23:45Z I3 PASS mentes bound_under_parent=False while absent
17 2026-08-01T11:25:57Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
17 2026-08-01T11:26:00Z I4 PASS guest init boot before=1785580975 after=1785580975 (must be equal)
17 2026-08-01T11:26:02Z I5 PASS degraded=False absent_events=0 target=felhom-backup
17 2026-08-01T11:26:02Z I6 PASS healthy state must carry no degraded message; got=None
17 2026-08-01T11:26:04Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
17 2026-08-01T11:26:04Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
17 2026-08-01T11:26:06Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
17 2026-08-01T11:26:08Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
17 2026-08-01T11:26:10Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
17 2026-08-01T11:26:12Z I11 PASS leaked=[]
18 2026-08-01T11:28:30Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
18 2026-08-01T11:28:33Z I3 PASS mentes bound_under_parent=False while absent
18 2026-08-01T11:30:45Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
18 2026-08-01T11:30:48Z I4 PASS guest init boot before=1785580975 after=1785580975 (must be equal)
18 2026-08-01T11:31:38Z KILL-CTRL PASS controller recovered after kill
18 2026-08-01T11:33:53Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
18 2026-08-01T11:33:57Z I3 PASS adatok bound_under_parent=False while absent
18 2026-08-01T11:36:10Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
18 2026-08-01T11:36:12Z I4 PASS guest init boot before=1785580975 after=1785580975 (must be equal)
18 2026-08-01T11:36:42Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.287771352s", "last_run": "2026-08-01T11:36:41.854058541Z", "success": true}, "enabled": true, "running": false}
18 2026-08-01T11:37:27Z REDEPLOY PASS grafana redeployed
18 2026-08-01T11:38:08Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.269077889s", "last_run": "2026-08-01T11:38:07.785076049Z", "success": true}, "enabled": true, "running": false}
18 2026-08-01T11:39:12Z I7 PASS app=rallly want=C10-C018-A-133727 got=C10-C018-A-133727 snap=helyi restore_ok=True rto=42.0s
18 2026-08-01T11:39:15Z I5 PASS degraded=False absent_events=0 target=felhom-backup
18 2026-08-01T11:39:15Z I6 PASS healthy state must carry no degraded message; got=None
18 2026-08-01T11:39:17Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
18 2026-08-01T11:39:17Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
18 2026-08-01T11:39:19Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
18 2026-08-01T11:39:21Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
18 2026-08-01T11:39:24Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
18 2026-08-01T11:39:26Z I11 PASS leaked=[]
19 2026-08-01T11:40:18Z KILL-CTRL PASS controller recovered after kill
19 2026-08-01T11:41:02Z REDEPLOY PASS homebox redeployed
19 2026-08-01T11:41:44Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.240196615s", "last_run": "2026-08-01T11:41:43.294094532Z", "success": true}, "enabled": true, "running": false}
19 2026-08-01T11:42:48Z I7 PASS app=rallly want=C10-C019-A-134102 got=C10-C019-A-134102 snap=helyi restore_ok=True rto=42.1s
19 2026-08-01T11:45:03Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
19 2026-08-01T11:45:06Z I3 PASS adatok bound_under_parent=False while absent
19 2026-08-01T11:47:20Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
19 2026-08-01T11:47:22Z I4 PASS guest init boot before=1785580975 after=1785580975 (must be equal)
19 2026-08-01T11:49:38Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
19 2026-08-01T11:49:41Z I3 PASS mentes bound_under_parent=False while absent
19 2026-08-01T11:51:54Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
19 2026-08-01T11:51:56Z I4 PASS guest init boot before=1785580975 after=1785580975 (must be equal)
19 2026-08-01T11:52:25Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.074516344s", "last_run": "2026-08-01T11:52:25.638816606Z", "success": true}, "enabled": true, "running": false}
19 2026-08-01T11:52:28Z I5 PASS degraded=False absent_events=0 target=felhom-backup
19 2026-08-01T11:52:28Z I6 PASS healthy state must carry no degraded message; got=None
19 2026-08-01T11:52:30Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
19 2026-08-01T11:52:30Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
19 2026-08-01T11:52:32Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
19 2026-08-01T11:52:34Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
19 2026-08-01T11:52:36Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
19 2026-08-01T11:52:39Z I11 PASS leaked=[]
20 2026-08-01T11:53:24Z BACKUP PASS {"db_dump": {"count": 1, "duration": "33.58133879s", "last_run": "2026-08-01T11:53:17.277403988Z", "success": true}, "enabled": true, "running": false}
20 2026-08-01T11:55:40Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
20 2026-08-01T11:55:43Z I3 PASS mentes bound_under_parent=False while absent
20 2026-08-01T11:57:56Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
20 2026-08-01T11:57:58Z I4 PASS guest init boot before=1785580975 after=1785580975 (must be equal)
20 2026-08-01T12:00:14Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
20 2026-08-01T12:00:17Z I3 PASS adatok bound_under_parent=False while absent
20 2026-08-01T12:02:30Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
20 2026-08-01T12:02:33Z I4 PASS guest init boot before=1785580975 after=1785580975 (must be equal)
20 2026-08-01T12:03:17Z REDEPLOY PASS grafana redeployed
20 2026-08-01T12:03:58Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.633748751s", "last_run": "2026-08-01T12:03:57.032814639Z", "success": true}, "enabled": true, "running": false}
20 2026-08-01T12:05:02Z I7 PASS app=rallly want=C10-C020-A-140317 got=C10-C020-A-140317 snap=helyi restore_ok=True rto=42.2s
20 2026-08-01T12:05:52Z KILL-CTRL PASS controller recovered after kill
20 2026-08-01T12:05:55Z I5 PASS degraded=False absent_events=0 target=felhom-backup
20 2026-08-01T12:05:55Z I6 PASS healthy state must carry no degraded message; got=None
20 2026-08-01T12:05:57Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
20 2026-08-01T12:05:57Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
20 2026-08-01T12:05:59Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
20 2026-08-01T12:06:01Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
20 2026-08-01T12:06:03Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
20 2026-08-01T12:06:05Z I11 PASS leaked=[]
21 2026-08-01T12:06:37Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.742931297s", "last_run": "2026-08-01T12:06:36.352784544Z", "success": true}, "enabled": true, "running": false}
21 2026-08-01T12:07:26Z KILL-CTRL PASS controller recovered after kill
21 2026-08-01T12:09:42Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
21 2026-08-01T12:09:45Z I3 PASS adatok bound_under_parent=False while absent
21 2026-08-01T12:11:58Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
21 2026-08-01T12:12:01Z I4 PASS guest init boot before=1785580975 after=1785580975 (must be equal)
21 2026-08-01T12:12:45Z REDEPLOY PASS homebox redeployed
21 2026-08-01T12:15:01Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
21 2026-08-01T12:15:04Z I3 PASS mentes bound_under_parent=False while absent
21 2026-08-01T12:17:17Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
21 2026-08-01T12:17:19Z I4 PASS guest init boot before=1785580975 after=1785580975 (must be equal)
21 2026-08-01T12:18:01Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.215452442s", "last_run": "2026-08-01T12:18:01.043730281Z", "success": true}, "enabled": true, "running": false}
21 2026-08-01T12:19:03Z I7 PASS app=rallly want=C10-C021-A-141719 got=C10-C021-A-141719 snap=helyi restore_ok=True rto=41.4s
21 2026-08-01T12:19:43Z REBOOT PASS controller back after guest reboot
21 2026-08-01T12:19:46Z I5 PASS degraded=False absent_events=0 target=felhom-backup
21 2026-08-01T12:19:46Z I6 PASS healthy state must carry no degraded message; got=None
21 2026-08-01T12:19:48Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
21 2026-08-01T12:19:48Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
21 2026-08-01T12:19:50Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
21 2026-08-01T12:19:52Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
21 2026-08-01T12:19:54Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
21 2026-08-01T12:19:56Z I11 PASS leaked=[]
22 2026-08-01T12:20:51Z BACKUP PASS {"db_dump": {"count": 1, "duration": "35.295073244s", "last_run": "2026-08-01T12:20:46.946104153Z", "success": true}, "enabled": true, "running": false}
22 2026-08-01T12:21:54Z I7 PASS app=rallly want=C10-C022-A-141958 got=C10-C022-A-141958 snap=helyi restore_ok=True rto=41.9s
22 2026-08-01T12:22:38Z REDEPLOY PASS homebox redeployed
22 2026-08-01T12:23:20Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.506763469s", "last_run": "2026-08-01T12:23:08.027088655Z", "success": true}, "enabled": true, "running": false}
22 2026-08-01T12:25:35Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
22 2026-08-01T12:25:38Z I3 PASS adatok bound_under_parent=False while absent
22 2026-08-01T12:27:51Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
22 2026-08-01T12:27:54Z I4 PASS guest init boot before=1785586762 after=1785586762 (must be equal)
22 2026-08-01T12:28:43Z KILL-CTRL PASS controller recovered after kill
22 2026-08-01T12:30:58Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
22 2026-08-01T12:31:01Z I3 PASS mentes bound_under_parent=False while absent
22 2026-08-01T12:33:14Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
22 2026-08-01T12:33:17Z I4 PASS guest init boot before=1785586762 after=1785586762 (must be equal)
22 2026-08-01T12:33:20Z I5 PASS degraded=False absent_events=0 target=felhom-backup
22 2026-08-01T12:33:20Z I6 PASS healthy state must carry no degraded message; got=None
22 2026-08-01T12:33:22Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
22 2026-08-01T12:33:22Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
22 2026-08-01T12:33:24Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
22 2026-08-01T12:33:26Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
22 2026-08-01T12:33:28Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
22 2026-08-01T12:33:30Z I11 PASS leaked=[]
23 2026-08-01T12:34:02Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.296490438s", "last_run": "2026-08-01T12:34:02.179324267Z", "success": true}, "enabled": true, "running": false}
23 2026-08-01T12:36:18Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
23 2026-08-01T12:36:21Z I3 PASS mentes bound_under_parent=False while absent
23 2026-08-01T12:38:34Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
23 2026-08-01T12:38:37Z I4 PASS guest init boot before=1785586762 after=1785586762 (must be equal)
23 2026-08-01T12:39:17Z BACKUP PASS {"db_dump": {"count": 1, "duration": "26.836521386s", "last_run": "2026-08-01T12:39:17.164422568Z", "success": true}, "enabled": true, "running": false}
23 2026-08-01T12:40:21Z I7 PASS app=rallly want=C10-C023-A-143837 got=C10-C023-A-143837 snap=helyi restore_ok=True rto=41.9s
23 2026-08-01T12:41:11Z KILL-CTRL PASS controller recovered after kill
23 2026-08-01T12:41:56Z REDEPLOY PASS grafana redeployed
23 2026-08-01T12:44:11Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
23 2026-08-01T12:44:14Z I3 PASS adatok bound_under_parent=False while absent
23 2026-08-01T12:46:27Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
23 2026-08-01T12:46:30Z I4 PASS guest init boot before=1785586762 after=1785586762 (must be equal)
23 2026-08-01T12:46:32Z I5 PASS degraded=False absent_events=0 target=felhom-backup
23 2026-08-01T12:46:32Z I6 PASS healthy state must carry no degraded message; got=None
23 2026-08-01T12:46:34Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
23 2026-08-01T12:46:34Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
23 2026-08-01T12:46:36Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
23 2026-08-01T12:46:38Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
23 2026-08-01T12:46:40Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
23 2026-08-01T12:46:43Z I11 PASS leaked=[]
24 2026-08-01T12:47:26Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.434160964s", "last_run": "2026-08-01T12:47:25.920976261Z", "success": true}, "enabled": true, "running": false}
24 2026-08-01T12:48:31Z I7 PASS app=rallly want=C10-C024-A-144645 got=C10-C024-A-144645 snap=helyi restore_ok=True rto=42.5s
24 2026-08-01T12:49:02Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.477758487s", "last_run": "2026-08-01T12:49:00.748171817Z", "success": true}, "enabled": true, "running": false}
24 2026-08-01T12:51:17Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
24 2026-08-01T12:51:20Z I3 PASS mentes bound_under_parent=False while absent
24 2026-08-01T12:53:33Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
24 2026-08-01T12:53:36Z I4 PASS guest init boot before=1785586762 after=1785586762 (must be equal)
24 2026-08-01T12:54:25Z KILL-CTRL PASS controller recovered after kill
24 2026-08-01T12:56:40Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
24 2026-08-01T12:56:44Z I3 PASS adatok bound_under_parent=False while absent
24 2026-08-01T12:58:57Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
24 2026-08-01T12:59:00Z I4 PASS guest init boot before=1785586762 after=1785586762 (must be equal)
24 2026-08-01T12:59:45Z REDEPLOY PASS grafana redeployed
24 2026-08-01T12:59:47Z I5 PASS degraded=False absent_events=0 target=felhom-backup
24 2026-08-01T12:59:47Z I6 PASS healthy state must carry no degraded message; got=None
24 2026-08-01T12:59:50Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
24 2026-08-01T12:59:50Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
24 2026-08-01T12:59:52Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
24 2026-08-01T12:59:54Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
24 2026-08-01T12:59:56Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
24 2026-08-01T12:59:59Z I11 PASS leaked=[]
25 2026-08-01T13:02:17Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
25 2026-08-01T13:02:20Z I3 PASS adatok bound_under_parent=False while absent
25 2026-08-01T13:04:33Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
25 2026-08-01T13:04:36Z I4 PASS guest init boot before=1785586762 after=1785586762 (must be equal)
25 2026-08-01T13:05:26Z KILL-CTRL PASS controller recovered after kill
25 2026-08-01T13:07:41Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
25 2026-08-01T13:07:45Z I3 PASS mentes bound_under_parent=False while absent
25 2026-08-01T13:09:58Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
25 2026-08-01T13:10:00Z I4 PASS guest init boot before=1785586762 after=1785586762 (must be equal)
25 2026-08-01T13:10:40Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.129870607s", "last_run": "2026-08-01T13:10:40.81939141Z", "success": true}, "enabled": true, "running": false}
25 2026-08-01T13:11:44Z I7 PASS app=rallly want=C10-C025-A-151000 got=C10-C025-A-151000 snap=helyi restore_ok=True rto=42.0s
25 2026-08-01T13:12:26Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.439198049s", "last_run": "2026-08-01T13:12:14.38822499Z", "success": true}, "enabled": true, "running": false}
25 2026-08-01T13:13:10Z REDEPLOY PASS homebox redeployed
25 2026-08-01T13:13:13Z I5 PASS degraded=False absent_events=0 target=felhom-backup
25 2026-08-01T13:13:13Z I6 PASS healthy state must carry no degraded message; got=None
25 2026-08-01T13:13:15Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
25 2026-08-01T13:13:15Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
25 2026-08-01T13:13:17Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
25 2026-08-01T13:13:19Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
25 2026-08-01T13:13:21Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
25 2026-08-01T13:13:24Z I11 PASS leaked=[]
26 2026-08-01T13:14:10Z REDEPLOY PASS grafana redeployed
26 2026-08-01T13:16:26Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
26 2026-08-01T13:16:29Z I3 PASS adatok bound_under_parent=False while absent
26 2026-08-01T13:18:42Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
26 2026-08-01T13:18:44Z I4 PASS guest init boot before=1785586762 after=1785586762 (must be equal)
26 2026-08-01T13:19:25Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.181611353s", "last_run": "2026-08-01T13:19:24.6968582Z", "success": true}, "enabled": true, "running": false}
26 2026-08-01T13:20:31Z I7 PASS app=rallly want=C10-C026-A-151844 got=C10-C026-A-151844 snap=helyi restore_ok=True rto=42.9s
26 2026-08-01T13:21:20Z KILL-CTRL PASS controller recovered after kill
26 2026-08-01T13:23:36Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
26 2026-08-01T13:23:39Z I3 PASS mentes bound_under_parent=False while absent
26 2026-08-01T13:25:52Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
26 2026-08-01T13:25:55Z I4 PASS guest init boot before=1785586762 after=1785586762 (must be equal)
26 2026-08-01T13:26:24Z BACKUP PASS {"db_dump": {"count": 1, "duration": "27.266368272s", "last_run": "2026-08-01T13:26:24.55559188Z", "success": true}, "enabled": true, "running": false}
26 2026-08-01T13:26:27Z I5 PASS degraded=False absent_events=0 target=felhom-backup
26 2026-08-01T13:26:27Z I6 PASS healthy state must carry no degraded message; got=None
26 2026-08-01T13:26:29Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
26 2026-08-01T13:26:29Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
26 2026-08-01T13:26:31Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
26 2026-08-01T13:26:33Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
26 2026-08-01T13:26:35Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
26 2026-08-01T13:26:37Z I11 PASS leaked=[]
27 2026-08-01T13:27:22Z BACKUP PASS {"db_dump": {"count": 1, "duration": "34.006310162s", "last_run": "2026-08-01T13:27:15.871339683Z", "success": true}, "enabled": true, "running": false}
27 2026-08-01T13:28:14Z BACKUP PASS {"db_dump": {"count": 1, "duration": "33.534168063s", "last_run": "2026-08-01T13:28:08.289273214Z", "success": true}, "enabled": true, "running": false}
27 2026-08-01T13:29:17Z I7 PASS app=rallly want=C10-C027-A-152722 got=C10-C027-A-152722 snap=helyi restore_ok=True rto=41.8s
27 2026-08-01T13:31:33Z I1 PASS mentes absent -> got=backup_target_absent sev=error (want backup_target_absent/error)
27 2026-08-01T13:31:36Z I3 PASS mentes bound_under_parent=False while absent
27 2026-08-01T13:33:49Z I1-pair PASS mentes return -> got=backup_target_restored sev=info (want backup_target_restored/info)
27 2026-08-01T13:33:52Z I4 PASS guest init boot before=1785586762 after=1785586762 (must be equal)
27 2026-08-01T13:34:41Z KILL-CTRL PASS controller recovered after kill
27 2026-08-01T13:36:57Z I2 PASS adatok absent -> got=storage_disconnected sev=error (want storage_disconnected/error)
27 2026-08-01T13:37:00Z I3 PASS adatok bound_under_parent=False while absent
27 2026-08-01T13:39:14Z I2-pair PASS adatok return -> got=storage_reconnected sev=info (want storage_reconnected/info)
27 2026-08-01T13:39:16Z I4 PASS guest init boot before=1785586762 after=1785586762 (must be equal)
27 2026-08-01T13:40:00Z REDEPLOY PASS homebox redeployed
27 2026-08-01T13:40:03Z I5 PASS degraded=False absent_events=0 target=felhom-backup
27 2026-08-01T13:40:03Z I6 PASS healthy state must carry no degraded message; got=None
27 2026-08-01T13:40:04Z I10 PASS rallly/SECRET_PASSWORD type=secret must travel; in_unit=True
27 2026-08-01T13:40:04Z I10 PASS rallly/DB_PASSWORD type=secret must travel; in_unit=True
27 2026-08-01T13:40:07Z I10 PASS homebox/HBOX_AUTH_API_KEY_PEPPER type=secret must travel; in_unit=True
27 2026-08-01T13:40:09Z I10 PASS grafana/GF_SECURITY_ADMIN_PASSWORD type=password must NEVER travel; in_unit=False
27 2026-08-01T13:40:11Z I10 PASS papra/AUTH_SECRET type=secret must travel; in_unit=True
27 2026-08-01T13:40:13Z I11 PASS leaked=[]
Can't render this file because it contains an unexpected character in line 7 and column 37.
documentation/tests/campaign10-evidence-2026-07-31/state/rto.tsv
+28
View File
@@ -0,0 +1,28 @@
1 2026-08-01T07:25:51Z rallly tier1 42.3 True
1 2026-08-01T07:41:34Z rallly tier1 41.8 True
2 2026-08-01T07:55:56Z rallly tier1 41.7 True
3 2026-08-01T08:14:04Z rallly tier1 44.3 True
4 2026-08-01T08:22:14Z rallly tier1 42.0 True
5 2026-08-01T08:41:13Z rallly tier1 43.0 True
6 2026-08-01T08:44:12Z rallly tier1 41.6 True
7 2026-08-01T09:07:38Z rallly tier1 42.2 True
8 2026-08-01T09:12:45Z rallly tier1 42.3 True
9 2026-08-01T09:25:22Z rallly tier1 41.9 True
10 2026-08-01T09:48:24Z rallly tier1 41.6 True
11 2026-08-01T09:51:05Z rallly tier1 41.7 True
12 2026-08-01T10:15:50Z rallly tier1 41.6 True
13 2026-08-01T10:19:22Z rallly tier1 42.2 True
14 2026-08-01T10:31:54Z rallly tier1 42.4 True
15 2026-08-01T10:52:04Z rallly tier1 41.8 True
16 2026-08-01T11:09:59Z rallly tier1 38.8 True
17 2026-08-01T11:21:26Z rallly tier1 41.5 True
18 2026-08-01T11:39:12Z rallly tier1 42.0 True
19 2026-08-01T11:42:48Z rallly tier1 42.1 True
20 2026-08-01T12:05:02Z rallly tier1 42.2 True
21 2026-08-01T12:19:03Z rallly tier1 41.4 True
22 2026-08-01T12:21:54Z rallly tier1 41.9 True
23 2026-08-01T12:40:21Z rallly tier1 41.9 True
24 2026-08-01T12:48:31Z rallly tier1 42.5 True
25 2026-08-01T13:11:44Z rallly tier1 42.0 True
26 2026-08-01T13:20:31Z rallly tier1 42.9 True
27 2026-08-01T13:29:17Z rallly tier1 41.8 True
1 1 2026-08-01T07:25:51Z rallly tier1 42.3 True
2 1 2026-08-01T07:41:34Z rallly tier1 41.8 True
3 2 2026-08-01T07:55:56Z rallly tier1 41.7 True
4 3 2026-08-01T08:14:04Z rallly tier1 44.3 True
5 4 2026-08-01T08:22:14Z rallly tier1 42.0 True
6 5 2026-08-01T08:41:13Z rallly tier1 43.0 True
7 6 2026-08-01T08:44:12Z rallly tier1 41.6 True
8 7 2026-08-01T09:07:38Z rallly tier1 42.2 True
9 8 2026-08-01T09:12:45Z rallly tier1 42.3 True
10 9 2026-08-01T09:25:22Z rallly tier1 41.9 True
11 10 2026-08-01T09:48:24Z rallly tier1 41.6 True
12 11 2026-08-01T09:51:05Z rallly tier1 41.7 True
13 12 2026-08-01T10:15:50Z rallly tier1 41.6 True
14 13 2026-08-01T10:19:22Z rallly tier1 42.2 True
15 14 2026-08-01T10:31:54Z rallly tier1 42.4 True
16 15 2026-08-01T10:52:04Z rallly tier1 41.8 True
17 16 2026-08-01T11:09:59Z rallly tier1 38.8 True
18 17 2026-08-01T11:21:26Z rallly tier1 41.5 True
19 18 2026-08-01T11:39:12Z rallly tier1 42.0 True
20 19 2026-08-01T11:42:48Z rallly tier1 42.1 True
21 20 2026-08-01T12:05:02Z rallly tier1 42.2 True
22 21 2026-08-01T12:19:03Z rallly tier1 41.4 True
23 22 2026-08-01T12:21:54Z rallly tier1 41.9 True
24 23 2026-08-01T12:40:21Z rallly tier1 41.9 True
25 24 2026-08-01T12:48:31Z rallly tier1 42.5 True
26 25 2026-08-01T13:11:44Z rallly tier1 42.0 True
27 26 2026-08-01T13:20:31Z rallly tier1 42.9 True
28 27 2026-08-01T13:29:17Z rallly tier1 41.8 True
documentation/tests/campaign10-evidence-2026-07-31/state/runner.log
+60
View File
@@ -0,0 +1,60 @@
09:17:54 === Campaign 10 Phase B start — max 1 cycles, deadline 10:17 ===
09:17:56 cycle 1: detach_nontarget -> redeploy_app -> backup -> restore_verify -> detach_target -> kill_controller
09:31:19 cycle 1 done — violations so far: 0
09:31:19 === finished after 1 cycles, 0 violations ===
09:33:35 === Campaign 10 Phase B start — max 40 cycles, deadline 15:33 ===
09:33:37 cycle 1: detach_nontarget -> redeploy_app -> backup -> restore_verify -> detach_target -> kill_controller
09:47:09 cycle 1 done — violations so far: 0
09:47:11 cycle 2: kill_controller -> detach_nontarget -> backup -> redeploy_app -> restore_verify -> detach_target
10:00:42 cycle 2 done — violations so far: 0
10:00:44 cycle 3: detach_target -> redeploy_app -> backup -> detach_nontarget -> kill_controller -> restore_verify
10:14:17 cycle 3 done — violations so far: 0
10:14:19 cycle 4: detach_nontarget -> backup -> redeploy_app -> restore_verify -> detach_target -> kill_controller
10:27:50 cycle 4 done — violations so far: 0
10:27:52 cycle 5: redeploy_app -> detach_nontarget -> kill_controller -> detach_target -> backup -> restore_verify
10:41:26 cycle 5 done — violations so far: 0
10:41:28 cycle 6: kill_controller -> restore_verify -> detach_target -> detach_nontarget -> backup -> redeploy_app
10:54:59 cycle 6 done — violations so far: 0
10:55:01 cycle 7: detach_nontarget -> detach_target -> kill_controller -> redeploy_app -> restore_verify -> backup -> reboot_guest
11:09:13 cycle 7 done — violations so far: 0
11:09:15 cycle 8: redeploy_app -> kill_controller -> restore_verify -> detach_nontarget -> detach_target -> backup
11:22:48 cycle 8 done — violations so far: 0
11:22:50 cycle 9: kill_controller -> restore_verify -> redeploy_app -> detach_nontarget -> backup -> detach_target
11:35:57 cycle 9 done — violations so far: 0
11:35:59 cycle 10: detach_nontarget -> redeploy_app -> kill_controller -> detach_target -> restore_verify -> backup
11:49:07 cycle 10 done — violations so far: 0
11:49:10 cycle 11: restore_verify -> detach_nontarget -> detach_target -> backup -> redeploy_app -> kill_controller
12:02:40 cycle 11 done — violations so far: 0
12:02:43 cycle 12: redeploy_app -> detach_nontarget -> kill_controller -> detach_target -> backup -> restore_verify
12:16:02 cycle 12 done — violations so far: 0
12:16:05 cycle 13: kill_controller -> redeploy_app -> restore_verify -> backup -> detach_target -> detach_nontarget
12:29:25 cycle 13 done — violations so far: 0
12:29:27 cycle 14: backup -> restore_verify -> detach_nontarget -> redeploy_app -> kill_controller -> detach_target -> reboot_guest
12:43:28 cycle 14 done — violations so far: 0
12:43:30 cycle 15: detach_target -> backup -> kill_controller -> redeploy_app -> restore_verify -> detach_nontarget
12:56:54 cycle 15 done — violations so far: 0
12:56:56 cycle 16: backup -> detach_target -> detach_nontarget -> restore_verify -> kill_controller -> redeploy_app
13:12:43 cycle 16 done — violations so far: 0
13:12:49 cycle 17: redeploy_app -> detach_nontarget -> backup -> kill_controller -> restore_verify -> detach_target
13:26:12 cycle 17 done — violations so far: 0
13:26:14 cycle 18: detach_target -> kill_controller -> detach_nontarget -> backup -> redeploy_app -> restore_verify
13:39:26 cycle 18 done — violations so far: 0
13:39:28 cycle 19: kill_controller -> redeploy_app -> restore_verify -> detach_nontarget -> detach_target -> backup
13:52:39 cycle 19 done — violations so far: 0
13:52:41 cycle 20: backup -> detach_target -> detach_nontarget -> redeploy_app -> restore_verify -> kill_controller
14:06:05 cycle 20 done — violations so far: 0
14:06:07 cycle 21: backup -> kill_controller -> detach_nontarget -> redeploy_app -> detach_target -> restore_verify -> reboot_guest
14:19:56 cycle 21 done — violations so far: 0
14:19:58 cycle 22: restore_verify -> redeploy_app -> backup -> detach_nontarget -> kill_controller -> detach_target
14:33:30 cycle 22 done — violations so far: 0
14:33:32 cycle 23: backup -> detach_target -> restore_verify -> kill_controller -> redeploy_app -> detach_nontarget
14:46:43 cycle 23 done — violations so far: 0
14:46:45 cycle 24: restore_verify -> backup -> detach_target -> kill_controller -> detach_nontarget -> redeploy_app
14:59:59 cycle 24 done — violations so far: 0
15:00:01 cycle 25: detach_nontarget -> kill_controller -> detach_target -> restore_verify -> backup -> redeploy_app
15:13:24 cycle 25 done — violations so far: 0
15:13:26 cycle 26: redeploy_app -> detach_nontarget -> restore_verify -> kill_controller -> detach_target -> backup
15:26:37 cycle 26 done — violations so far: 0
15:26:39 cycle 27: backup -> restore_verify -> detach_target -> kill_controller -> detach_nontarget -> redeploy_app
15:40:13 cycle 27 done — violations so far: 0
15:40:13 === finished after 27 cycles, 0 violations ===
documentation/tests/campaign10-evidence-2026-07-31/state/status.txt
+4
View File
@@ -0,0 +1,4 @@
cycle=27
phase=finished
utc=2026-08-01T13:40:13Z
violations=0