Files
felhom.eu/documentation/tests/campaign2-evidence-2026-07-07/evidence/T-SOAK-LOOP/invariants.log
T
admin 7714f1f6e4 docs(tests): Campaign 2 report + evidence — DR primitives, self-update, storage, faults, endurance
Full-system validation on demo 9201 (controller v0.102.0 / agent v0.73.0). Headline:
ONE genuine pilot-critical finding — the non-root agent cannot pct-restore the
standard Felhom guest because it has bind-mount mountpoints (mp8 /mnt/felhom-drives,
mp9 bootstrap) and pct restore refuses bind mounts unless root. This breaks BOTH
--selftest=restore-test AND the offsite DR bring-up (R1), linked to a missing
VM.Audit/VM.Allocate on the agent token (R2). The large raw FAIL count is dominated
by one harness bug (auth-session expiry mis-scoring P3-P9) plus D1-injection/soak-probe/
dryrun-journal/escrow-blob calibration issues; the product where actually exercised was
healthy with ZERO resource leaks over ~7h. Demo node healed to pre-campaign state
(auth off, notifications restored, no scratch leaks). No production code changed.

Evidence snapshot + README under campaign2-evidence-2026-07-07/ (cookie jars scrubbed).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
2026-07-07 14:26:52 +02:00

171 lines
14 KiB
Plaintext

1783411187 iter=1 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783411271 iter=2 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=53)
1783411357 iter=3 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=49)
1783411443 iter=4 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=47)
1783411528 iter=5 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=42)
1783411612 iter=6 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=50)
1783411699 iter=7 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=44)
1783411783 iter=8 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=43)
1783411868 iter=9 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=49)
1783411953 iter=10 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783412039 iter=11 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=46)
1783412124 iter=12 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783412208 iter=13 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=44)
1783412293 iter=14 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=44)
1783412378 iter=15 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=54)
1783412463 iter=16 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783412548 iter=17 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=56)
1783412634 iter=18 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=47)
1783412719 iter=19 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=44)
1783412803 iter=20 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=47)
1783412890 iter=21 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=47)
1783412975 iter=22 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=47)
1783413060 iter=23 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=42)
1783413145 iter=24 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=50)
1783413230 iter=25 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=46)
1783413315 iter=26 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=50)
1783413400 iter=27 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=44)
1783413485 iter=28 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783413570 iter=29 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783413655 iter=30 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=51)
1783413740 iter=31 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=49)
1783413827 iter=32 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783413912 iter=33 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=46)
1783413997 iter=34 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=53)
1783414082 iter=35 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=42)
1783414167 iter=36 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=50)
1783414252 iter=37 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=44)
1783414337 iter=38 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783414422 iter=39 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=47)
1783414507 iter=40 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=50)
1783414592 iter=41 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=46)
1783414677 iter=42 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783414763 iter=43 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783414847 iter=44 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=44)
1783414932 iter=45 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=46)
1783415017 iter=46 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783415102 iter=47 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=42)
1783415187 iter=48 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783415272 iter=49 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783415357 iter=50 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=49)
1783415441 iter=51 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=52)
1783415526 iter=52 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=53)
1783415611 iter=53 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=52)
1783415696 iter=54 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=44)
1783415781 iter=55 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=47)
1783415866 iter=56 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783415952 iter=57 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=47)
1783416037 iter=58 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783416122 iter=59 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783416208 iter=60 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=44)
1783416293 iter=61 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783416377 iter=62 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783416462 iter=63 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783416550 iter=64 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783416635 iter=65 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=43)
1783416719 iter=66 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=89)
1783416805 iter=67 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=51)
1783416890 iter=68 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783416975 iter=69 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=46)
1783417060 iter=70 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783417146 iter=71 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=43)
1783417231 iter=72 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783417316 iter=73 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=46)
1783417401 iter=74 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783417486 iter=75 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=42)
1783417570 iter=76 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783417655 iter=77 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=41)
1783417743 iter=78 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783417827 iter=79 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=43)
1783417912 iter=80 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783417996 iter=81 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783418082 iter=82 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=43)
1783418167 iter=83 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783418252 iter=84 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783418338 iter=85 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=50)
1783418423 iter=86 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=43)
1783418508 iter=87 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=42)
1783418593 iter=88 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783418678 iter=89 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783418763 iter=90 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783418847 iter=91 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=46)
1783418933 iter=92 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=44)
1783419018 iter=93 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=58)
1783419103 iter=94 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=51)
1783419188 iter=95 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=47)
1783419273 iter=96 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783419358 iter=97 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=46)
1783419443 iter=98 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=49)
1783419528 iter=99 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=50)
1783419613 iter=100 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=58)
1783419698 iter=101 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=49)
1783419784 iter=102 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=50)
1783419869 iter=103 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=44)
1783419954 iter=104 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=46)
1783420039 iter=105 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783420125 iter=106 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=46)
1783420210 iter=107 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=49)
1783420295 iter=108 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=46)
1783420379 iter=109 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=50)
1783420466 iter=110 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=44)
1783420551 iter=111 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783420635 iter=112 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=46)
1783420721 iter=113 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=49)
1783420806 iter=114 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=43)
1783420891 iter=115 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=52)
1783420975 iter=116 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=44)
1783421061 iter=117 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=51)
1783421146 iter=118 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=42)
1783421231 iter=119 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=47)
1783421316 iter=120 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783421401 iter=121 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783421486 iter=122 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=41)
1783421571 iter=123 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=51)
1783421657 iter=124 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=41)
1783421741 iter=125 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783421826 iter=126 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=47)
1783421912 iter=127 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=46)
1783421997 iter=128 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=49)
1783422082 iter=129 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783422166 iter=130 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=51)
1783422252 iter=131 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=50)
1783422337 iter=132 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783422421 iter=133 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=51)
1783422507 iter=134 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=49)
1783422592 iter=135 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783422676 iter=136 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783422762 iter=137 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=49)
1783422848 iter=138 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=44)
1783422933 iter=139 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=51)
1783423018 iter=140 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=51)
1783423104 iter=141 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=51)
1783423189 iter=142 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783423274 iter=143 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=49)
1783423359 iter=144 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=50)
1783423445 iter=145 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783423530 iter=146 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783423614 iter=147 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=41)
1783423698 iter=148 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783423783 iter=149 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=42)
1783423868 iter=150 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783423953 iter=151 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=49)
1783424039 iter=152 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=44)
1783424124 iter=153 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=43)
1783424208 iter=154 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=47)
1783424293 iter=155 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783424378 iter=156 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=50)
1783424463 iter=157 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=47)
1783424548 iter=158 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=44)
1783424634 iter=159 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=51)
1783424719 iter=160 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=45)
1783424803 iter=161 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=49)
1783424888 iter=162 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=46)
1783424974 iter=163 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=47)
1783425059 iter=164 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783425143 iter=165 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=50)
1783425230 iter=166 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=47)
1783425315 iter=167 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=46)
1783425400 iter=168 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=48)
1783425485 iter=169 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=50)
1783425571 iter=170 op=canary app=mealie ok=0 reasons= responsive(code=302 ms=54)