9.2 KiB
Part 3 — the data-loss candidates: research (night of 2026-10-22)
Research by a helper (read only; it changed nothing). Saved by the main session from the helper's report (the helper's
own write was refused by the harness). Baselines: controller cbc944c (v0.339.0), agent 2d5840b (v0.162.0), felhom.eu
1b442bcc. Architecture read: 07 §6.1.1, §10.2 (rows R-359, R-87), the design headers in R-899 nightowed.go,
nightchain.go, offbox_clear.go, R-822 offbox_window.go, hub F-14 pbsdr.go. Live reads: demo-hp's agent journal
(backup lines — demohp-whole-box-completions.txt) and docker ps on 9202. The main session's proofs and fixes are
recorded in the night report; this file is the input.
| Item | Verdict | Reason | Action |
|---|---|---|---|
| 1187a symlink | REAL (code) | appexport/fabplan.go:292-310 follows a link; the tar write fails; export.go:712 drops the whole drive folder; the export still ends „done" |
fix + red-proof, P2 |
| 1187b owner/symlink | REAL (code) | appexport/restore.go:1138-1183 handles only dirs and regular files, no chown |
fold with (a) |
| 1187c stopped-app import | REAL (code) | restore.go:355-374 removes the volumes only when the app RUNS; docker cp merges |
9202 proof, then refuse |
| 1187d | PLAUSIBLE | no restoreOpBlocked check in StartImport |
fold into (c) |
| 1187f | REAL (code) | the bundle's HDD_PATH is used; an empty one gives a relative path | stays in the row |
| 1187g | REAL (code) | classifyDBImage does not know postgis (adventurelog, dawarich); psql without ON_ERROR_STOP |
small fix, 9202 proof |
| 1188b | REAL in code, unlikely | backup/restore.go:218 drops the rm error, then untars into the old volume |
P4 |
| 1188c | ALREADY FIXED | v0.337.0, TestR1188c_ALeftoverTempDoesNotBlockTheRetry |
close |
| 1188d | REAL (wording) | 30-min cap; scratch is kept, so a second press probably continues (not measured) | measure, then the sentence |
| 1188e | REAL, P4 | offbox_restore.go:675-679 reads latest, _ = id |
P4 |
| 1189a | REAL, half-live | agent localapi/server.go:1179-1192: a future archive time gives a negative age |
agent fix needs the vouch |
| 1189b | SPLIT | the skipped nights after an off-cycle run: FIXED by R-899 (v0.304.0), shown in the journal; the ~3 min/night creep is still real | close that half; creep → P4 |
| 1189c | REAL in code | quiesce waits for the catch-up and the update leg, not for a restore | 9202 recipe |
| 1189d | REAL (code) | proof 05:30 and integrity 06:00 are daily jobs; the catch-up runs only the 3 backup legs; no hub age alarm | own row, ~15-line fix |
| 1189e | REAL (code) | the hub checks „since midnight"; any HH:MM window passes | hub row |
| 1189f | OPERATOR DECISION | design question | — |
| 1189g | REAL, P4 | spring-forward 02:30 → 03:30 collides with tier2 | unit test |
| 1189h | not re-read | — | stays |
| 1189i | REAL, P4 UX | check and begin are not atomic (web/handlers.go:1860/1881) |
P4 |
| 1190a | REAL (code) | the ceremony stages the off-site password only if off-site is configured; hub „abc"→"" demotes the good package | unit tests, ~6-line fix |
| 1190b | REAL (code) | the global bar ignores recoveryOffer(); the start handler does not refuse |
409 fix, P2 |
| 1190c | PLAUSIBLE | — | read-only fingerprint compare for Tester4 |
| 1190d | OPERATOR DECISION | — | — |
| 1190e/f/h | not re-read | — | stay |
| 1190g | REAL (code) | a nil AppConfig drops the app from liveDriveBinds, so its live folder can be listed (and deleted) as kept |
fix, P2 |
| 1205 | OPERATOR DECISION | the F-14 gate is deliberate (ruling 2026-07-13) | two options to the operator |
| 1206 | no loss on the Felhom tier (by design, R-822); the NAS tier open | the guard refuses a prune while any snapshot is future-dated; the NAS tier prunes without the guard | pure unit test with the policy simulator |
R-1187(c) — a .fab import into a STOPPED app merges into the old volumes. REAL (code)
controller/internal/appexport/restore.go:355-374 removes the volumes (RemoveStackVolumes = compose down --volumes,
cmd/controller/main.go:3566-3591) only when IsStackRunning (main.go:3467-3471: Running or Degraded). Every other
state (household-stopped, crashed, error, starting) „overwrites config in place". restoreVolumeData (:736-772) then
runs volume create (a no-op on an existing volume) and docker cp - (:717-732), which adds and overwrites but
never deletes: the result is old ∪ bundle. The drive half (restoreHDDData :590-648, extractTar) merges in both
branches. Harm: a broken app (not running) is the likeliest import target; a stale SQLite -wal beside the bundle's
.db is a corruption shape. „No SQL dump by default" alone is NOT-A-FINDING: with stop ticked the DB volume is captured
cold, which is consistent (export.go:258-279, :366; appbackup/dbdump.go:142-144 lists only running containers).
Fix: executeImport, ~8 lines — (B) refuse an import over an installed app that is not running (now); (A) remove its
volumes too, only together with the D22 put-back copy. 9202 recipe (~20 min): a throwaway uptime-kuma; MARK-A into the
volume, export (stop ticked), MARK-N; control: import while RUNNING → MARK-N gone; MARK-N again, Stop, import → MARK-N
survives + the debug line „deployed but not running — will overwrite config in place".
R-1189(b) — split
The skipped nights after an off-cycle run are ALREADY FIXED by R-899, controller v0.304.0 (2026-10-09),
quiesce/nightowed.go, TestR899_* (6). Proven on demo-hp: before the fix every off-cycle run was followed by a
skipped night (10-01 22:10 → no 10-02; 10-05 11:19 → no 10-06; 10-08 22:20 → no 10-09); after it every one was followed
by a night run (10-17 16:33 → 10-18 04:33; 10-19 03:25 → 04:32; 10-20 08:42 → 10-21 04:32). The creep is still REAL,
low harm: agent localapi/server.go:1184 is age >= Cadence, age counted from the archive START; the controller polls
every 5 min (quiesce.go:150) and the archive starts after the apps are stopped. Measured with no press between:
04:31:07 → 04:33:18 → 04:36:47 → 04:38:14, and 04:33:37 → 04:38:44 → 04:42:40 (~1.5–5 min/night). Unpressed, the start
reaches W+6h in ~70–80 nights, then one night is skipped (44 h is below the 48 h valve, quiesce.go:807-818); the hub's
26 h line (deadline.go:29) alarms the operator. Fix: quiesce/tiers.go — inside the window a tier is due when
age >= cadence - 4h (~6 lines, no customer data); proof: a unit test with an injected clock.
R-1189(d) — proof and integrity never run on a box that is off at 05:30–06:00. REAL (code)
main.go:1613 Daily("offsite-integrity","06:00"), :1635 Daily("offsite-proof","05:30"); the scheduler picks the
next future time (scheduler.go:499-518) and skips late fires (:341-346); the catch-up runs only the three backup
legs (main.go:1544-1553, catchUpLegs :4138). The comment at main.go:1601-1603 („checked the next day it is on")
is true only if the box is on at 06:00. The hub shows offsite.last_integrity_* (web/offsite_box.go:199-210) with
no age alarm. Design: 07 §10.2, R-359 („due-ness, not a weekday"). Fix: an After-hook on nightchain.CatchUp running
the due-gated integrity check and proof once; the comment corrected (~15 lines); optional hub age alarm. Red-proof: a
CatchUp test with a never-05:30 clock and a recording hook.
R-1190(a) — a ceremony with off-site not configured seals no repo password and supersedes the good package. REAL (code)
web/escrow_handlers.go:217-224 stages only if OffboxConfigured() (backup/offbox.go:695-710) — false when the target
is cleared or disabled, although ClearOffboxTarget deliberately keeps the password file (offbox_clear.go header).
The agent then seals no password (cmd/felhom-agent/main.go:2981-3003) and uploads sha "". Hub SaveHostEscrow sees
curSHA != "" and demotes the good package to retained custody (operator-only). OffsiteRecoveryOffer reads "" as
„covers no key" (offbox.go:1855-1873) and says nothing. Fix: stage when the password file exists (~6 lines;
PushOffboxPasswordForEscrow offbox.go:800 reads only the file). Red-proof: a web test asserting the stage fn is
called; a hub store test pinning „abc"→"" ⇒ superseded.
Other mechanism lines
1187a fabplan.go:292-310 (a FIFO also hangs the export on os.Open) · 1187b restore.go:1138-1183 · 1187g
restore.go:1047-1056, 933-943 vs appbackup/dbservices.go:37 (the dump is --clean --if-exists, so ON_ERROR_STOP is
safe; copy dbdump.go:775-784) · 1188b backup/restore.go:218 (StopStack is compose down, stacks/manager.go:1444)
· 1188d web/offbox_handlers.go:455 (scratch kept, offbox_restore.go:373-392) · 1188e offbox_restore.go:675-679,
marker SnapshotID :458 · 1189c quiesce.go:213-275, :565 · 1189e hub monitor/deadline.go:416-419, 555-565 ·
1189i web/handlers.go:1860, 1881 · 1190b web/escrow_banner.go:23-35, templates/backups_remote.html:144-160
(its comment names the harm, R-199), escrow_handlers.go:180-245, reachable in offer shape (c) offbox.go:1866-1870 ·
1190g stacks/kept.go:220-235, 336-360, 372-389; deploy.go:1113-1125 (a parse error gives nil) · 1205 hub
web/pbsdr.go:236-248, store.go:3293-3297 (deliberate; operator ruling 2026-07-13) · 1206 backup/offbox_window.go:22-24,
130-137.