Compare commits

...

152 Commits

Author SHA1 Message Date
admin 2fa1efc5e5 docs(REPORT): confirming cycle on v0.216.0, and persistence proven live
gates / gates (push) Successful in 9s
- 09:31:35Z on 0.216.0: '2 disk(s) evaluated, 0 alert(s)' — the count now
  matches the 2 persisted records, closing the disagreement that exposed R-335.
- The 0.215.0 -> 0.216.0 redeploy replaced the container and the state file
  came back with a changed_at written by the PREVIOUS version, so the new
  container loaded the pre-restart record instead of re-baselining. Scenario L
  observed on real hardware, not just through the production-path unit test.
- R-332 narrowed accordingly: what remains unproven is an already-ALERTED disk
  not re-alerting after a restart.
2026-08-14 11:33:11 +02:00
admin 330e4a051e docs(REPORT): v0.215.0 -> v0.216.0 run report
gates / gates (push) Successful in 9s
Includes the two clean live cycles, the warning-vs-warn notification_log proof,
the 13 red-proof outcomes (A reported as a finding — the spec's mutation for it
is not a valid red-proof), and section 14 on R-335, the aliasing defect found
live in v0.215.0 and fixed in v0.216.0.
2026-08-14 10:33:22 +02:00
admin 90f2545679 fix(disk-health): one physical disk must be evaluated once per run (R-335)
gates / gates (push) Successful in 9s
Found on live hardware two hours after the v0.215.0 deploy, by noticing the
release's own positive observable disagreed with its own persisted artefact:
the check logged '3 disk(s) evaluated' while disk-health-state.json held two
records. demo-hp's c11-scratch and felhom-backup are the same NVMe and share
a durable id, so one disk was walked twice per run.

Not cosmetic. The loop writes a disk's record before the next entry reads it,
so the second copy of an aliased disk consumed the FIRST copy's write as its
prior: the disk sustained against ITSELF and reached Hiba on a first sighting,
defeating truth-table row 6 — the rule that separates a one-hour benign
excursion from a false critical. It would also have emitted two identical
events for one drive. Latent on demo-hp only because all counters are zero.

Each diskKey is now evaluated once per run. Both entries stay marked seen so
neither looks like a disappeared disk, and the card still renders both rows —
the dedup is about state and alerts, not display.

Red-proof run and reverted: deleting the guard makes the first sighting emit
Kind:2 (Hiba-from-sectors) at 8 sectors.
2026-08-14 10:30:22 +02:00
admin 8144a70a72 docs(v0.215.0): CHANGELOG, CONTEXT decisions, README feature, REUSE entries
gates / gates (push) Successful in 10s
- CHANGELOG leads with the severity fix and the live warning-vs-warn proof.
- CONTEXT records the settled decisions so they are not re-litigated: Hiba is
  the label for predicted failure (no fourth word); sustain before count and
  why; the provenance of 64/55/60; phase 2 owns the new SMART attributes
  because they are a wire change under G-1; phase 1 state is one record per
  disk, not a series.
- README documents the 14-row ladder, the persisted state, the hourly cadence
  and the five message shapes.
- REUSE pins the severity wire contract on PushEvent — the defect's real home,
  so the next typo'd severity is caught at the table rather than in production
  — and records priorFor vs cardPriorFor, which differ by one observation and
  make the chip disagree with the email if mixed up.
2026-08-14 08:34:31 +02:00
admin 34d83f5a02 feat(disk-health): poll hourly, not 6-hourly — measured, not assumed
gates / gates (push) Successful in 9s
Part 4 was gated on a measurement. On demo-hp (Tier 0) the controller's real
/disks fetch — fetchDisks, the same path the check uses, not the 60s card
cache — costs min 0.805s / median 0.821s / max 0.841s over 10 calls, all HTTP
200, across 3 physical disk rows (2 distinct devices). Median is 6x under the
5s bar, so the <5s branch applies and the interval drops 6h -> 1h.

Why it matters: the one real failing drive's benign excursion lasted about ONE
HOUR and cleared completely. A 6-hourly sampler can land either side of an
excursion like that, see nothing, and then catch the terminal run half a day
late. The smartd history that produced the whole analysis sampled every 30
minutes and only just resolved the shape.
2026-08-14 08:21:55 +02:00
admin c24f1920d9 test(disk-health): Group L must run TWO checks after the restart
One check cannot distinguish a loaded state from a silent re-baseline — a
forgetful controller is also silent on its first check. It betrays itself on
the second, when the rebuilt prior makes the disk look newly sustained and it
alerts all over again. Caught while building the companion red-proof: with the
state load skipped, the single-check version still passed.
2026-08-14 08:12:44 +02:00
admin bb50e1293c fix(disk-health): the alert that never sent — severity, a real Hiba level, and a memory that survives a restart
Three defects made the disk-health feature silent in exactly the case it
exists for. Evidence: felhom.eu documentation/audits/DIAG-smart-passed-trap-2026-08-14.md

1. SEVERITY (the one that changes whether anything arrives at all).
   NotifyDiskHealthDegraded emitted severity "warn", which is NOT in the
   hub's accepted set {info,warning,error,critical}. The hub coerced it to
   "info" (hub/internal/api/handler.go) and severityNotifies dropped it
   (hub/internal/notify/dispatcher.go), so every Figyelmeztetes-level disk
   alert was filed as an informational notice and emailed to NOBODY, on the
   customer and the operator leg alike. Now "warning". DiskAlertKind.Severity()
   is exported so the contract is checkable from any package.

2. NO LEVEL ABOVE "worth an eye". smart_status.passed CANNOT fail on
   unreadable sectors (attrs 187/197/198 all carry thresh 0 and a normalized
   value floors at 1), so Hiba was unreachable for this whole fault class.
   DiskVerdictFor now takes a DiskPrior and implements a 14-row top-down
   ladder: sustained unreadable sectors, a count too large to be a blip (64),
   unreadable+remapping together, overheating, NVMe critical flag or spent
   endurance all reach Hiba. No fourth label — predicted failure is "Hiba".

3. IT SPOKE ONCE, AND FORGOT ON RESTART. The baseline was in-memory, so a box
   that rebooted while a disk was failing never alerted again; and between 8
   and 352 sectors nothing was emitted at all. State is now persisted
   (disk-health-state.json, atomic tmp+rename), the decision compares against
   the last ALERTED verdict (collapsing flaps to one alert while letting a
   genuine escalation fire immediately), and a disk already at Hiba re-alerts
   once it has BOTH doubled its count and waited out a 24h cooldown.

The card replays the same prior the check used (diskRecord.PriorSawUncorrectable)
so the chip and the email cannot disagree — the property the shared verdict
function exists to guarantee, now pinned rather than asserted.

Tests: 12 scenario groups A-L. Group L builds the Server through web.NewServer,
the same call main.go makes, over a real file.
2026-08-14 08:10:59 +02:00
admin 3e3ee94b7b REPORT: live 422 proven on hardware; R-308 withdrawn (my quoting bug, not a stale credential)
gates / gates (push) Successful in 12s
2026-08-12 19:05:23 +02:00
admin ae10f64806 REPORT: controller v0.214.0 — the screen stops hedging, and the claim guard grew a surface
gates / gates (push) Successful in 18s
2026-08-12 18:49:56 +02:00
admin 3ed5e3e770 v0.214.0 — the recovery screen stops hedging about a code it can now check (R-311)
gates / gates (push) Successful in 13s
MinAgent: 0.129.0

What was already right: the screen did not bluntly accuse. R-222/R-226 hedged,
naming both causes and the kept package, and saying it could not tell them apart.
That was honest - and it could not tell them apart because nothing ever looked.
Agent v0.129.0 looks, so the hedge becomes an answer.

New class RecoveryCodeOpensRetained on HTTP 422, gated by
FeatureRetainedRecoveryClass (MinAgent 0.129.0). The gate is SEPARATE from the
R-224 one because the two name different agent versions and a box can sit between
them, where a 422 is a shape we did not design. ClassifyRecoveryFailure therefore
takes both flags; the compiler found every call site.

The message says the code is correct, names the supersession date, says the
earlier package is kept, and says the CURRENT backups are unaffected - the half a
customer will otherwise assume wrong. It promises NO restore: there is no
in-product route to a set-aside store (R-312) and the retained package may itself
predate the repository-password field. It routes to support, which can do it.

The claim guard grew a surface and immediately convicted something. It scanned
templates only, while every recovery message is a Go string in a handler - the
highest-stakes copy in the product, never scanned. It now scans recovery_handlers.go
too, and found a PRE-EXISTING unregistered claim on its first run.

Six handler tests asserting which SENTENCE the customer sees; red-proofs asserted
applied, including: 422 unconditional makes an agent that never looked read as
having looked, and routing 400 to the new class congratulates a mistype.
2026-08-12 18:42:02 +02:00
admin 3168a78935 REPORT: v0.213.0 pinned-fingerprint condition, red-proofs, claim guard
gates / gates (push) Successful in 12s
2026-08-12 15:39:08 +02:00
admin 89712563a0 R-302: the abandon banner promises only what the box can still see is true
gates / gates (push) Successful in 10s
The retrieval clause rendered unconditionally on every page and is false on a
reachable state - the same screen where the orphan card says we cannot tell.

The condition is a fingerprint PINNED at the decision, not a comparison against
the current key. The obvious proxy asks about the wrong key: the set-aside
copies were written under an older key the box no longer has, so on a
twice-rebuilt box the proxy promises about copies nothing can open. Demonstrated
- under the proxy, the replaced-package and legacy cases both flip back to
promising.

The pin is a recorded assumption and says so: nothing on the box records which
key wrote those copies. Empty is not a match. A countdown started before this
carries no pin and takes the cautious branch, not a backfill.

A sweep of all 36 templates found a fourth instance (backups page, same
condition applied) and a fifth (the confirmation screen, correctly left alone -
true at the moment of the decision).

New retrieval_promise_gate registers each claim with a reason rather than
banning a verb: a string ban failed twice, and the honest replacement copy
contains the stem.
2026-08-12 15:27:29 +02:00
admin 1b66010298 REPORT: v0.212.0 orphan card second promise
gates / gates (push) Successful in 16s
2026-08-12 14:05:45 +02:00
admin 68f3e12398 R-299: the orphan card's second promise, and a guard that matched one inflection
gates / gates (push) Successful in 14s
The explanation paragraph - the always-visible half of the card - still ended
'a hozzajuk tartozo helyreallitasi koddal kesobb visszaallithatok lehetnek',
the same unevaluable claim v0.211.0 removed from the confirm block below it.

It survived because the spec called that line accurate, and because the
regression guard asserted the SINGULAR form while the card carried the plural,
which does not contain that substring. The guard now matches the stem, so any
conjugation fails it. The two accurate halves are kept.

Also: the guard's failure message sliced rendered HTML at a byte offset and cut
Hungarian mid-character; it now slices on rune boundaries.
2026-08-12 13:49:15 +02:00
admin f87be3575f REPORT: correct installer publication status
gates / gates (push) Successful in 12s
2026-08-10 14:20:22 +02:00
admin 38f4535bfa REPORT: golden 0.211.0 baked and published; only the Day-0 vouch remains
gates / gates (push) Successful in 14s
2026-08-10 14:20:08 +02:00
admin 397d62136f REPORT: v0.211.0 written, not delivered - bake and Day-0 approval outstanding
gates / gates (push) Successful in 13s
2026-08-10 13:58:45 +02:00
admin 86a78c6767 R-294/R-295: orphan card stops promising restorability; one name per secret
gates / gates (push) Successful in 14s
The orphan card told a customer their set-aside off-site history may be
restorable later with their recovery code. The discriminator lives on the hub
and no wire field carries it, so the box rendering that card cannot evaluate the
promise. Copy replaced per the spec: state what happens, decline what we cannot
know and say why, name a route.

The claim page called the same three-word dashboard code two different names
depending on branch, one of which collides with the ten-word escrow code.
Retired 'Visszaallito kod'; the name is now constant and the sentence changes.
Naming only - a test pins that a reset code is still accepted.

secret_in_markup_gate no longer convicts Go template comments, which are
stripped before render; still convicts a real rendered secret.
2026-08-10 13:53:21 +02:00
admin b762a37097 R-280: attach list from mounted-but-unregistered filesystems; two-clicks promise made conditional
gates / gates (push) Successful in 17s
After a reinstall the data drive could not be re-attached through any dashboard
route: both candidate lists came from the agent's unclaimed-disk scan, and the
rebuilt box's drives are claimed. The restore page said it was two clicks while
pointing at an empty picker.

The attach list now also carries the controller's own mounted-but-unregistered
filesystems. initialize is untouched, so the format wizard's system/backup
protection is unchanged. The 'two clicks' sentence is conditional on the picker
being non-empty, and says something true and actionable when it is not.
2026-08-10 13:41:32 +02:00
admin c732fe1283 v0.210.0 — R-259 and R-258: two pictures that were not true
gates / gates (push) Successful in 18s
Both are one shape: something the box already knows, drawn as its opposite.

R-259 — A DISK WE FAILED TO READ WAS DRAWN AS A HEALTHY EMPTY DISK. readDiskUsage
(internal/system/info_linux.go) logged a statfs failure at DEBUG and returned, leaving the caller's
TotalGB/UsedGB/AvailGB/Percent at zero — and usageColor(0) is "nominal". The dashboard's
most-looked-at meter therefore rendered "0.0 GB / 0.0 GB (0%)" with a 0%-wide bar in the healthy
colour. "We could not look" and "there is plenty of room" were the same picture.

readDiskUsage now returns whether the measurement succeeded; SystemInfo gains DiskKnown and
HDDKnown (HDDConfigured is not a substitute: it says a path was configured, not that reading it
worked); and the template draws NO figure, NO percentage and NO meter fill when unknown, saying
"A tarhely merete most nem olvashato ki." instead. A healthy box is byte-identical, colour band
included.

This session rules the convention (felhom.eu CONTEXT.md S-39): an explicit `...Known bool` companion
beside the figures, checked in the template — the shape Offbox.StatsKnown already uses, whose own
comment says "a 0%-wide bar over an unread store is a picture of emptiness, and a picture is a
claim". Pointers and separate error fields are both legitimate Go, but a codebase with three
dialects cannot be gated (ROADMAP G-3 was blocked on exactly this). Existing call sites NOT
converted.

R-258 — THE PER-APP BACKUP TICK WAS GREEN ON PRESENCE, AND RED ONLY ON A GLOBAL CONDITION.
buildAppBackupRows set Tier1LastStatus from status.LastDBDump.Success, which is the box's single
most recent dump RUN, whichever app it belonged to. An app whose own dump failed showed a tick as
long as some other app dumped successfully afterwards; an app with no database took the nil branch
and went green on the mere existence of a restore point.

appDumpVerdict now reads THIS app's own entries in DBDumpStatus.Results (matched on
DumpResult.DB.StackName, failure = non-nil Error). Three states: any failing database -> error; all
clean -> ok; no result recorded -> NO verdict and no icon, titled "Errol a mentesrol nincs
eredmenyunk." The recovery unit carries no per-run outcome of its own, so green cannot honestly be
derived from presence. The global tier1DBStatus label is untouched — it is correct as a global.

RECENCY IS DELIBERATELY NOT ADDED. A tick over a three-week-old restore point is a real weakness,
but an age threshold means inventing a number and the time is already printed beside the icon.
Recorded as an observation, not changed.

AN EXISTING TEST WAS ASSERTING THE DEFECT AND WAS CORRECTED, NOT DELETED:
TestBuildAppBackupRows_Tier1FromRestorePoints expected "ok" for a status with no LastDBDump at all —
green from nothing but a file's existence. It now expects no verdict; its real subject, the
Tier1LastRun time, is unchanged.

The dashboard test EXTRACTS the meter block from the shipped template rather than copying it: a
copied block drifts, and a drifted copy passes while the page it claims to cover has changed — the
fixture-is-not-the-wire mistake this project has now hit twice.

Six red-proofs across both parts, each with the mutation asserted applied.

No new tag on any declared wire — report/builder.go maps into its own types and is untouched;
wire_contract_gate.py confirmed green.

go build / go vet / go test ./... green (28 packages), controller_gates --fast all OK, both run
separately from this commit.
2026-08-08 16:29:52 +02:00
admin fcffaf573a v0.209.0 — R-247: the box stops saying a false thing about its own recovery package
gates / gates (push) Successful in 17s
The answer was on the wire and was discarded at the boundary, for the third time.

The hub has sent `escrow_stale` in the report ACK since v0.57.0 (json:"escrow_stale,omitempty").
report.EscrowStatus had no field for it, so encoding/json dropped it, and an empty restic_pw_sha256
had exactly one possible reading here: "hash-less supersession".

On demo-hp that reading was false in EVERY clause for four days, and the box told the customer so in
its own words. The hub HAD the hash and was withholding it because the escrow row carries a stale
flag (R-246); there had been no supersession; and the bundle DID cover the password — the hashes
matched exactly.

Fixed by receiving the field. EscrowStatus.Stale decodes, and reconcileEscrowed tells the two
conditions apart: a withheld hash now reports that the hub has flagged the row and is withholding,
that this box therefore cannot verify its bundle either way, and that it is NOT established that the
bundle fails to cover the password. The genuinely hash-less case keeps its wording.

Deliberately NOT changed, and said rather than skipped: the stale verdict itself (the hub's flag is
still the hub's verdict; runs still continue), and the customer-facing Hungarian card copy. Clearing
the wrong flag is an operator act hub-side (R-246); re-wording the card is UI work with its own
review path. This change is the wire and the diagnosis.

Found by felhom.eu/scripts/wire_contract_gate.py (G-1), which was built first and seen failing on 40
fields before anything was fixed, and which now refuses any new field of this shape.

go build / go vet / go test ./... green, run separately from this commit.
2026-08-08 08:46:29 +02:00
admin 37b5ba08a7 REPORT: v0.208.0 — both R-254 sites, the guard's measured holes, and what the live read could not prove
gates / gates (push) Successful in 13s
Records the deliverables, and is explicit about the limit on the live half: the
curl of an app info page could not be done, and names exactly what was tried —
crafty-controller is the only app declaring initial_credentials and is deployed
nowhere, and demo-hp's dashboard password in ~/.config/credentials no longer
authenticates (200 with no session cookie). A probe of the new routes was
discarded because its control killed it: real and bogus paths both 302 behind the
auth middleware.

§7.2's answer including the part that contradicts the task's premise: no line in
the repo says 'no silent auto-fill'; the rule is CONTEXT.md:2070 about accidental
EMPTY-password deployments. The hidden input is deliberate and untouched.

§7.4's measurement: the gate covers all 36 templates and catches a launder through
a local variable, but is blind to a secret under a neutral page-data key — the
exact shape of site two. Runtime coverage is 4 of 27 pages. Filed as R-255 rather
than described as complete.

§7.3: no evidence of actual exposure on the fleet, with the limit stated — it is a
current-state measurement and nothing recorded reads, which was part of the fault.

Also corrects v0.207.0's report: html/template STRIPS HTML comments; they do not
ship in the response body. Measured.
2026-08-07 21:29:13 +02:00
admin 27d1165962 v0.208.0 — R-254: the last two secrets leave the page source, plus a gate against a fourth
gates / gates (push) Successful in 17s
Site one. app_info.html rendered {{.InitialCreds.Password}} into a hidden span —
a REAL per-install credential, read live out of the running container, in the
response body of every render. The page now carries the non-secret half plus a
boolean; the value comes from POST /apps/<slug>/initial-credentials/reveal, which
RE-READS the container rather than serving a cached copy (caching it in the
handler would put it back in the body one layer in). no-store, CSRF-covered,
logged as an act. Both buttons go through it. A reveal that cannot read the value
SAYS SO rather than returning an empty string that renders as a blank password.

Site two, established before changing. The hidden input is NOT the defect and was
left alone: it fires only pre-deploy, and README §318 documents why the value must
round-trip — the customer notes the generated secrets down and submitting them
back is what makes the saved value the same one they saw. The defect was the
neighbouring READONLY input, which on an ALREADY-DEPLOYED app rendered the secret
into a page with nothing to submit. Fixed by POST /stacks/<name>/auto-field/reveal,
authorised by requiring a type:secret auto-field of that stack. Both directions
pinned.

The premise that this contradicted a repo rule does not hold: the rule is
CONTEXT.md:2070 'Password fields require explicit input — prevents accidental
empty-password deployments', about EMPTINESS. No line in the repo says 'no silent
auto-fill'.

The gate. scripts/secret_in_markup_gate.py, registered in controller_gates.py,
convicts any template expression that names a secret unless allowlisted with a
reason. Its limits are MEASURED and in its docstring: it catches a launder through
a local variable (the assignment names the secret) but is blind to a secret
arriving under a neutral page-data key — verified both ways. That is the shape of
site two, which this gate would NOT have caught. The runtime body assertion covers
all shapes but only 4 of 27 page templates; the other 23 are R-255, filed rather
than glossed. Two nets, different holes, both named.

Correction to v0.207.0's report: HTML comments do NOT ship in the response body
here — html/template strips them, text/template does not. Measured. A red-proof
planting a secret in a comment therefore correctly does not fail.
2026-08-07 21:20:26 +02:00
admin 62998aab4f REPORT: v0.207.0 — R-249 before/after on a live box, the census, and what was NOT proven live
gates / gates (push) Successful in 18s
Records the deliverables: the raw response body before (1 occurrence, v0.206.0)
and after (0, v0.207.0) with a positive control in both directions; the §7.1
census finding two more instances of the render-then-hide pattern (R-254, one a
real per-install secret); §7.2's decision and why the promise was the wrong half;
every changed Hungarian string; all eight tests with their red-proof outcomes.

States plainly what was NOT proven live: R-252/R-253's notices could not be
rendered on VM 325 because both states are rebuild-only and the box re-registers
a drive on restart — the live run therefore exercised Scenario E instead, and the
notices are pinned at the template + predicate level with red-proofs.

Also records that red-proof D caught a fault in my own work: the explanatory HTML
comment quoted the old sentence, and HTML comments ship in the response body, so
the contradiction was still on the page and the assertion forbidding it could
never fail.
2026-08-07 18:33:37 +02:00
admin 8dbbc98ff2 v0.207.0 — R-249: the retrieval passphrase leaves the page body; R-252/R-253: two refusals learn to say what to do
gates / gates (push) Successful in 18s
R-249. settings_security.html rendered the passphrase into a display:none
span behind a Megjelenit button. That toggle stops a browser DRAWING the value
and nothing else — the plaintext was in the response body of every render, so a
curl of the page returned it. Found by exactly that: it landed in a session
transcript while driving the documented rebuild path.

The codebase already stated this rule for the recovery code and this page did not
follow it (escrow_handlers.go: 'reveal (claim XHR only — R is NEVER templated
server-side into HTML)'). The page now carries only HasRetrievalPassword; the
value comes from POST /settings/retrieval-password/reveal — CSRF-covered because
POST, no-store, and LOGGED as an act, which reading it off the markup never was.

The tests assert the RAW RESPONSE BODY. Every test that asked what the customer
sees passed while the bytes carried the secret; that is why this survived.

Census: the render-then-hide pattern appears twice more — app_info.html (a real
per-install app password in a hidden span) and deploy.html. Filed as R-254, NOT
fixed here.

R-252. A rebuilt box keeps its drives but loses their REGISTRATION. The restore
page now states that before the customer presses anything, says the backups and
drives are both still there, and links to Tarhely > Meghajtok. Page and resolver
ask ONE question — HasRestoreDestination() reads the same
GetSchedulableStoragePaths() the scratch resolver reads.

R-253. The list promised 'a visszaallitas elobb ujratelepiti' three lines above a
refusal that fired BECAUSE the app was not installed. The promise was the wrong
half: reconstitution writes to the app's own GetStackHDDPath, which exists only
once the CUSTOMER has chosen a drive at deploy time. Auto-reinstalling would mean
the product making that choice for them. Copy now says to install first and routes
to /stacks/<app>/deploy.

Both notices are conditional — a healthy box renders as before, pinned by a test
that fails if either becomes unconditional.
2026-08-07 18:04:26 +02:00
admin 3d3b4496f3 REPORT.md — R-241 fixed, deployed, live-validated on both demo boxes
gates / gates (push) Successful in 22s
Scenario A's live result first: on demo-hp in the rebuilt shape, no key was
minted on the real start-up offsite-apply path, and the hub received the state
it reports instead - offsite.state=awaiting_recovery_key with enabled:false.
Key restored byte-identical afterwards.

Includes Q4's seven rows mapped to the three states, the SEC 7.2 choice and
why, SEC 7.3's answer on the new-code button, every changed Hungarian string
quoted, all nine red-proofs with what was mutated, the R-245 reasoning, and
three observations noticed but not acted on.
2026-08-07 12:23:55 +02:00
admin 0a9158d53e docs for v0.206.0: CHANGELOG, CONTEXT, REUSE, README
gates / gates (push) Successful in 19s
CHANGELOG v0.206.0 with the ruling that reversed the fix, the three changes,
the SEC 7.2 staleness decision, Q7's closed trap, and the two bugs the tests
caught rather than review.

CONTEXT carries the three rules this session established, in the form the next
session needs them:
  - a box does not create a repository key while the hub holds a sealed
    package for it;
  - the fact that answers a question must be kept where the question is asked;
  - fix the state, do not remember that it is wrong.

REUSE gains four rows, each carrying the trap rather than just the signature:
the mint guard is a CONJUNCTION and t.Enabled is load-bearing in the derived
predicate; the discriminator ships INERT unless wired in main.go's confirmer
literal; the countdown removes BOTH halves or neither and must be driven by an
injected clock; and the epoch must be synced FIRST and unconditionally or the
falling edge is lost.

README documents the three customer-visible changes and the operator levers.

No version literal was edited: the controller version is ldflags-only.
2026-08-07 12:12:35 +02:00
admin 72368654e4 R-241 part 5: escalating reminders, and operator levers for a running countdown
REMINDERS (SEC 2.3). The offer epoch now stamps when it began, and the
undecided reminder escalates in EMPHASIS at 1, 3, 7 and 14 days.

THE READING IS STATED BECAUSE THE SPEC IS AMBIGUOUS, and it is written into
the code where it can be corrected. For an ABANDONING box, 5/3/1 are
unambiguously days REMAINING before a deletion. An undecided box has no
deadline - nothing counts down to anything, because SEC 7.5 deliberately does
NOT auto-abandon - so 14/7/3/1 cannot be "remaining" and are taken as days
ELAPSED, with the wording firming up rather than the bar appearing and
disappearing. If the operator meant something else, one function changes.

The stamp is re-set on every entry into the offered state, so a box that
settles and is later rebuilt starts its ladder again instead of inheriting an
old one.

OPERATOR LEVERS (SEC 7.5). --abandon-status, --abandon-extend=N and
--abandon-stop on the controller CLI, beside the existing operator
subcommands. They exist because the path that ACTUALLY happens is the customer
telephoning, and support needs something to press.

They live on the CLI and not in the customer UI deliberately: extending a
deletion the customer asked for is an operator judgement, and a customer who
wants it stopped already has the self-service route - they recover with their
code, which cancels it.

BOTH REFUSE RATHER THAN NO-OP, in two situations: when no countdown is
running, and when the store has already been deleted. A silent success is the
thing an operator most easily mistakes for "handled" - they would tell the
customer their data was safe when it is gone. Pinned by two tests.

--abandon-extend counts from NOW, not from the old due date, and a test proves
the old date passes without deleting anything.

Green: go build, go vet, go test ./... all pass; controller gates OK.
2026-08-07 12:08:11 +02:00
admin de39e47f53 R-241 part 4: the three-state surface, and the copy tells the truth about the date
FULL PAGE ONCE PER ENTRY, NOT ONCE EVER. "Most nem" used to set a flag that
nothing ever cleared, so a box that abandoned its history and was rebuilt
months later - a genuinely NEW situation - would never see the page again. The
offer now carries an EPOCH, advanced on the edge into the offered state, and a
dismissal is recorded against the epoch it was made in. A fresh entry passes
the dismissal by arithmetic, with nothing to clear and nothing that can be
forgotten to clear.

That is NOT the flag the operator's ruling forbids. The forbidden thing
remembers that the customer decided so the screen can be suppressed while the
state stays wrong. This records WHICH SITUATION a dismissal was about.

A REAL BUG, caught by the test and not by review: the first draft returned
early from recoveryInterrupts when the offer was false, so the FALLING edge
was never recorded, RecoveryOfferActive stayed true through a settled period,
and the next entry counted as a continuation. The page never came back - the
exact defect the epoch exists to fix, reintroduced inside the fix. The sync is
now unconditional and the ordering is commented as load-bearing.

THREE LEVERS, THREE SCOPES, and none of them removes the route:
  - clicking the bar away  -> a browser SESSION cookie, cleared on login, so
    the reminder is genuinely back at the next login. Nothing persisted.
  - "ne emlekeztessen ujra" -> durable, epoch-scoped, silences the BANNER ONLY.
    It starts no countdown, abandons nothing, and a fresh entry reminds again.
  - "most nem" -> suppresses the full page only, as before.
The entry point on /backups/remote is bound to the OFFER and to nothing else,
pinned by a test that fires all three dismissals and asserts it survives.

SEC 7.3 / Q7 - THE TRAP DOES NOT SURVIVE THIS SESSION. While a recovery is
outstanding the "Helyrealitasi kod letrehozasa" button is UNAVAILABLE, not
merely captioned: creating a new code seals the current key, demotes the
package that opens the earlier history to retained custody that no shipped
path can read (R-199), and re-enables the recovery screen through the orphan
route while invalidating the code that screen accepts. A warning beside a
button is a warning people click past. The card now explains and points at
/recovery instead.

SEC 2.4 - the abandon confirmation changes with the behaviour. It used to
promise "felretesszuk - nem toroljuk". It now states the grace in days (from
the constant the countdown actually uses, never a literal in prose), that the
sealed package goes with it, that the customer can change their mind, where
the date is visible, and that the question does not come back afterwards.

The countdown is shown on /backups/remote for the WHOLE window - the bar
elsewhere is a nudge, this is the record, and a deletion date must be findable
on a quiet day too.

Tests: once-per-entry across a full settle-and-re-enter cycle; the banner
dismissal proven to be a session cookie (MaxAge 0, no Expires) and to persist
nothing; the opt-out proven to silence the banner while leaving the offer, the
route and the countdown untouched, and to remind again on a fresh entry; the
entry point surviving all three dismissals; a settled box showing nothing; and
the back-redirect refusing "//evil.example".

An existing test (TestRecovery_E) was updated: it asserted the legacy boolean,
which the epoch replaces. It now asserts the dismissal landed on the current
epoch, which is the stronger property.

Green: go build, go vet, go test ./... all pass; controller gates OK.
2026-08-07 12:01:30 +02:00
admin a5d90ff801 R-241 part 3: abandoning starts a 14-day countdown that ends the question
Until now "set aside" renamed the remote store and touched neither the escrow
nor the key, so the hub went on holding a sealed package for a key the box no
longer used. Shape (c) compares those two, finds them different, and offers
recovery - correctly, and for ever. A customer who had already said "I do not
want the old data" would be asked again at every login.

The operator's ruling is that the answer is NOT a "they decided" flag: fix the
state, do not remember that it is wrong. So the decision starts a countdown,
at the end of which the set-aside store and the sealed package that protects
it are removed TOGETHER. Afterwards shape (c) has nothing to compare and the
offer falls silent on its own - because the state is right, not because
something remembers it once was not.

THE GRACE IS REAL. The recovery offer stays reachable for the whole 14 days;
that is the change-of-mind path, and a grace in which recovery is impossible
would be decorative.

BOTH HALVES OR NEITHER. Removing only the store leaves a package that opens
nothing; removing only the package leaves ciphertext nobody can ever decrypt.
The two cannot be atomic across two machines, so it is a two-phase commit:
delete the store, record a durable marker, and keep DECLARING
offsite.abandon_purge_requested until the hub's ACK stops reporting a
superseded package. A crash between the halves re-declares on the next sweep;
it never leaves the pair half-removed and silent.

HUB HALF - SEC 8.2 ANSWERED: yes, the hub was needed, and only for this.
store.PurgeSupersededEscrowForCustomer is the one place R-198's retention is
ever undone, and it never touches host_escrow (the package covering the key
the box uses now). The handler acts on the DECLARATION, never an inference,
and is placed immediately BEFORE the ACK is built - so
GetEscrowStatusForCustomer reads the effect and the SAME response closes the
box's two-phase commit. No second round-trip and no window where the box
thinks it is still owed. felhom-agent was NOT touched.

The countdown starts in ResetOrphanedRepo, NOT in the shared helper: the
helper is also the unclaimed auto-reset path, where nobody decided anything,
and an as-delivered box tidying a stranger's leftover store must not get a
customer's deletion clock. Pinned by a test.

Cancellation is wired into the recovery unlock, BEFORE the tier-up and the
listing - those can fail, and a countdown surviving a successful unlock
because a later step errored would delete the history the customer just
proved they can open.

The sweep is a Daily job at 05:10, not on the backup leg: it must run on a box
whose tier is not configured for runs. Quiet by construction on every box with
no countdown, and that silence is asserted.

Tests (all clock-injected; SEC 7.4 forbids shortening a live timer):
Scenario E (aside + package kept + countdown + offer still reachable, and
NOTHING deleted), Scenario F (both halves, the declaration repeating, the
close-out), Scenario G (cancel, path still nameable, no later deletion),
plus: not closed out while the package remains, a transport failure leaves the
countdown due and retrying, the no-op sweep issues zero remote commands, and
the unclaimed auto-reset starts no countdown.

RED-PROOFS, each with the mutation confirmed present in the file first:
  F1) store deletion skipped -> Scenario F FAILS (no rm issued)
  F2) declaration dropped from the report -> Scenario F FAILS (the hub is
      never asked; the package would outlive the store for ever)
  G)  CancelAbandon made a no-op -> Scenario G FAILS (uncancellable countdown)

Green: controller and hub both build, vet and test clean; controller gates OK.
NOTHING WAS DELETED ANYWHERE - the terminal step has only ever run against
in-test fakes.
2026-08-07 11:47:42 +02:00
admin a491abef6c R-241 part 2: the comparison the box already makes becomes the thing that offers recovery
THE FACT WAS COMPUTED EVERY CYCLE AND KEPT NOWHERE. EscrowAutoConfirmer.Reconcile
has compared the hub's restic_pw_sha256 against the local key on every ACK since
SLICE 3. On the final-walk venue it logged, at 03:28:03Z and thirty-five minutes
before the customer looked, "the hub's escrow blob does not cover the CURRENT repo
password (hub hash 30ef574f != local 9b4a9a9d)" - and dropped it. The recovery
screen, evaluating in the same process, went on asking a question that could not
see it.

Now persisted: settings.HubEscrowKeySHA256 + HubEscrowKeyCheckedAt, recorded
UNCONDITIONALLY in Reconcile beside RecordPresence and RecordSuperseded - same
place, same reason: the box that needs it most is the rebuilt one with no target,
on which every gate below returns early.

OffsiteRecoveryOffer gains SHAPE (c): the hub holds a package for a key OTHER than
the one we are using. (a) and (b) are both proxies for that question and both have
now been wrong in opposite directions - (a) goes false the moment anything mints,
(b) is unreachable while the escrow is pending.

SEC 7.2, decided deliberately and stated in the code:
  - a KNOWN DIFFERENCE offers, however old the reading. Age is not gated on. Both
    sides are local; only the hub's half can be stale, and what the hub holds does
    not change without a ceremony THIS box runs, which refreshes the hash on the
    next ACK. Gating on age would make a box offline from the hub silently stop
    offering - the exact failure this session removes. CheckedAt is persisted for
    diagnosis, not as a gate.
  - an ABSENT hash falls back to (a)/(b) and does NOT offer. "" is the hub
    positively saying its package seals no repository password (legacy hash-less
    escrow). Nothing to compare, and offering would put a permanent screen in
    front of every legacy box.

The write damper: CheckedAt refreshes on every ack carrying a hash, but a save is
skipped when both the hash and the UTC day are unchanged, so an idle box does not
rewrite settings.json every fifteen minutes. It records WHEN WE LAST HEARD, not
when it last changed - the R-100 distinction.

Tests: Scenario C (a differing key offers, with both proxies asserted false first),
Scenario D (a matching key offers nothing), fact 1 still required, shape (a) still
works, and both SEC 7.2 halves.

RED-PROOFS, each with the mutation confirmed present in the file first:
  D) hubHash != localHash conjunct dropped -> Scenario D FAILS (a healthy box
     offered recovery forever); Scenario C still passes
  WIRING) RecordEscrowKeyHash removed from the EscrowAutoConfirmer literal in
     main.go -> TestMainWiresRecordEscrowKeyHash FAILS. This is the ships-inert
     shape: unwired, everything compiles, every test in the package passes, the
     auto-confirm still works, and shape (c) reads an empty hash forever.

Green: go build, go vet, go test ./... all pass.
2026-08-07 11:33:42 +02:00
admin 763de3a025 R-241 part 1: the box does not mint a repository key over a sealed package
THE DEFECT. WriteOffboxSecrets auto-generated on ONE input - does the file
exist. Its two neighbours in the same file, OffsiteRecoveryOffer and
needsOffsiteCredential, both consult GetHubEscrowIdentityPresent(). The same
fact was available on three paths and used on two.

Measured on the final walk: a rebuilt box's credential self-heal reached here
at 03:18:06Z and minted 9b4a9a9d over a hub package sealing 30ef574f. The
recovery screen then correctly reported nothing recoverable under the key the
box held. The screen was honest; the minting was not. And the flag was not
merely available at that moment - it was the PRECONDITION of the chain that
reached this function, logged at 02:48:03Z, six ticks earlier.

THE GUARD IS A CONJUNCTION, deliberately: a package held AND no key present.
A box the hub holds nothing for mints exactly as before.

The refusal is a HOLDING state, not a failure. ApplyOffsiteTarget catches the
sentinel and still writes the transport (ssh key, known_hosts, coordinates),
so the recovery screen can bring the tier up the instant the escrowed key is
placed (R-219). Returning the error instead would leave needsOffsiteCredential
true forever and the hub re-staging a consumed credential on every cycle.

New declared state offsite.state=awaiting_recovery_key, shown INERT to every
existing hub reader from their code rather than assumed: offsiteheal acts on
exactly one string; isStale needs Enabled && escrowed and this carries
Enabled=false; the delivery checker skips the applied shape; an unknown state
string is ignored by encoding/json. So NO hub change is needed for this part.

OffboxAwaitingRecoveryKey is DERIVED, not stored - the operator's ruling that
the state should be fixed rather than remembered, applied to this field too.

t.Enabled is load-bearing in that predicate and was MISSING in the first
draft. The existing TestOffsiteDeclare_DisabledTargetIsNotStranded caught it,
not review: a customer who switched off-site off is not awaiting anything.
Now pinned from the new predicate's own side as well.

Tests: Scenario A (no key written; transport still written; apply holds and
stages nothing), Scenario B (first-time box still mints), idempotency, the
nil-settings fail-safe, and the Scenario E carve-out.

RED-PROOFS, each with the mutation confirmed present in the file first:
  A) guard block deleted   -> both Scenario A tests FAIL with
     "R-241 REGRESSION: apply minted a repository password over the sealed
     package"; Scenario B still passes (the mutation is specific)
  B) guard over-widened (hub-package conjunct dropped) -> Scenario B FAILS
     with a first-time box unable to start; Scenario A still passes

Green: go build, go vet, go test ./... all pass; controller_gates all OK.
2026-08-07 11:25:58 +02:00
admin c6b69d888e v0.205.0 — a run that skipped an app the customer selected is not successful (R-234)
gates / gates (push) Successful in 21s
THE VERDICT. The R-203 block already said "a warning beside a success is read as a
success" and applied it to ONE of the two shapes it describes: an app missing a
declared mandatory FOLDER made the run incomplete, while an app skipped ENTIRELY
still reported ok. Both do now. Which skips count, decided by measurement:
selected+deployed with no recovery unit YES; selected but NOT deployed no (named,
with what to do — a box left amber by an app somebody removed is a status nobody
reads); disconnected/decommissioned drive no (own signal); nothing selected no.
LastSuccess and SnapshotCount still record what WAS captured.

THE FILED MECHANISM WAS NOT THE MEASURED CAUSE, and saying so is the point. §3
stated that toggling an app on leaves it without a bundle so the first run skips
it. Measured on demo-hp: the run's own pre-dump phase calls captureAllRecoveryUnits
for every DEPLOYED stack, through admitApp, before the push — a unit moved aside
was RECREATED and the run reported ok. That state does not survive a run.

What actually produced the 2026-08-06 sequence: the manual run was dropped by the
single-flight while an earlier run was still going. runOffboxBackup returned nil,
the handler had already answered "A tavoli mentes elindult", and the card then
showed the PREVIOUS run's green verdict — read as covering the app just selected.
The decision is now taken synchronously in the handler and a dropped request says
so. The nightly path still returns nil on purpose: nobody asked, and it retries.

§7.3 measured before deciding: CaptureRecoveryUnit writes a few KB of compose +
manifest, only ENUMERATES dumps rather than creating them, is idempotent and does
NOT stop the app — and already runs inside the off-site run. So there is no wait to
remove for a deployed app and NOTHING was built.

28 packages ok, 9/9 gates. Four red-proofs, each asserted to have applied. Fixture
note: the shared provider's ListDeployedStacks returned nil, so Scenario A first
passed for the wrong reason; fixed with an opt-in deployed set that defaults to nil.
2026-08-06 21:58:21 +02:00
admin 53e9bf0224 v0.204.0 — the restore list is keyed on the store (R-237); the size gate stops refusing in silence (R-238)
gates / gates (push) Successful in 26s
R-237: /backups/restore listed apps that are CURRENTLY DEPLOYED and CURRENTLY
TOGGLED ON for future off-site backups. A rebuilt box has neither, so a household
that had just lost everything was shown nothing to restore while the repository
held their snapshots — measured live on the R-201 re-walk. To restore an app you
had to select it, to select it you had to have installed it, and to know what to
install you had to see the backup you could not see.

The store is now the source of the list (offsite_restore_list.go), built on the
existing R-193 OffsiteInventoryList. Installed-ness became a property OF a row,
never a filter on it. Every case is answered rather than hidden: a snapshot for an
app that is not installed is offered and says it will reinstall first; an installed
app with no snapshot is shown as having nothing; an unreadable store renders as
UNKNOWN (R-225's rule, one screen over) AND keeps the action, because "we could
not look" is not "there is nothing"; no-target is its own state. The felhom-offbox
and _shares marker tags are excluded from the app list.

R-238 classified as a HARNESS ARTIFACT: mode=full without confirm=1 is step 1 of a
deliberate two-step — it starts no job by design and redirects carrying
&full_prep=<app>, which deriveWizardStep requires to reveal the commit. A driver
that did not carry it forward landed back on the intent step. The operator's
browser run completed the same restore. The wizard's precedence rules were NOT
re-keyed: a stale ?full_prep= must never resurrect a commit button mid-restore.

The residue WAS real and is fixed: neither branch of that step wrote anything to
the log, so a refusal — including by the headroom gate — left no trace on the box.
Both branches now log, and so does the concurrent-op refusal.

resolveWizardApp is removed: it was dead once the gate moved, and its test pinned
the defect's behaviour (an untoggled app refused), which would have read as policy.

28 packages ok, 9/9 gates OK. Three red-proofs, each asserted to have applied.
2026-08-06 16:44:05 +02:00
admin 4d349d1106 REPORT + CONTEXT for v0.203.0: the retry shape, the marker answer, R-220's shape
gates / gates (push) Successful in 10s
Records the decisions rather than only the code:
- POLL not ACK, decided on Scenario B against the ACTUAL promises — the
  no-target message gives no deadline and the card says 'within a day', so a
  5-minute tick is inside both and no text needed changing. If either promise
  tightens to minutes, go ACK-driven.
- The marker question: applied_marker lives in the guest's DataDir, which a
  rebuild destroys, so it cannot suppress a legitimate re-run. Left alone.
- R-220 candidate (b), corroborated rather than a wider prefix, reading
  /proc/mounts because the lsblk args are pinned in sudoers.

Live: Scenario C proven on demo-hp WITH a positive control — the job ran once
and logged nothing. A first reading counted 2 lines that turned out to be the
start-up reconcile, not the retry; the instrument was corrected before the
conclusion. Scenarios A and E are deliberately NOT live-proven here: both need a
rebuilt box, and that state arises naturally in Part 4.
2026-08-06 13:05:30 +02:00
admin 9dc26459ea v0.203.0: the box collects what the hub staged for it (R-218 consume half) + R-220's message
gates / gates (push) Successful in 10s
R-218's declaration half shipped in v0.201.0 and works. Its consume half never
existed. Reconcile ran exactly twice per process — at start-up and when the
recovery screen drives it — and BOTH fire before the hub has anything staged,
because the hub stages in RESPONSE to the declaration those runs precede.

Measured on the R-201 re-walk: unlock reconcile 11:43:07, hub staged 11:44:57
saying 'next cycle', a full report cycle ran 11:55:46, still unconsumed at
12:06. A guest command line applied it in 18 seconds — everything correct except
the trigger.

Bridge.RetryIfDeclared re-runs the SAME reconcile on a 5-minute tick, driven from
the box's own published declaration (OffboxReportStatus().State) — the very
statement the hub acts on, so the two cannot disagree.

Poll, not an ACK flag, decided on the promise: the no-target message says 'amint
megvannak' (no deadline) and the card says 'within a day'. Five minutes is inside
both by a wide margin and needs no hub change.

It stops by construction — a healthy box does no work and logs nothing — and the
settle gate is deliberately kept via ReconcileWhenSettled.

The marker was investigated and left alone: applied_marker lives in the guest's
DataDir, which a rebuild destroys, so it cannot suppress a legitimate re-run.

R-220's customer half: the refusal no longer tells the customer to choose from a
list that may be empty. It names the rebuild, points at the Meghajtók page, and
promises no outcome.

Red-proofs: remove the retry -> credential uncollected (the dead end reproduced);
drop the stop condition -> a healthy box hammers the hub; call Reconcile instead
of ReconcileWhenSettled -> settle gate bypassed; restore the old sentence -> the
impossible action returns.

28 packages ok, vet clean, all controller gates OK.
2026-08-06 12:56:12 +02:00
admin 66d80efb9f docs: R-168 is CLOSED — the "CI is still owed" sentence was stale (R-229 part 2)
gates / gates (push) Successful in 19s
Corrected in all four instruction files across all four repos. Found while confirming this
session own push by run ID, which is precisely the check that catches it.

In felhom-agent/CLAUDE.md the sentence contradicted the same file release section, which
already said R-168 mails the failure -- a contradiction inside one instruction file, the exact
class the R-229 work exists to find.

REPORT.md deliberately NOT overwritten in the sibling repos: a one-line docs correction must not
destroy the record of their last real implementation.
2026-08-06 11:02:55 +02:00
admin 7db42c5fec docs: CLAUDE.md becomes a core plus path-scoped rules (R-229)
gates / gates (push) Successful in 12s
215 lines -> 110 (92 effective; block-level HTML comments are stripped before
injection and never reach the model, verified empirically on Claude Code 2.1.222
with a control and a treatment run).

Four new .claude/rules/*.md, each with a paths: glob list so it loads only when a
matching file is read: gates, ui-hungarian, backup-paths, agent-coupling.

The ## Layout tree was deleted as derivable; REUSE.md already owns the per-package
seams its annotations stood in for. The host/access table was deleted in favour of
a pointer to documentation/operations/nodes.md -- it carried three defects at once:
demo-felhom given as the LAN fallback address as if it were the route, a pinned
"agent 0.93.0" against the project's own no-versions-in-docs rule, and the claim
that no drill VM was provisioned on demo-hp. Measured live: qm list shows VM 300
drill-r50. felhom-agent/CLAUDE.md was right; this file was wrong.

Kept verbatim: the seven session-critical invariants, the F9 live-validation fence,
the end-of-session checklist.

controller_gates.py registers the shared instructions gate (felhom.eu/scripts/,
never copied here; an absent sibling clone FAILS).

Docs only -- no Go, no version bump, no image, no deploy.
Ledger: felhom.eu/documentation/audits/LEDGER-instruction-trim-2026-08-06.md

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JJc8sAGRWmavP3rMtdpkr2
2026-08-06 09:38:27 +02:00
admin a62bb3874b REPORT + CONTEXT: v0.202.0's rule, its live proof, and what was NOT verified live
gates / gates (push) Successful in 9s
CONTEXT gains the rule so it outlives the bug: on the unlock path the customer
is blamed only after a real attempt REFUSED their code; every other outcome,
including an unclassifiable one, says something else. Plus the two things that
must not be 'fixed' into it — elapsed time is never a classifier, and the error
TEXT is never read (when the distinction was not a value, the agent was changed
to provide one).

REPORT states the split honestly: the AGENT half is proven live on the venue
(400 -> 502 -> 400, same wrong code, only the hub's reachability changed), while
the controller's message selection rests on handler tests and red-proofs,
because /recovery correctly redirects since F7 set the old data aside and
restoring that state is the reconfiguration §11 forbids. Also records that the
correct codes were shredded by the previous session, so the live re-run used a
WRONG code — which makes the test harder, not weaker.

Two venue changes stated because they were not asked for, both restorations: a
fresh dashboard password (the previous session shredded it, leaving the box
impossible to log into) set through the supported --print-reset-code escape
hatch, and one normal off-site run to populate stats_known.
2026-08-06 08:34:19 +02:00
admin 7534ea203d CHANGELOG: controller v0.202.0 (MinAgent 0.126.0)
gates / gates (push) Successful in 10s
Declares the coupling: an agent below 0.126.0 answers 400 for both a fetch
failure and a wrong code, so FeatureRecoveryFailureClass withholds the refusal
reading and the 400 degrades to the neutral message. The gate blocks nothing —
it only decides whether the customer may be told to check their typing.
2026-08-06 08:19:07 +02:00
admin c7446f2d6a R-225/R-227/R-228 Parts 2-4: unknown is not zero, the gateway speaks Hungarian, the set-aside is visible
R-225 — an unread store said '0 pillanatkép / 0 / 50 GB' above a card stating
it held backups under another key. An SFTP listing found snapshot f3d9cd67 and
12 535 KB really there; snapshot_count and repo_size_bytes were simply ABSENT
and the zero value spoke for them. StatsKnown is now NAMED, for the same reason
OffsiteInventory.Empty is: zero is what an unread store and an empty one both
look like, and on the wire 'absent' and '0' are the same bytes. The fill bar
renders only when the fill is known — a 0%-wide bar is a picture of emptiness,
and a picture is a claim. A measured zero still says zero.

R-227 — WHICH LAYER ANSWERS: traefik, and this repo generates its config. But
traefik v3 serves no static files, so a branded proxy page needs a new always-up
container for every 502 on the box — out of proportion, and scoped in the report
rather than built. Shipped instead: the unlock posts via fetch and answers a
gateway failure in Hungarian without leaving the page. Progressive enhancement —
with no JS the plain POST is unchanged and still shows the proxy's error, which
the report says plainly rather than implying otherwise.

R-228 — the set-aside history was recorded in orphaned_renamed_to and read by
nobody: a census found zero references in any template or handler, while 12 535
KB sat at that path. It is surfaced as two facts and stops. It does NOT promise
the history can be reopened, because it cannot be by anyone today (R-199's
inventory is unbuilt) — and the set-aside CONFIRMATION copy was corrected for
the same reason: 'a helyreállítási kód nélkül többé nem lesznek megnyithatók'
implied that WITH the code they could be. The field's own comment called it
'recovery-code-recoverable', which was the same over-promise in the code.

Tests: scenarios F, G, H as render tests per branch of each gate. Red-proofs,
each demonstrated failing then restored: remove the StatsKnown guards (F,
'R-225 RETURNED: an unread store reports a snapshot COUNT of zero'), delete the
set-aside block (H). The F assertion on the fill bar is scoped to the bar's own
container — a bare width:0% search matched unrelated elements and would have
passed for the wrong reason.

28 packages ok, vet clean, all controller gates OK (the emoji gate caught a
warning sign in a template comment).
2026-08-06 08:17:48 +02:00
admin 1e759a16ec R-224/R-226 Part 1: why the unlock failed decides what we say
The failure branch was a two-way choice — superseded? M4 : M1 — and BOTH are
statements about the customer's code. rerr was never inspected, so a hub that
refused, an agent that was stopped and a genuinely mistyped code all produced
the same accusation. Measured live 2026-08-05 with a CORRECT current code: hub
firewalled off 0.0556s, agent stopped 0.0299s, against ~1.0s for a real unseal.

Five classes, from the VALUE and never the text:
  hub-unreachable    502/503 from the agent — the code was NOT used
  agent-unreachable  no agent verdict at all (transport) — NOT used
  no-bundle          404
  bundle-too-old     409
  asked-and-refused  400 — the ONLY class that may mention typing
  unknown            everything else -> NEUTRAL, the safe default

agentapi.RecoveryRefusal carries the status as a value (refusalError flattened
it into a sentence, and a sentence is not something a caller can branch on).

THE OLD-AGENT CASE IS WHY THIS NEEDS A COUPLING. Agent < 0.126.0 answers 400
for both a fetch failure and a wrong code, so a 400 from one cannot be read as
a refusal. FeatureRecoveryFailureClass (MinAgent 0.126.0) withholds that
reading and the 400 degrades to neutral. The gate BLOCKS NOTHING — it only
decides whether the customer may be told to check their typing.

R-226: the superseded message now names BOTH possibilities and restores the
ten-words prompt. The two are indistinguishable at the engine; the honest
message says so. It still does not promise the earlier package can be opened.

Elapsed time is logged (it is what diagnosed this) and is NEVER a classifier.

Tests: scenarios A-E at the HANDLER + the classifier table asserting the same
sentence under two statuses classifies two ways. Red-proofs, each demonstrated
failing then restored: delete the 502 case (A), remove the mistype clause (C),
default to the accusation (D), route an instant transport failure to the typing
message (E).

Two existing tests encoded the defect and were corrected, not deleted: the web
fake returned a BARE error for 'wrong code' (which is the shape of a failure we
cannot classify), and R-222's test forbade any mention of typing on a
superseded box — half of which R-226 deliberately reverses.

28 packages ok, vet clean, all controller gates OK.
2026-08-06 08:06:57 +02:00
admin 05cf352a2f docs: CAMPAIGN-11 fix pass — REPORT + CONTEXT (controller v0.201.0)
gates / gates (push) Successful in 8s
2026-08-05 18:03:50 +02:00
admin a3499d1807 v0.201.0 — a correct recovery code is never called wrong again (CAMPAIGN-11) — MinAgent 0.125.0
gates / gates (push) Successful in 9s
R-216: the offsite key recovery is a coupled feature and now says so. featureProbes +
featureMinAgent 0.125.0 + a Supports gate at the unlock entry point, FAILING CLOSED — an
agent that cannot answer is named as such instead of the customer's code being blamed.
Measured live: a 404 from agent 0.120.0 came back as "we did not accept your recovery
code, check that all ten words", in 0.134 s, against a perfect code.

R-218: delete the repo-password short-circuit in needsOffsiteCredential. The declaration
stops when the TIER WORKS, not when a key exists — installing a key is the recovery
screen's whole job, so succeeding at recovery was switching off the mechanism that would
have delivered the coordinates to use it.

R-219: the unlock finishes the job — place the key, bring the tier up, then list. Without
it the promised listing could never render on the shape the screen exists for.

R-217: an unreadable store no longer claims to have opened with unattributable content
(the OffsiteInventory{} zero value). Opened / empty / unreadable are three states.

R-222: a code that is right about a RETAINED earlier package is named, not blamed. States
what the hub knows and promises nothing — no read path exists.

R-215: GET /recovery is gated on the same predicate as the interception.

Five red-proofs, each demonstrated failing and restored.
2026-08-05 17:48:08 +02:00
admin a315d623b8 docs: R-193 CLOSED — CONTEXT + REPORT (controller v0.200.0)
gates / gates (push) Successful in 10s
2026-08-05 12:56:43 +02:00
admin 62b85ecf13 CHANGELOG: controller v0.200.0 (R-193, the recovery screen)
gates / gates (push) Successful in 9s
2026-08-05 12:46:13 +02:00
admin 636c51e542 R-193: the recovery screen — unlocking, and only unlocking (v0.200.0)
A customer whose machine was rebuilt had everything needed to get their data
back and no way to find out: the only route was a command line. This is the
screen that closes that.

IT UNLOCKS, AND ONLY UNLOCKS (operator ruling). It explains, takes the recovery
code, opens the repository and shows what is in there — apps, dates, sizes. It
restores nothing: restore is already per-app and lives in the backups area, and
a screen that unlocks and then offers to overwrite is two decisions wearing one
button.

ONE CORE, TWO CALLERS. RecoverInstallCore is split out of RecoverAndInstall; the
CLI wrapper keeps its exit codes and printed lines byte-identical, and the
handler drives the same function. Two implementations of the one operation that
can permanently lose a customer's data would drift, and only one would be
tested. Asserted from source on both sides by AST.

THREE WAYS OUT, none a dismiss button: recover; 'most nem' (the full page stops
interrupting, the backups-area entry point stays PERMANENTLY, bound to the offer
and never to the postpone flag); and 'I do not want the old data' — confirmed
TWICE and reaching the SHIPPED move-aside, which sets aside and never deletes.

THE CODE IS HANDLED NO MORE LOOSELY THAN ON THE COMMAND LINE: POST body only,
never logged, never persisted, never echoed, cleared on every path, no-store,
autocomplete off. No lockout — the code is a ten-word phrase, and locking a
customer out of their own data for a typo is worse than anything it prevents.

TWO DEFECTS THE TESTS CAUGHT, both fixed: an UNCLAIMED (legacy-open) box would
have been shown the page, because RequireAuth passes such a box through; and the
inventory nil-dereferenced when no off-site target was configured, which is
exactly the pristine rebuilt shape.
2026-08-05 12:45:48 +02:00
admin be3c5fa7f6 docs: R-204 item 4 (box half) — CONTEXT + REPORT (controller v0.199.0)
gates / gates (push) Successful in 10s
2026-08-05 11:06:19 +02:00
admin 992803c10b CHANGELOG: controller v0.199.0 (R-204 item 4, box half)
gates / gates (push) Successful in 10s
2026-08-05 10:49:20 +02:00
admin a91f055960 pre-push: refuse a push from a clone outside the felhom workspace (R-204 rider)
Identical to the assertion added in felhom-agent 0404f60 and
app-catalog-felhom.eu ee2c810. See those commits for the reasoning.
2026-08-05 10:47:58 +02:00
admin 1214bae0a2 R-204 item 4 (box half): a rebuilt box DECLARES that it needs a credential (v0.199.0)
An absent off-site object has four meanings — never configured, mid-restart, a
transient config read failure, and rebuilt-and-stranded — and the hub cannot tell
them apart. The box can, from two local facts it holds with certainty, so it says
so instead of leaving the hub to deduce it from a silence (operator ruling).

The ACK's identity_blob_present is now recorded on EVERY ACK, before the gates
that used to discard it: on a box with no off-site target the auto-confirm returns
immediately, which is exactly a rebuilt box, so the one fact distinguishing it from
a box that never had off-site backups was thrown away every cycle.

The declaration needs BOTH halves — a fresh data area AND a hub-held recovery
package. Freshness alone is a box that never had off-site backups; dropping that
condition makes the whole fleet ask for credentials, which is what the Scenario B
test exists to catch.

The object carries enabled:false and zero sizes, which is what makes it inert to
the hub's existing fill and staleness checkers and to a pre-upgrade hub. A
configured box's JSON is byte-identical to v0.198.0's.
2026-08-05 10:47:51 +02:00
admin 68f195676b docs: R-204 items 1 & 3 — CONTEXT, REPORT, README (controller v0.198.0)
gates / gates (push) Successful in 9s
2026-08-05 07:37:25 +02:00
admin 33fcc502e4 CHANGELOG: controller v0.198.0 (R-204 items 1 and 3)
gates / gates (push) Successful in 10s
2026-08-05 07:17:32 +02:00
admin 2e936f43bf R-204 item 3: a restore says what it restored, and what it did not (v0.198.0)
mode=unit restores the recovery unit — the app's definition, configuration
and database dumps — and NOT the customer's own files: RestoreOffboxScratch
passes --include <unit path> and the userdata in the same snapshot is excluded
by it. The outcome was one sentence for both modes and named neither scope,
so on the last step of a disaster recovery the customer was told the app had
been restored after the thing they were looking for had not been.

restoreScratchOutcomeMsg states what came back, what did not, and the next
step that gets it. The wizard's intent card states its scope before the choice.
The full-restore size gate is untouched and pinned as unchanged; the default
stays unit, since all three wizard forms set mode explicitly.
2026-08-05 07:17:20 +02:00
admin 73b6dbc27d R-204 item 1: a freshly minted reset code works without a restart (v0.198.0)
--print-reset-code runs as a separate process and persists the new code;
the running server's cache was never told, so the code the customer was told
to type was refused until the controller restarted. Nothing said so — during
the 2026-08-04 drill that cost two attempts with an operator present.

effectiveClaimCode now reads through to the persisted state before applying
the settings-vs-config precedence, which is itself unchanged. Read-through,
not a TTL: a TTL would leave a window in which a superseded code still works,
which is worse than the bug. Fails closed on an unreadable state; an absent
file is not an error.
2026-08-05 07:17:13 +02:00
admin f4796e0d00 docs: R-203 contract + report (controller v0.197.0, proven live)
gates / gates (push) Successful in 10s
2026-08-04 18:53:47 +02:00
admin 58c703bd44 R-203 Part 2: a run that missed a MANDATORY directory is not a successful run (v0.197.0)
gates / gates (push) Successful in 8s
The gap was already detected and warned about, in Hungarian, naming the app and the folders --
that warning is what stopped the R-201 drill. The defect was that the run still reported `ok`
beside it, and a warning standing beside a success is read as a success.

last_status gains "incomplete": minted, because "ok" | "error" | "running" had nothing meaning
"it ran, and this app is not fully protected". NOT "error" -- the rest of the run worked and
what was captured is real, so SnapshotCount and the LastSuccess anchor still record it. Half a
backup is not no backup.

The gaps are now recorded STRUCTURALLY (offboxRunResult.mandatoryGaps), not only as prose, so
the verdict has something to act on. It reaches the operator through the EXISTING per-run digest
(backup_run_failures) rather than a new event type -- a new type is a two-repo change and the
hub drops anything outside allowedEventTypes.

The stat-filter gains the ClassMandatory check Tier 2 already had. It is a NO-OP today
(TierOffsite admits mandatory only), so no customer-visible warning disappears -- demonstrated
by widening the tier filter alone and watching the check hold the line.

ANTICIPATED: calibre-web on demo-hp has exactly this gap, so its off-site status becomes
incomplete the moment this ships. That is correct and is the point.

Red-proofs: my first Scenario-C proof PASSED because the test only reached offboxCaptureSet
while the mutation lives in runOffboxInternal -- a mutation the test cannot observe is not a
red-proof, and the fix was the test. The run-level test now fails under both mutations
(unreachable gap recording; unconditional ok).
2026-08-04 18:32:56 +02:00
admin a96c3d9473 R-203: the export-mount resolver takes the namespace root too (its own commit)
gates / gates (push) Successful in 9s
ExportDataMounts lives in delete.go, which reads as a destructive path. IT IS NOT: its single
production caller is the .fab export adapter, and nothing deletes based on its result. The
delete path's own guard, ProtectedHDDPaths, is layout-agnostic by construction -- it protects
BOTH <hdd>/... and <hdd>/felhom-data/... -- so deletion was never affected by the
namespace-root defect. That scope note is now in the function's doc comment, because the file
placement will mislead the next reader exactly as it misled the spec for this change.

Separated into its own commit anyway, so a change to a function whose filename says "delete"
is reviewable on its own.

An empty nsRoot falls back to hddPath -- the pre-R-203 shape -- so any caller not yet updated
keeps working on enrolled drives.

Tests cover both drive kinds and assert the NEGATIVE: no emitted path lies outside the app's
own data roots. Red-proof: leaving the site bare fails the system-drive row, emitting
/mnt/sys_drive/userdata where the canonical root is /mnt/sys_drive/felhom-data/userdata.
2026-08-04 18:21:17 +02:00
admin 73efb091d9 R-203: the app and its backup look in the same directory — one resolver, every caller
gates / gates (push) Successful in 9s
appbackup's path helpers take a NAMESPACE ROOT. Five call sites passed a bare DRIVE path.
On an enrolled drive the two coincide, so nothing showed; on the system-data fallback they
differ by exactly the felhom-data segment, and the app then bound a directory the off-site
capture set never looked at -- while the run reported ok. Measured live on demo-hp: the app
wrote to /mnt/sys_drive/userdata/media/books, the capture set looked for
/mnt/sys_drive/felhom-data/userdata/media/books.

THE RULE NOW HAS ONE EXPRESSION. appbackup.NamespaceRootFor / IsEnrolledDrive encode the
drive-kind comparison; backup.Manager.namespaceRoot and stacks.Manager.inGuest delegate to
it. There were already TWO copies and they differed -- the backup package's compared without
filepath.Clean, the stacks package's with it, so a trailing slash from config would have
flipped the mode in one and not the other.

Sites routed through it:
  - stacks/deploy.go withPathVars -> ${USERDATA_PATH}   (the live defect)
  - appexport/fabplan.go + export.go                     (via a new provider method)
  - web/handlers.go FileBrowser mounts                   (latent: the system drive is
    deliberately never a registered StoragePath, so this is the identity today)

ComputeFabBuckets now receives the namespace root, which is what ComputeCaptureSet has always
received -- so the export's classified paths and the backup's capture set describe the same
directories by construction instead of by coincidence.

Tests are table-driven over BOTH drive kinds, because this survived by being invisible on the
kind that already worked. Red-proofs observed: restoring the bare-path call fails the
system-drive row with the two paths differing by /felhom-data; inverting the drive-kind
comparison fails every enrolled row.
2026-08-04 18:17:05 +02:00
admin 532f5712a8 docs: R-200 Part 0 shipped; R-203 recorded (mandatory userdata dir missing from the offsite snapshot while the run says ok)
gates / gates (push) Successful in 9s
2026-08-04 15:00:40 +02:00
admin 1b1366bb6e controller v0.196.0: the recovered key installs itself (R-200 plumbing half) -- MinAgent 0.125.0
gates / gates (push) Successful in 8s
--recover-offsite-install is the sibling of --recover-offsite-check: same fetch/unseal path
through the agent, same STDIN discipline for R, but it PLACES the recovered repository
password via InjectOffboxPassword so a rebuilt box reopens the history it inherited.

Doing this by hand would put the offsite DATA key through a terminal, a clipboard and shell
history. In-process the value goes agent -> this process -> the 0600 file and is rendered
nowhere.

The confirmation is a SECOND invocation: without --confirm-install it prints both hashes and
writes nothing, so the operator sees the comparison before any write is possible.

Three outcomes, named distinctly: installed (no local password -- the rebuilt-box shape),
unchanged (identical key already present, nothing written), refused (a DIFFERENT key present;
installing would clobber the key the current repository is encrypted under, and no force
option is offered). Exit 2 for the refusal, distinct from 1 for a failed step.

Red-proof: removing the confirmation gate makes the dry run write, failing the test. The
R-persistence test carries a positive control -- a planted copy is found, then removed and not
found -- because an absence check is worth only what its sensitivity is.
2026-08-04 14:27:38 +02:00
admin bdab80c933 docs: R-200 diagnostic — CONTEXT + REPORT (proven live on demo-felhom)
gates / gates (push) Successful in 10s
2026-08-04 13:56:01 +02:00
admin 9640e51321 controller v0.195.0: prove the offsite key comes back (R-200 plumbing half) -- MinAgent 0.125.0
gates / gates (push) Successful in 10s
--recover-offsite-check is a docker exec diagnostic in the shape of --print-reset-code: it
reads the customer's recovery code from STDIN, asks the agent to fetch this host's sealed
bundle and open it, and reports whether the recovered key matches the one on disk BY SHA256.
Two hashes and a verdict; never a password, never R, never a blob.

R comes from stdin and not a flag because a flag value is visible in ps, in shell history, in
a container's command line and in any transcript of the session that ran it.

IT COMPARES; IT DOES NOT INSTALL. The recovered password is never written to
offbox/repo_password -- installing changes a live box on a path nobody has walked, and that
link is next session's, with the drill around it. A test asserts the data dir is byte-unchanged
after a check; its red-proof (adding the install call) fails it.

Exit codes: 0 match, 2 clean MISMATCH, 1 a step failed -- "it failed" and "it worked and
disagreed" must never share a status. A box with no local password reports distinctly: that is
the rebuilt-box shape, where the next step is to install rather than compare.

Nothing customer-reachable ships here: no card, no form, no preview.
2026-08-04 13:42:50 +02:00
admin 0887fd676d REPORT: R-182 — the run digest, the live proof, and the red-proof that did not fail first time
gates / gates (push) Successful in 9s
2026-08-03 13:59:42 +02:00
admin 88897a224e v0.194.0 — one operator email per backup run, and nothing dropped without a trace (R-182)
gates / gates (push) Successful in 8s
MEASURED, not supposed. On 2026-08-03 nine per-app recovery_unit_capture_failed
events reached the hub and TWO operator emails went out. The hub's operator
cooldown key is customerID:eventType(+tier) and that event carries `app` but no
`tier`, so the key held no app identifier: the first refused app took the hour's
slot and every other app's failure was discarded BEFORE anything was written
down, leaving no row on any channel.

The obvious fix — put `app` in the key — was ruled against: on a full disk it
produces one email per app, the volume problem wearing the correctness problem's
clothes.

internal/backup/runsummary.go: a per-run collector with exactly admissionSet's
lifetime, fed by all three write legs, emitting backup_run_failures ONCE at the
end and only when something failed. A clean run emits nothing.

The per-app event stays and becomes the RECORD — the hub routes it record-only,
stored and logged every time, never competing for an email slot. The record and
the notification are now different things.

Deliberate skips (disconnected, decommissioned) are excluded: they have their
own alert, and a nightly email about an unplugged drive is one the operator
learns to ignore.

A manual run always reports: the digest carries a unique run_id the cooldown
cannot collapse. Someone pressing the button is actively trying to get a backup.

THE PERIODIC SWEEP GETS A DIGEST TOO. With the per-app event now record-only, a
capture failure found between runs would be recorded and never notified — a new
silence introduced while closing one. That path emits a digest with NO run_id,
so the ordinary 1-hour cooldown caps it exactly as before while the mail now
lists every failing app instead of whichever was first.

A refusal is recorded ONCE, where the verdict is taken, not at the three legs
that consult it — R-181's contract is one verdict per app per run. Noting it per
leg listed one refused app three times and produced "2 of 1 apps failed". Found
by the digest's own test, not in review.

Silence is safe because the hub's deadline check raises expected_backup_missed
from report freshness, independently of any mail this box sends
(monitor/deadline.go:396,417). Confirmed, not assumed.

7 new tests, 4 red-proofs. The main.go seam walk did NOT fail on its first
attempt — the AST test walked the backup package and not main.go; the test was
fixed and the mutation re-run rather than the pass recorded.
2026-08-03 13:46:14 +02:00
admin db0d4b129d REPORT: R-181 — the reserve, the live proof, the du measurement and the teardown
gates / gates (push) Successful in 9s
2026-08-03 11:37:33 +02:00
admin 6c43bf6156 v0.193.1 — the refusal's size estimate is rendered in bytes, not "0.00 GiB" (R-181 follow-on)
gates / gates (push) Successful in 9s
Found by v0.193.0's own live proof run. The estimate was printed fixed to two
decimal GiB, so every app under ~10 MB rendered as "estimated 0.00 GiB write" —
which reads as "no estimate was available" and is the opposite of what happened.
Observed live on demo-hp 08:59:46: opengist's real 178 KB estimate printed as
0.00 GiB.

Shipped in the same session because it is the same defect class R-181 is about:
a message an operator cannot rely on is worse than no message.

The arithmetic is unchanged and still in GiB — the reserve's own unit, so the
comparison against FloorFreeGiB reads directly. Only the rendering moved to
humanizeBytes. estimatedWriteGiB -> estimatedWriteBytes, with the GiB conversion
done once at the point of comparison.
2026-08-03 11:05:02 +02:00
admin fef07c3923 v0.193.0 — the reserve guards the write that fills the disk, and its promise is true (R-181)
gates / gates (push) Successful in 9s
B2's capture floor (v0.192.0) was consulted in exactly ONE place —
captureAllRecoveryUnits, which writes a few KB. The two legs that write the BULK
into the same backups/primary/<app> tree, the DB dump and the volume dump, ran
FIRST and unguarded. Measured live on demo-hp 2026-08-03 06:40:03: opengist's
volume dump wrote 2.0 GB with no check, free fell to 1.0 GB, and the floor then
refused the cheap write it had already lost the argument to. Its refusal message
claimed "the previous unit is untouched" — measured false: that app's tar had
gone 182,272 B -> 2,147,666,432 B under a stale manifest. Sixth entry in
CLAUDE.md's table of shipped guarantees the code did not provide.

Fix: ONE admission verdict per app per run (internal/backup/admission.go), taken
before that app's FIRST write and covering all three legs — they write under one
per-app root, which is why one verdict can honestly cover them.

- Lazy, at the app's first write, NOT once at run start: app A's dump can put app
  B under the reserve, so a run-start verdict reads a disk that no longer exists.
- Remembered for the run, never re-decided between an app's own legs — that is
  the split this closes. Reset per run.
- Placed ahead of DumpAppVolumesSafe, which stops the stack as its first act, so
  a refused app is never bounced. After the volume-less check, which has no write.
- Exactly one operator alert per refused app per run.
- Leg order unchanged: volume dumps still precede the capture.

The floor is now SIZE-AWARE: it asks whether THIS app's write would cross the
reserve, not only whether the filesystem is already below it — which is how an
app was admitted at 96% and then allowed to write 2 GB. Estimate = the app's
previous .sql + .tar on disk. No history -> headroom-only, deliberately, and the
alert says so.

A container-based du per volume was MEASURED and rejected: 66 timed runs on
demo-hp guest 9201, median ~355 ms/volume (341-404) on volumes holding tens of
KB — container start-up, not the walk. Decisive on top: docker run needs the
writable layer, so it can fail under exactly the pressure the reserve handles.

The message was NOT weakened; the behaviour was moved so the wording became true.
It now also names which term bound. Every claim is checked against a sha256
fingerprint of the tree it describes, never against the log line.

Still refuses and never deletes: nothing here is generational.

11 new tests through the production functions. The DB leg cannot run without
Docker, so its gate is pinned by an AST walk of backup.go asserting admitApp
precedes DumpOne (strings.Contains is insufficient — a commented-out call still
contains the string). 4 red-proofs demonstrated failing then restored.
2026-08-03 10:53:48 +02:00
admin 4be6467b50 v0.192.0 — the capture floor replaces the bulkhead (R-165, decision B2)
gates / gates (push) Successful in 8s
Ships BEFORE the disk-layout merge it exists for, and is harmless on a box
that never gets it. The mp1 partition was a BULKHEAD as well as a ceiling:
it kept a runaway capture from filling the space the container runtime
needs, because /var/lib/docker was a different filesystem. After the merge
it is the same one, and a full Docker data-root is a stopped box.

The floor sits in captureAllRecoveryUnits, checked BEFORE anything is
written: below the reserve, that ONE app's capture is refused, its previous
unit is left byte-identical, the R-158 alert fires with the space figures,
and the loop continues.

Two terms whichever binds first (97% used / 1 GiB free) in fillwatch's
shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer
is always warned before a refusal can happen — a floor that fires before
its own warning is a silent failure wearing a threshold.

Headroom, never unit size: a per-unit cap would be R-163 rebuilt inside one
volume. Refuses, never deletes: nothing here is generational, so pruning
could only destroy a different app's only local copy; pruneStalePrimaryDirs
is an orphan sweep, not retention, and must not be repurposed.

Tests 1184 -> 1191. One fixture strengthened mid-red-proof: the "old 20 G
ceiling is gone" test sat at exactly 20 GB and survived a literal
UsedGB > 20 cap — hollow. Now 120 GB, and the mutation fails it.
2026-08-03 06:30:19 +02:00
admin d5be67b913 REPORT: CI run ids and conclusions (all three commits green)
gates / gates (push) Successful in 9s
2026-08-02 23:57:07 +02:00
admin 9a3c4855d7 v0.191.2 — a quiet fill check now says so (R-167)
gates / gates (push) Successful in 9s
Earned during v0.191.1's own live validation. After the customer had been
warned, a restart produced ZERO fillwatch lines — equally consistent with
'ran and chose silence' and 'never ran'. Proving the checker was alive
needed a deliberate crossing into the critical band.

For an edge-triggered check the quiet run IS the healthy steady state, so
that ambiguity is permanent rather than rare. Check now logs a per-RUN
summary on every run, counting unreadable separately from healthy so a
drive that has quietly gone unreadable cannot read as 'all fine'.
2026-08-02 23:37:04 +02:00
admin 5adae4dad9 v0.191.1 — the fill check also runs at startup (R-167)
gates / gates (push) Successful in 9s
Found live on 9201: neither sched.Daily nor sched.Every fires on
registration, so a box booting with a filesystem already over the line
would stay silent for up to 24h — the R-100 shape, and the same gap the
hub's own checkers avoid by leaving already-breached keys unseeded at init.

The watcher now runs once 90s after startup as well. Safe because the check
is edge-triggered against persisted state: an already-warned filesystem
stays silent. The delay lets mounts settle so a drive still returning reads
as unreadable and is skipped rather than warned about. Pinned by an AST
assertion — the schedule registration alone no longer satisfies the test.
2026-08-02 23:27:35 +02:00
admin cf48214f6c v0.191.0 — warn before the wall comes down (R-167, R-158, R-174)
gates / gates (push) Successful in 9s
R-167: new internal/fillwatch warns the CUSTOMER before a filesystem fills.
It emits the PRE-EXISTING disk_warning/disk_critical pair, which was
allowlisted, copy'd, default-enabled and checkbox'd with no producer in any
repo — the sixth "built but never wired" instance here. Two threshold terms
(85% or 5 GiB free; critical 95%/2 GiB) because a percentage alone lies at
both ends of this fleet's size range. Edge-triggered on escalation only,
state persisted, hysteresis dead zone at 75%/7 GiB pinned by a test. A nil
usage read is never a warning and never clears one. Per filesystem, never
per app. Daily 03:30, before the nightly app-data legs.

R-158: new unitNotify seam fires per app when a Tier-1 recovery-unit capture
fails, loop continuing, carrying the target filesystem's used/free bytes.
Operator-tier (recovery_unit_capture_failed) — deliberately NOT backup_failed,
which is customer-enabled and would email the customer about a failure they
cannot act on. D-c overrides R-158's own proposal here.

R-174: the app-stop guard no longer starts apps onto MISSING drives — a
regression in v0.189.0 code, found by review and closed the same session.
SetStarter got the raw stack manager, whose StartStack has no drive gate,
and Recover runs at startup. R-171 one path over. bootDriveGate could not be
reused whole (its holder #2 is the guard's own marker, and holders #1/#2 read
vars assigned after Recover runs), so holder #3 is extracted into a shared
driveStartGate with a test pinning the delegation. ErrStartRefused splits a
refusal from a failure: both keep the marker, only Failed alarms, because
routing a deliberate hold into NotifyBackupFailed is the same false alarm.

Tests 1157 -> 1184. All red-proofs demonstrated failing and restored.
2026-08-02 23:18:51 +02:00
admin 95eb5c2c1a REPORT: record every CI run id, run number and conclusion
gates / gates (push) Successful in 9s
2026-08-02 20:38:52 +02:00
admin e6311f9fbc docs: CONTEXT + REPORT for v0.190.0 (R-157 A / R-170 / R-171)
gates / gates (push) Successful in 8s
2026-08-02 20:34:48 +02:00
admin 4bad6e06c9 CHANGELOG: the boot window's sample refreshes first
gates / gates (push) Successful in 9s
2026-08-02 20:17:24 +02:00
admin dcc3363d2f boot window: sample REFRESHES first — a cached fleet made 'settled' meaningless
gates / gates (push) Successful in 9s
Found by live validation on 9201, not by review. GetStacks() is the Manager's
in-memory map refreshed by the scheduler every 10s; sampling it every 5s without
refreshing means two identical samples can mean the cache did not update rather
than that the fleet settled. A container removed ~5s before the window closed was
still in the sampled fleet and the sweep logged 'no boot-orphaned apps' for an app
that had none. sampleBootFleet now refreshes first; a refresh error degrades
rather than aborting the window.
2026-08-02 20:17:12 +02:00
admin 582135f861 v0.190.0 — the boot settle window, both gates on intent, and R-171
gates / gates (push) Successful in 8s
R-171 (a regression v0.189.0 introduced, CONFIRMED on hardware before any fix
was written). Replacing isBootOrphan's container-count term with recorded intent
made a drive-gate-stopped app read as a boot orphan: the gate stops apps with
`compose down` (zero containers) and never touches desired_state, because it is
not the customer. Observed on 9201 with the drive held unmounted — the sweep
found and started it, burned both attempts, and handed it to the dead-app alarm.
The write hazard did not materialise (the unbound mountpoint is host-root-owned
and the guest is unprivileged) but that protection is accidental and untested.
New consumer-side seam bootrecon.StartGate, fail-safe (cannot determine ⇒ do not
start), wired in main.go. The rule is not new: the API's startGatedByMissingDrive
already refuses this; the sweep bypassed it.

R-157 mechanism A. The sweep looked once at T+5s, deriving candidates from a
fleet docker was still restoring — three of six hard resets. Now a settle-then-
sweep window: sample every 5s, settled after 3 identical samples, sweep ONCE at
the end; ends on settled or a 50s budget, and the log says which. The budget is
50s because settle+budget+one retry must stay under the 90s dead-app grace — a
test rejected 60s at 95s. A window that overruns emits a LATE RECOVERY warn
rather than the grace being widened to hide it.

Widening the window made two more holders reachable, so the one gate covers all
three: an absent drive, a quiesce, and an in-flight app-data operation — reusing
quiesce.SuppressedStacks() and a new read-only AppStopGuard.HeldStacks().

R-170. shouldRecreateOnBoot now reads desired_state with the identical three-way
table; absent keeps the old hasContainers behaviour exactly. Its comment argued
for the container count and was rewritten. presentStable is untouched. The two
gates' agreement is pinned from both sides against one fixture table.

27/27 packages green; 6 red-proofs observed FAIL then restored.
2026-08-02 19:56:20 +02:00
admin 3446609420 REPORT: record all three CI run IDs and their conclusions
gates / gates (push) Successful in 9s
2026-08-02 18:58:56 +02:00
admin a8f7c61d41 docs: CONTEXT + REPORT for v0.189.0 (R-166)
gates / gates (push) Successful in 9s
2026-08-02 18:58:11 +02:00
admin dbcb306fcf v0.189.0 — desired state + the app-stop crash marker (R-166 / D-b)
gates / gates (push) Successful in 8s
The box stops inferring the customer's intent from a container count and reads
what they actually asked for.

Part 1 — desired state. AppConfig gains a tri-state `desired_state`
(""/running/stopped), written ONLY by the customer's own action: the API action
switch, DeployStack, UpdateOptionalConfig's redeploy branch, and the .fab
import. Intent is written BEFORE the act and a failed write REFUSES the act.
StartStack/StopStack are deliberately not writers — 14 callers, only 2 are the
customer. bootrecon.isBootOrphan now reads intent instead of len(Containers)>0,
which closes R-157 mechanism B (a power cut or interrupted deploy left an app
with zero containers, read as a deliberate stop, and stranded silently).

ABSENT MEANS UNKNOWN, NEVER "running": every pre-v0.189.0 app.yaml reads absent,
so the legacy fallback is byte-identical to the old rule. A running-only startup
backfill converges the unambiguous cases; `stopped` is never inferred.

Part 2 — backup.AppStopGuard, a persisted marker over every stop→work→start
window (volume dump, offbox reconstitute, .fab export). Its own file, never
quiesce's. Written before the stop, cleared only after a restart that succeeded,
kept when one fails. Recover() completes before the boot reconciler is launched
and returns its outcome, which main.go reports on the existing backup_failed
event once the notifier exists. A defer is not the mechanism — a SIGKILL runs
none (Campaign 8 fault 10).

Also: SaveAppConfig rebuilt AppConfig field-by-field (the R-100 shape) and would
have dropped desired_state on every save across nine call sites. Replaced with
copy-and-overlay. Measured: app.yaml does not round-trip unknown YAML keys.

No hub change, no agent coupling, no user-visible string. 27/27 packages green;
7 red-proofs observed FAIL then restored.
2026-08-02 18:40:17 +02:00
admin e7c44c0e0f docs: CHANGELOG + REPORT for the CI workflow (no version bump)
gates / gates (push) Successful in 9s
2026-08-02 16:35:42 +02:00
admin dcc400e175 ci: run the gate entry point on every push (R-168)
gates / gates (push) Successful in 9s
Reports, does not refuse — pushes go straight to main with no pull request, so there is no
merge for a status check to stand at. The refusing half is .githooks/pre-push, which is
per-clone and --no-verify-able; this half notices when that was skipped.

CI reproduces the workspace's SIBLING LAYOUT on purpose: this repo's entry point invokes
the shared reuse_refs_check that lives in the felhom.eu clone next door and is never copied
here, and this repo's REUSE.md cites wgsync/reconciler.go, which lives in the hub. Without
the sibling the gate fails closed with 'gate is MISSING' — correctly, but for the wrong
reason. No uses: step anywhere; no version bump; nothing built or deployed.
2026-08-02 16:27:25 +02:00
admin eaded79b18 REPORT: gate enforcement session (no version bump) 2026-08-02 15:37:02 +02:00
admin 7c32c74140 docs: CHANGELOG + README for the gate entry point (no version bump)
README's Enforcement block now names controller_gates.py as THE entry point rather than
listing five of the seven gates individually. CHANGELOG records the allowlist entry, the
runner, the hook, the seam test and their red-proofs, and states the no-bump explicitly so
the omission reads as a decision.
2026-08-02 15:28:40 +02:00
admin 8cb3d7af91 gates: one entry point (controller/scripts/controller_gates.py) + pre-push hook
A census of all thirteen gate scripts across the four felhom repos on 2026-08-02 found that
every check a CLAUDE.md names was passing and two of the four nobody is told to run were
failing. This repo had seven gates and CLAUDE.md named two; four more were reachable only via
a line in REUSE.md, and docker_run_volume_path_gate.py — RED at census time — through one line
in REUSE.md and nothing else.

controller_gates.py runs all seven plus reuse_refs_check on the repo root, streams each gate's
own output, and exits worst-wins non-zero. The shared reuse checker stays in felhom.eu/scripts/
and is invoked across the workspace — never copied here, which would recreate the drift it
detects; an absent sibling clone FAILS the gate and prints the path tried.

.githooks/pre-push runs it with --fast and refuses the push. Per-clone and --no-verify-able,
both stated in the hook itself; a manual run WARNS when the clone is unarmed.

test_controller_gates.py is a SEAM test — it asserts each member gate's own distinctive stdout,
not the runner's summary line. Red-proofed: an inert run_gate still prints 'all controller
gates OK' and exits 0, and turns the seam test red.

Tooling only: no Go change, no image build, no deploy, no version bump.
2026-08-02 15:22:53 +02:00
admin c432f701dd gate: allowlist appexport/estimate.go named-volume -v mount (R-29 leg a)
realVolumeSize mounts the NAMED VOLUME read-only into a throwaway alpine to du it
from a container view. No host path is involved — docker resolves the volume name
daemon-side — so it is the same safe class as the internal/backup/backup.go entry.
The gate was right to demand review; this diff is that review, on its own, because
burying an allowlist widening inside a feature commit is how an allowlist stops
meaning anything.

Tooling only: no Go change, no build, no deploy, no version bump.
2026-08-02 15:11:08 +02:00
admin 4115e88f68 REPORT: D5 — restore from the drive alone (v0.188.0), proven live 2026-07-30 17:00:32 +02:00
admin 4ed938cce4 D5: an app restore works from the drive alone (v0.188.0)
The recovery unit on the customer's drive now carries the PORTABLE secret
class, so Tier-1/Tier-2 restore no longer depends on the whole-guest tier.
A customer needs the drive and nothing else.

Part 0's rulings overturned the brief's recommendation, on evidence:
- the data_key flag is untrustworthy (4+ encryption keys the catalog itself
  labels as such are unflagged) -> R-127
- a DB password is not resettable in practice: POSTGRES_PASSWORD is ignored
  once PGDATA is non-empty, so a regenerated value leaves the app unable to
  authenticate against its own restored rows while the dump replay still
  reports success (proven on a throwaway postgres:16-alpine)

Ruling (operator): type:secret travels, type:password never does, minus the
nonPortableSecrets code register. Plaintext -- withholding the internet-
reachable class is what licenses that, and the two are coupled.

Precedence: the UNIT WINS over the guest -- the unit's secrets were captured
in the same run as the dumps beside them, so they match the data being
restored. The fail-closed data-key gate is unchanged.

Secret values are never logged; the manifest records NAMES only.
2026-07-30 16:33:06 +02:00
admin 2f27a363d5 R-108: network storage may not host an app's data namespace (v0.187.0)
This is D5's precondition and it is now met.

An app's namespace root IS its backup root: namespaceRoot returns a non-system
drive path as-is, so the recovery unit lands at <HDD_PATH>/backups/primary/<stack>/.
On a NAS that sits inside the share, which FileBrowser binds WHOLE — share root,
:rslave, download:true.

The bind was NOT narrowed, and establishing why inverted the fix. The share-root
:rslave bind is load-bearing (a 2026-07-22 probe proved an in-container access
through it wakes the idle automount trigger), and scoping is undefinable anyway:
apps on a share store at <share>/<app>, there is no userdata/ layer, and creating
one would write Felhom convention onto a customer's own NAS, which R-67 forbids.
So the browsing surface cannot be narrowed and the backup tree must never be
placed under it. Operator ruling: refuse the placement, keep the browse bind.
Tier 2 already refuses network targets for this reason (F-6C-1).

Nothing stranded: zero apps on network storage across all six hub customers
including Peti. R-67's browse capability is byte-identical.

FIVE surfaces, not the four the register named — settings.RefuseAsAppNamespace is
the single predicate. The deploy POST is the real boundary (it accepts any
caller-supplied HDD_PATH; DeployStack validates only os.Stat). Surface 4,
handleStorageDecommission mode=migrate, guarded only its SOURCE, so a whole
namespace could be decommissioned ONTO a NAS — that one is not in the register.

Fails closed: /mnt/felhom-drives holds both kinds, Kind exists only on a
registered path, so an unregistered path under that root refuses.

Supersedes README's "NAS backup locality — decision A" (v0.118.0).

9 tests, all non-effect (nil stackMgr, so a guard that misses panics rather than
passing). 4 red-proofs, each mutation asserted to have landed.
Suite rc=0, 27 packages, 0 FAIL. vet rc=0. Template + emoji gates OK.
2026-07-30 14:10:20 +02:00
admin b331f18424 v0.186.0 — R-114 + R-112: tell the truth about the backup target, then show it
Two defects E-2d found on a real box, fixed in this order deliberately: the
message is corrected BEFORE it is put on screen, because switching on a banner
that lies is worse than a silent one.

R-114 — the third state. resolveBackupTargetState had two outcomes: a disk
claims the target (healthy), or nothing does (degraded, "the backup is on the
system disk"). The state "configured, and its drive is gone" had no branch, so
it fell into the second and inherited its message AND its offer. Observed live
with the target detached: degraded:true, target:"felhom-backup" plus the
system-disk copy (false -- the backup was on a drive that had vanished) plus
offer_path naming that same vanished drive as the remedy.

New BackupTargetState.TargetAbsent discriminates. Degraded keeps its meaning
("is there a problem") so the wire contract is unchanged for every consumer;
TargetAbsent answers "which problem", because the two have opposite remedies --
attach any second drive, versus reconnect THAT one. Copy routed through
degradedMessageFor so one place still decides what a customer reads. The offer
is suppressed on the branch itself, NOT left to firstOfferableDrive's
Disconnected skip: that flag is set by the agent-side gate in another repo
(R-113), and this state must be correct independently of it.

R-112 — the state finally has a consumer. The endpoint was byte-correct and
nothing in the product ever asked for it: templates fetch 18 distinct
/api/storage/* endpoints and backup-target[/assign] were the only two with zero
references. Server-rendered on /backups now, following the existing
SingleCopyWarning banner pattern -- not a 19th JS fetch, because a banner that
needs JavaScript to appear is one more thing that can silently not happen.
backupTargetView returns nil for healthy and unknown so those render nothing at
all. The offer control POSTs to the existing assign endpoint behind the standard
inline confirm, never auto-submits, and surfaces restart_required honestly
instead of adding a self-restart.

Scenario E (the seam test) drives backupsHandler over httptest and asserts the
RENDERED HTML -- handler -> view -> resolver -> template. It deliberately does
not call the resolver and assert a string, which would prove the resolver that
was never broken. Deleting the one line that sets data["BackupTarget"]
reproduces the R-112 state and fails every render assertion.

Tests 326 -> 338 (+12) in internal/web; suite green (27 packages); both template
gates pass. Three red-proofs run and reverted, files byte-identical after.

MinAgent unchanged at 0.113.0: R-114 reads BackupTarget/MountPath/GuestPath/Role,
none of which R-113 altered (it changed BoundUnderParent, which this code does
not read). demo-hp on agent 0.113.0 is not held.

The absent copy is verbatim the hub's customerMessages["backup_target_absent"]
so the banner and the email tell one story -- filed as a two-repo drift risk,
not solved.

NOT LIVE-VALIDATED. Scenario C cannot occur on a healthy box; Session C proves it.
2026-07-29 19:21:32 +02:00
admin cdaeb36972 v0.185.1 — E-2: the offer endpoints were mounted where nothing routed to them
Registered as /api/backup-target inside ServeStorageAPI, which main.go mounts ONLY
at /api/storage/. Live result: endpoint not found, while every unit test passed --
the tests called the handlers directly and never travelled the mount. Caught by
the first live call, which is why the live call is part of the procedure.

Moved to /api/storage/backup-target[/assign]. A new test asserts the dispatcher
source contains both paths, so a handler nothing routes to fails the suite --
the seam-wiring rule applied to a route rather than a button.
2026-07-29 09:13:13 +02:00
admin 3f7cf2a965 v0.185.0 — E-2 Parts 3+4: the offer, and the honest degraded state
The half that makes the rest work: a degraded backup target recorded only in
config is the silent-degradation pattern this arc has spent a week removing.

Part 3 -- POST /api/backup-target/assign moves the target via the agent's
POST /backup/target. It is the ONLY writer of the role: registration does not set
it, the drive-gate does not, no scheduler does. Declining is not calling it. The
agent returns restart_required rather than restarting itself, because restarting
with a backup in flight records a spurious tier failure for a backup that
actually succeeded (E-1 did exactly that).

Part 4 -- GET /api/backup-target returns the state and, when degraded, Hungarian
copy in FACT -> CONSEQUENCE -> REMEDY order, pinned by a test: a customer told
only the fact cannot act on it.

Healthy renders NOTHING -- no badge, no reassurance, no tonal change.
degradedMessageFor is the single decision point, so exactly one place could start
decorating a working box. Red-proofed: reassuring on the healthy branch fails
Scenario E.

UNKNOWN is not degraded: an unreachable or pre-R-82 agent means we could not ask,
which is not evidence of degradation (R-88 Part 2's class).

A HOLLOW TEST caught by its own red-proof: TestUnknownStateRendersNothing used
{Known:false} with Degraded left false, so it passed even with the !Known guard
deleted -- the second condition covered for it. Now {Known:false, Degraded:true},
which fails properly. Without the red-proof the test would have been decoration.

State is derived from the AGENT, never from our intent flag: on the two boxes
migrated by hand in E-1 the intent was never recorded while the drive really is
the target.

MinAgent: 0.113.0
Green gate: build + vet + test rc=0 (27 packages), run separately from this commit.
2026-07-29 09:10:20 +02:00
admin 4d6c8a6056 v0.184.1 — E-2b keying fix: the backup-target branch was unreachable
Caught before deploy by tracing a.Path back to its source, not by a failure. The
0.184.0 image is superseded and must not be shipped.

ReconcileDriveGates resolves the target as isTarget[a.Path], and a.Path is the
REGISTERED StoragePath -- for an external drive that is the GUEST path
/mnt/felhom-drives/<name>, not the agent's host MountPath (/mnt/<name>) that
/disks reports. driveTargetByPath keyed on MountPath alone, so the lookup never
matched: every absent drive, the target included, fell through to the generic
storage_disconnected.

The alarm would have looked wired, passed its own unit tests, shipped, and been
silently wrong on exactly the drive it exists for -- the same defect class E-2b
was opened to fix, one level down.

Now keyed under BOTH paths, mirroring planDriveGates which already registers
present[] under GuestPath and MountPath for the same reason.

Red-proof: MountPath-only keying fails with "the backup target is not resolvable
by its GUEST path -- the gate passes a.Path (the registered StoragePath), so the
backup-target branch would never fire".

Green gate: build + vet + test rc=0, run separately from this commit.
2026-07-29 08:30:36 +02:00
admin c1a63de1c7 v0.184.0 — E-2b + Part 5: wire the drive-absent alarm that was never called
NotifyStorageDisconnected and NotifyStorageReconnected were defined and called
from NOWHERE. Registered in allowedEventTypes, in DefaultEnabledEvents, and given
a Hungarian message on the hub -- and never invoked. A drive going absent produced
apps stopped, a WARN log and a UI badge, then silence on every channel. Verified
against the gitignored-cmd/ trap with a positive control. Fifth instance of this
class, found by E-2 Phase 0 rather than by a failure.

A drive that is ONLY a backup target has no apps to stop, so it was silent twice.

ReconcileDriveGates now calls both halves. When the absent drive is the whole-guest
backup target it raises the more specific backup_target_absent (error) instead --
never both, since two mails for one event trains people to ignore the channel --
and recovers as backup_target_restored (info, the existing pairing-gated pattern;
severityNotifies NOT widened). The recovery mirrors the alarm's choice or the
operator cannot match them.

Which drive is the target comes from the AGENT (/disks backup_target, >= 0.112.0),
not from our StoragePath.BackupTarget: that is customer INTENT, and on the two
boxes migrated by hand in E-1 the intent was never recorded while the drive really
is the target. An older agent omits the field -> false -> generic alarm, never a
wrong one.

Before this an absent backup target had NO prompt signal: the tier stays DUE
(targetStoragePresent checks name presence, never reachability), so the only
evidence was its own failure at the next due cycle, up to ~24h away. The R-100
shape.

Tests observe the WIRE, not a mock, because the failure class is "nothing
arrives": a real Notifier posts to an httptest hub and the test asserts the event
type and severity that actually went out.

MinAgent: 0.112.0
Green gate: build + vet + test rc=0 (27 packages), run separately from this commit.
2026-07-29 08:21:25 +02:00
admin ff058a4f10 E-2 Part 1: the backup-target role on StoragePath (foundation, not yet wired)
Foundation only, no version bump -- nothing customer-visible changes yet. The
offer UI (Part 3), the degraded banner (Part 4) and the controller half of the
absent-target signal (Part 5) are NOT here; they are tracked as E-2 in
OPEN-ITEMS.md so this cannot become a sixth "seam built but never wired". The
fifth was found by E-2's own Phase 0: NotifyStorageDisconnected is defined and
called nowhere, so a drive going absent emits no event at all today.

StoragePath gains BackupTarget bool -- the sibling role to Schedulable/
IsDefault/Kind. It is INTENT, not truth: the authority is the agent's
backup.local_backup_target, and truth is read from GET /backup/tiers. This
records what the customer ASSIGNED so the controller can render the state,
notice the drive going absent, and detect drift.

Invariants, each pinned by a test asserting the CONSEQUENCE not the mechanism:
  - a drive NEVER acquires the role by appearing (registration does not set it)
  - exactly one carrier; assigning moves rather than duplicates
  - sticky: a new bigger/faster drive does not steal an assigned target
  - an absent target STAYS assigned -- clearing on disconnect would be a silent
    retarget by omission ("no target configured" instead of "drive missing")
  - a network share is refused (R-108 risk model; the role is about LOCAL disk
    failure)

Red-proof C: adding auto-elevation to AddStoragePath fails
TestRegisteringDrivesNeverAssignsTheBackupTarget with
  registering drives assigned the backup target "/mnt/hdd_1"

Attributes may suggest and refuse the absurd, never select: demo-felhom's backup
drive is an external USB HDD and BOTH demo boxes report removable=0, so a
transport rule disqualifies the reference drive and a removable rule finds no
candidate at all.

Green gate: build + vet + test rc=0, run separately from this commit.
2026-07-29 08:00:47 +02:00
admin fd50a73e65 C9-F1 + C9-F2: a restore that restored nothing, and a crash loop nobody saw (v0.183.0)
Both are the system reporting healthy while the customer is not, and both live in the same
status-derivation code. Neither is fixed by making the system quieter.

C9-F1 (HIGH) — Tier-2 writes recovery-unit/ on EVERY run and RestoreTier2Files has never read
it (tier2_restore.go:101-104 reads hdd/ + userdata/ only). Phase 0 enumerated all 53 catalog
templates against both demo boxes: 43 apps have NO readable subtree, so the button stopped the
app, restored 0 files, restarted it and said "Nincs hiányzó fájl — minden fájl megvan a helyén."
— at the moment the customer pressed it because files were missing, with 156 MB of BookStack's
data unread in the same copy. 9 apps have file legs but never their DB or volumes, so the same
sentence was also a clean bill of health over data never opened (immich: 1.3 GB Postgres unit).

Honesty half shipped: a pre-flight coverage check refuses UP FRONT without stopping the app and
NAMES the action that works; a run that proceeds claims only what it EXAMINED and discloses that
the database and volumes are not covered. Completeness is filed as C9-F1b — routing to the
Tier-1 unit restore puts a destructive operation behind a non-destructive button, so its confirm
copy has to carry that difference. C9-F4 filed: nothing reads the Tier-2 recovery-unit/ mirror,
so the second local copy that exists for drive loss is unreachable by any customer action.

C9-F2 (HIGH) — a crash loop was counted as working. StateRestarting is deliberately NOT added to
IsDownState (that alarms on every deploy fleet-wide, the over-correction F-A1 nearly cost us); a
sustained run becomes down after crashLoopAfter = 5m, set above the 120s deploy timeout, Mealie's
60s start_period and R-97b's 180s grace. The dashboard counter uses the same predicate, so it no
longer contradicts the alarm on the same screen. README's claim that faults "still surface as
restarting" was a wish with no test — corrected in place; it is the seventh such instance.

Six red-proofs observed, including the one that matters most: adding StateRestarting to
IsDownState fails the brief-restart test with "every deploy and update would page the operator".
go test ./... rc=0, 27 packages, run and read separately from this commit.
2026-07-28 18:53:56 +02:00
admin d8b3279731 REPORT + CONTEXT: R-101 + F-DIAG (v0.182.0), rendered dialog proven live 2026-07-28 16:46:16 +02:00
admin 3f048e042b R-101 + F-DIAG: the restore dialog names the last SUCCESSFUL copy (v0.182.0)
Tier2LastRun is the attempt clock and was rendered as 'Legutóbbi másolat' in the restore
confirm dialog. New LastSuccess + SuccessTracked anchor; tier2Update makes the three
rebuild sites safe by construction. F-DIAG: six distinct causes, target-aware redaction.
2026-07-28 16:36:47 +02:00
admin 3db8bfb953 REPORT + CONTEXT: R-100 producer half (v0.181.0), proven live 2026-07-28 13:36:00 +02:00
admin e000e201af R-100: record the offsite last-SUCCESS anchor (v0.181.0)
LastRun records an attempt, not a result. New OffboxTarget.LastSuccess, set only on the
success branch via the pure offboxAnchorAfterRun rule, carried to the hub as last_success.
Closes two silent-wipe sites (settings save, hub re-apply).
2026-07-28 13:12:37 +02:00
admin 4056feccee REPORT + CONTEXT: F-OBS shipped (v0.180.0); same shape found and fixed in the agent watchdog 2026-07-28 11:22:04 +02:00
admin fb91c8d766 F-OBS: the dead-app check gets a positive observable (v0.180.0)
deadapp-check had no observable at default info level: its per-cycle line goes
through Scheduler.dbg(), gated on logging.level==debug, so on a default box it is
never PRODUCED (not merely filtered) and cannot reach the always-DEBUG ring. A
30s interval also puts it on the scheduler's quiet path. 'No alarms' was
therefore indistinguishable from 'the detector never ran' — which undermines
confidence in the F-CRIT-1 fix in the field.

A periodic summary, not a line per run: at 30s a per-run line is 2880 lines/day,
which is why the original author chose silence. Every 20th scan (~10 min) emits
one INFO with the scan count, apps evaluated and apps down. A test pins the
cadence so it cannot be widened into uselessness.

Also corrects the 'unquiesce guaranteed by defer' comment — fault 10 established
the guarantee is the crash marker plus Recover().
2026-07-28 10:27:28 +02:00
admin a63409c843 docs: F-CRIT-1 + F-A1 — REPORT, CONTEXT (v0.179.0)
Six red-proofs with observed failure text, the bounds chosen for contention
(15m retry / 3h alarm) justified against the agent's own 120m PBS restore-test
ceiling and the 12m01s longest observed restore-test, and the live replay of
both findings on demo-hp with the hub DB as arbiter:
  - failed restart alarmed 9s after grace expiry; banner shows (stopped)
  - a deliberate user stop stayed silent through 9 dead-app scans
  - 409 contention produced 0 operator emails (8 -> 8)
  - a real transport failure produced 1 (8 -> 9)
Same box, same event type; the only difference is 409 vs a genuine error.
2026-07-28 09:47:55 +02:00
admin 079265ad8e F-CRIT-1 + F-A1: one alarm that never fired, one that fired wrongly (v0.179.0)
F-CRIT-1 — an app that failed to restart after a quiesce never alarmed, for two
independent reasons, either of which alone kept it dead:
  1. restartAll returned nothing, so the failure was logged and dropped and no
     caller could learn a customer's app had not come back. It now returns the
     stacks that failed; both call sites record the outcome.
  2. classifyRunStates whitelists StateStopped on invariant I1 ('StateStopped
     means the user stopped it'). The quiesce loop stops stacks by the same
     compose-down path, so a failed restart is also StateStopped and was
     whitelisted into silence. Loop.FailedRestarts() is now the only thing that
     lifts the whitelist, so genuine user stops stay silent (v0.164.0 pinned).

F-A1 — HTTP 409 is the agent's single-flight gate refusing while a restore-test
holds it, not a failure. agentapi now returns a typed *StatusError on POST, the
adapter maps 409 -> quiesce.ErrTierBusy, and the loop defers: no breaker, no
event, no operator email, tier stays DUE.

Two traps avoided. Silence: contention outliving contentionAlarmAfter (3h, set
by the agent's own 120m PBS restore-test ceiling) raises its own BLOCKED signal.
App thrash: removing the failure treatment also removes the breaker's deferral,
so a contended tier is dropped BEFORE anything stops (contentionRetryAfter 15m,
against a 12m01s longest observed restore-test).

Three comments corrected; the invariant rule added to both CLAUDE.md copies.
Six red-proofs, all observed failing.
2026-07-28 08:50:11 +02:00
admin 8f46495426 seam sweep: move the TieredBackend witness into production code (no version bump)
A witness in a _test.go file fires on go test and go vet but NOT on go build
alone — and a build-only step is exactly how the R-88 Part 2 near-miss would have
shipped. Moved beside the type it pins, and added one for AgentVersionReporter.

No defect found: quiesceBackend and *Client both satisfy their interfaces today.
No version bump, no deploy — compile-time only.
2026-07-27 18:23:50 +02:00
admin ca013c8d27 docs: REPORT for controller v0.178.0 (R-88 Part 2 + R-97c)
Records the valve-licence table, why a missing field means legacy rather than
unknown, and the near-miss: TieredBackend is runtime-asserted, so the signature
change built and vetted clean while the adapter silently stopped satisfying it.
2026-07-27 18:12:35 +02:00
admin 86ea482fc1 controller v0.178.0 — R-88 Part 2: only a positive 'never' fires the valve
MinAgent: 0.105.0. scheduledRunAllowed fired on any nil age; it now requires a
licence from valveLicensed, which grants it for AgeStateAbsent and for a LEGACY
agent, and refuses it for AgeStateUnknown. An unreadable storage no longer
masquerades as a first-ever backup and no longer quiesces apps outside the window.

A missing wire field means legacy, not unknown — deliberately. Treating it as
unknown would stop the valve firing on un-upgraded boxes and starve genuinely new
ones. Degrade logged once; unrecognised future values also map to legacy.

Caught in passing: TieredBackend is satisfied by a RUNTIME assertion, so the
signature change compiled and vetted clean while quiesceBackend silently stopped
satisfying it — which would have degraded every box to the single-tier path with
no error. Added a compile-time witness.

Also corrects the notifier comment that claimed operator-only came from a missing
customerMessages entry; enforcement is hub-side operatorOnlyEvents (hub 0.79.0).
2026-07-27 18:08:56 +02:00
admin ba8bf9cd75 docs: REPORT for R-97 (controller v0.177.0 + hub v0.78.0)
Includes the cooldown keying finding (per-event-type only; fixed narrowly with a
tier suffix), the 180s grace window derivation, and the end-to-end event evidence
with its control: the new type 200s, a bogus type 400s, and notification_log shows
one operator row and zero customer rows.
2026-07-27 17:07:10 +02:00
admin e9c99566b0 R-97: a failing backup is heard, and stops blaming the apps (v0.177.0)
R-97a: internal/quiesce had no route to the hub at all — three failed whole-guest
backups on 2026-07-27 produced zero events. TierNotifier is a seam (not an import),
wired by an init-only setter because main.go builds the notifier after the loop.
Edge-triggered: the failure fires when the R-88 breaker ARMS, not per retry, and
recovery rides recordSuccess's existing bool. Uses NEW operator-only event types;
reusing backup_failed would have emailed the customer in Hungarian about a backup
they cannot act on, since it has a customerMessages entry and is in live
enabled_events. Requires hub >= v0.78.0.

R-97b: v0.164.0's state filter cannot see an app caught MID-RESTART, which is how
BookStack alarmed. The fix is a suppression window keyed to the quiesce CYCLE,
consumed at the same single derivation point. 180s grace, derived from the deploy
flow's 120s health timeout and Mealie's 60s start_period; it expires, so an app
that genuinely fails to come back still alarms.
2026-07-27 17:01:41 +02:00
admin ccefff4f39 docs: record POSITIVE post-deploy verification for v0.176.0
The quiesce loop is silent by design when nothing is due, so 'no error lines' is
not evidence. The observable that is: every /backup/due makes the agent read that
tier's storage content, which lands in pveproxy/access.log. Both tiers polled
every 5 min, both HTTP 200, tick phase consistent with the deploy restart.
2026-07-27 16:36:30 +02:00
admin b8598361b8 docs: REPORT for R-88 Part 1 (v0.176.0)
Includes the Phase 0 findings: the nil is agent-side (case a, a type boundary
that cannot represent 'unknown'), the four restarts were deliberate rather than
self-update, and the loop produced zero backup_failed events because quiesce
never imports notify (filed R-97).
2026-07-27 16:27:21 +02:00
admin 32200c7b5f R-88 Part 1: a failing backup stops re-quiescing (v0.176.0)
internal/quiesce had no failure counter, no backoff and no breaker, and the driver
is a plain 5-minute ticker — so a tier that was due and kept failing stopped and
restarted every customer app stack every 5 minutes indefinitely. Live on
demo-felhom 2026-07-27: three cycles in eleven minutes against an unreachable PBS
tier; it ended only because PBS recovered.

The breaker gates the QUIESCE, not the backup — the harm was the outage taken to
attempt it, so backed-off tiers are dropped before any stack is stopped. Per
target (a broken offsite tier must not suppress a healthy local one), 15m→30m→
1h→2h→4h capped, reset on success, never permanent, never applied to TriggerNow,
and stillRunning is not a failure. State is in-memory on purpose: forgetting a
backoff costs one attempt; persisting one could outlive the fix.

Part 3 invariant recorded on scheduledRunAllowed — a missing value means UNKNOWN,
and only a positive 'never' may fire the safety valve. Fourth instance of the
class (hub v0.12.0, v0.73.0, R-81, R-88).

Part 2 (unknown != never) is NOT in this commit: the agent returns byte-identical
responses for 'read errored' and 'never backed up', so the controller cannot tell
them apart. That needs an agent wire change and is tracked separately.
2026-07-27 16:21:58 +02:00
Claude Code 3f0420ff9c REPORT: correct a STALE deployment section — v0.175.0 IS deployed on both boxes
The section read 'NOT deployed'. True when written (17:56 CEST), stale 26 min
later when I deployed (18:22 CEST) and did not update it. REPORT.md is defined
as 'overwrite with the most recent state', so an artifact this project validates
against was left asserting the opposite of reality.

Precision matters on the apparent contradiction: a 17:01 quiesce log 'before' a
17:56 commit is a TIMEZONE artifact (controller logs UTC, git commits CEST), not
evidence of a false claim. In one clock: REPORT 17:56 -> deploy 18:22 -> quiesce
19:01. Consistent. That host=CEST/controller=UTC mismatch is a recorded project
trap and caught me once earlier the same day.

Also records the live multi-tier quiesce evidence and what remains unproven.
2026-07-26 20:43:02 +02:00
Claude Code f5e106440d REPORT: R-82 Slice B (controller v0.175.0) — unit-proven, deliberately NOT deployed
The multi-tier path only engages against an agent serving /backup/tiers, and the
agent rollout was still mid-flight, so both production controllers remain on
v0.173.0. Names the three things that are therefore unproven live.
2026-07-26 17:56:10 +02:00
Claude Code 9e5ea56853 v0.175.0 — R-82: a tier that overruns the quiesce bound defers the rest
Operator ruling 2026-07-26: let the first backup run as long as needed; other
backups shouldn't start until finished.

A first FULL offsite snapshot runs for hours, far past max_quiesce. When that
bound elapses the app resumes (unchanged), but the loop then started the NEXT
tier while the first was still uploading. Now it breaks and defers the rest to
a later poll — vzdump still holds the guest lock, so the second start would be
refused by the agent (409, v0.99.0) or fail on the lock, and a failed backup
never satisfies a cadence, so the tier would retry into the same wall forever.

pollTier returns (phase, stillRunning, err). The app still resumes exactly once.

Red-proof observed and restored; full suite green.
2026-07-26 15:06:10 +02:00
Claude Code de96efc0c5 v0.174.0 — R-82 Slice B: one quiesce window, two backup tiers
MinAgent UNCHANGED — degrades gracefully against ANY older agent.

The agent gained per-target tiers in v0.97.0. The controller owns quiescing,
so the multi-tier schedule is reconciled here: every due tier is collected up
front and run inside ONE quiesce window (one stop, N sequential backups, one
resume). Two cycles on the weekly night would mean two app outages for one
night's work.

Dedup rule: local-only -> one quiesce; PBS-only -> one quiesce; BOTH due ->
ONE window with both backups inside; neither -> no quiesce.

- quiesce.TieredBackend + BackupTier + ErrTiersUnsupported (optional extension)
- agentapi: BackupTiers/BackupDueFor/StartBackupFor/BackupStatusFor;
  targetQuery("") yields an EMPTY suffix so untargeted hits the pre-R-82 route
  byte-for-byte
- Loop.resolveDueTiers = the dedup rule in one place, agent order preserved
- quiesceAndPollTiers + pollTier: app stays quiesced until the LAST tier
  snapshots (resuming earlier loses app-consistency on the DR tier). Consequence
  stated in the docs: both-due-night downtime = first tier's full backup + last
  tier's snapshot, which is why tiers run fast-first.
- Manual 'Mentes most' covers EVERY tier, due-ness ignored.
- Window-gate safety valve now uses the OLDEST due tier, so a stale DR tier
  cannot be starved by a fresher local one.

Capability detection: /backup/tiers 404 = pre-R-82 agent (the documented
route-probe mechanism). Not a featureProbes row on purpose — the loop needs the
tier LIST, not a yes/no. Degrade logged exactly once per process.

Tests +11, full suite green. Red-proofs #2 and #3 observed and restored.
2026-07-26 14:40:44 +02:00
admin 47fda06ba1 REPORT: R-77 v0.173.0 — Part 0 repair, red-proofs, live legs 1-4 2026-07-26 09:25:15 +02:00
admin 9056f01fae v0.173.0 — R-77: endpoint-drift detection, samba protected-set gate, channel log honesty
Source: felhom.eu/documentation/audits/DIAG-agent-channel-2026-07-26.md

bootstrap.DetectEndpointDrift names a controller.yaml vs bootstrap.json
local_api.endpoint divergence -- one ERROR carrying BOTH values and BOTH paths,
its own event type local_api_endpoint_drift, and its own Hungarian banner shown
ABOVE the channel banner because drift is the cause and "agent unreachable" the
symptom. It writes NOTHING: reconciling from bootstrap.json would clobber a
correct controller.yaml on any half-provisioned or hand-repaired guest, so the
authority ruling is deferred to R-78. Fail-safe silent on absent/unparseable/
incomplete bootstrap and on an empty endpoint (ensureLocalAPI's fill-if-missing
path is untouched). Fingerprint compared as a BOOLEAN only; token never
compared, logged or exposed.

EffectiveProtected now gates samba on Enabled && UserSet, mirroring BOTH of
reconcileSambaAt's early returns, and the doc comment is corrected in the same
change -- it claimed "detection and deployment agree in both directions" while
citing only !smb.Enabled, an assertion that went false when !smb.UserSet was
added. Not over-suppressed: sharing on WITH a password and a dead container
still alarms.

Channel log: the debounce placeholder is stateUnconfirmed (rendered "unseeded")
instead of "up", so a born-down channel no longer logs "up->down" and orUnseeded
stops being dead code. Logging only -- the placeholder is still matched in the
re-arm condition, so F2 born-down alerting is byte-for-byte unchanged and all
nine pre-existing channelhealth tests pass.

Tests 951 -> 959, all green. Red-proofs A (both directions), E and F.
MinAgent unchanged; felhom-agent untouched.
2026-07-26 09:13:52 +02:00
admin c7a3a90782 REPORT: R-75 v0.172.0 — Part 1.0 probe, switch audit, red-proofs, live legs 2026-07-26 08:30:43 +02:00
admin 8fadbd9891 v0.172.0 fixup 2: drop import/* from the carry-list
Found on the demo-hp live leg: with import, import/paperless and import/calibre
in the carry-list, the derived skeleton RE-CREATES a per-drive drop-zone on every
drive forever — the dead lookalike the canonical root exists to remove, and one
that is never backed up (class: excluded).

Not a zero-removals violation: nothing deletes what an existing box has. Both
demo boxes' old drop-zones were verified to hold zero files before the change.
2026-07-26 08:22:13 +02:00
admin 4773809334 v0.172.0 fixup: EnsureImportRoot must apply the convention to the parent userdata dir
Found on the demo-felhom deploy leg: ensuring only <sysNS>/userdata/import left
its parent at 755 root:root, because EnsureUserdataDir MkdirAll's intermediates
at plain 0755 and chmods only the leaf. That made the system drive's userdata
root the one on the box outside the 2775/gid-1000 convention.
2026-07-26 08:15:47 +02:00
admin 2958946517 v0.172.0 — R-75: canonical import root, catalog-derived skeleton, import surfaces
${IMPORT_PATH} = <system namespace root>/userdata/import — ONE drop-zone per box,
on the system drive, injected at BOTH compose-env builders with NO per-drive
fallback (unresolvable leaves it unset so compose fails loudly rather than
quietly building a second, dead drop-zone).

Third BindRoot (RootImport) + Import list in BackupSpec, extended through
ValidateBackupSpec/ClassifyBinds. Load-bearing: a stale `userdata: import/<app>`
entry against the moved bind would be a WHOLE-BLOCK reject, taking the app's
mandatory hdd classification with it.

Exhaustive-root audit: resolveAbs/structuralGuard/ComputeCaptureSet/
ComputeFabBuckets now take importRoot explicitly (an import bind resolved
against hddPath would name a directory on the wrong drive); unresolvable is
refused loudly into Skipped. GetImportRoot added to both provider interfaces.

Catalog-derived skeleton: UserdataSkeleton() -> UserdataSkeletonCarry() +
BuildUserdataSkeleton(), SORTED. The carry-list makes zero-removals true by
construction (`documents` is in no catalog app but on both boxes) and is the
fresh-box floor. The sort is not tidiness: the naive map-order derivation
measured 20 distinct outputs from 20 identical runs, which with fbNeedsRecreate
is a fleet-wide FileBrowser restart loop.

One authoritative compose parser: ParseComposeUserdataMounts now delegates to
ParseComposeClassifiableBinds. Import root excluded from per-app migration.

Surfaces: FileBrowser /srv/beolvasas source; app-page "Hova tegyem a fajlokat?"
with PathEscape deep links (never QueryEscape) and class-driven copy;
data_paths: annotation with the Fork-3 asymmetry; system-owned beolvasas SMB
share refused server-side at handler AND store, button omitted in template.

Caught on the way: the sharing template's row struct was function-local, so
adding {{if .System}} would have 500'd every share row. ShareRow is now
package-level and the render test uses the handler's own type.

Tests 915 -> 949, all green. MinAgent unchanged.
2026-07-26 08:12:57 +02:00
admin 3b672ba74c docs(CLAUDE.md): add demo-hp (t740) host row — designated drill/build VM host (no drill VM yet) 2026-07-25 09:59:11 +02:00
admin 7d5b0163ef v0.171.0: REPORT (overwrite) — device-model card labels live on 9201 2026-07-25 08:32:21 +02:00
admin f6a8249593 v0.171.0: disk-health card device-model label (pairs with agent v0.95.0)
agentapi.SmartSummary.ModelName mirrors the agent's model_name; the card row label
prefers the device model over the raw name/UUID, falling back to Name(+hint) on an
old agent. Additive. Test + red-proof (drop fallback -> A4 fails).
2026-07-25 08:23:30 +02:00
admin de14eedb9f v0.170.0: REPORT (overwrite) — root→launcher, gofmt, stale-note, gate-check finding, SMART spike 2026-07-25 07:57:00 +02:00
admin 9cc8424954 v0.170.0: root → Indítópult (302); Vezérlőpult at /dashboard; CLAUDE.md stale-note fix
/ now 302s to /launcher (ONE canonical URL per page — launcher body never served at /);
the Vezérlőpult keeps /dashboard + its nav slot (operator ruling, reversing v0.163.0).
Nav: Indítópult active on /launcher, Vezérlőpult href=/dashboard active there. Post-login
default (/) and the topbar logo (/) flow through the redirect; login target unchanged.
Repointed 2 dashboard-card tests /→/dashboard. Corrected the stale vacation/agent-DOWN
note in CLAUDE.md (agent is up at 192.168.0.162:8443).

Tests: 302 target+status, /dashboard 200, nav hrefs/active; red-proof verified.
2026-07-25 07:44:08 +02:00
admin 2487681396 style: gofmt normalization — no logic changes
gofmt -w across the controller tree (46 files) so gofmt -l is empty — disarms the
formatting landmine where a targeted edit + accidental gofmt -w swept ~46 unrelated
files. Pure formatting: whitespace + gofmt's optional-semicolon removal in reflowed
inline closures. One doc comment reworded ('' -> 'the empty string') to avoid gofmt's
Go-1.19 doc-comment typographic substitition ('' -> curly quote) muddying its meaning.
No build/vet/test behavior change.
2026-07-25 07:37:02 +02:00
admin 0a582ea07b v0.169.x: REPORT — appended (disk-health card + degradation notification, live-validated on 9201) 2026-07-24 21:41:02 +02:00
admin dbf631312e v0.169.1: disk-health card excludes logical/network storage (pbs/lvm/nfs/cifs)
The agent defaults SMART to UNKNOWN on non-physical targets, so they showed as
spurious 'Nincs adat' rows. isPhysicalDisk now excludes those types (card + check).
Test strengthened: a PBS/LVM fixture with UNKNOWN SMART must still be excluded.
2026-07-24 21:30:54 +02:00
admin c97975c1df v0.169.0: disk-health card + degradation notification (Lemezek állapota)
Consumes the agent v0.94.0 smart payload (MinAgent floor unchanged; feature-detect
by presence). One pure verdict fn agentapi.DiskVerdictFor shared by the dashboard
card and the 6h check. Card via a 60s /disks TTL cache (anti-smartctl-storm);
unreachable agent -> Nincs adat, page never blocks. disk-health-check (6h) emits
disk_health_degraded on a degradation only vs an in-memory baseline (first run
silent, recovery/UNKNOWN never notify, multi-attr -> one event). No global banner
(deliberate). Pairs with the hub allowlist bump.

Tests: verdict table (>=90 red-proof), notifier emit, check first-run-silent
(red-proof), degradation-once, recovery-silent, UNKNOWN-excluded, FAILING-critical,
nil-smart card, TTL cache.
2026-07-24 21:27:16 +02:00
admin e164fef70c v0.168.0: REPORT — live-validated backup window on 9201 (save/reschedule/restore, box left at 02:30) 2026-07-24 20:59:15 +02:00
admin 82c67e32e1 v0.168.0: customer-configurable backup window (Mentési időablak)
ONE setting (window start W) drives every nightly leg at fixed, never-stored
offsets: DB dump at W, tier-2 at W+60m, off-box at W+105m (wrap-safe). Precedence
settings > controller.yaml db_dump_schedule > 02:30.

- scheduler.UpdateDaily: retime a daily job at runtime (no restart) via a per-job
  buffered resched chan + a select case in runDailyJob.
- new pure package internal/backupwindow (LegTimes/GateWindow/EffectiveWindow).
- quiesce disk-tier window gate: scheduled cycles run only inside [W+2h,W+6h) with a
  safety valve (age>cadence+24h runs regardless); manual TriggerNow never gated.
  Backend.Due now also returns the backup age (from the agent's own /backup/due).
- backup page: Mentési időablak card (time input + derived leg/gate rows); POST
  /backups/window validates -> saves -> UpdateDaily x3 -> flash.

Tests: 5 groups, all red-proofed. Agent/cadence//backup/due untouched.
2026-07-24 20:55:44 +02:00
admin e33c1aeabc v0.167.1: REPORT note — centered sidebar logo, live-verified on 9201 2026-07-24 13:59:29 +02:00
admin d37bb1eb6a v0.167.1: center the sidebar logo (margin: 0 auto)
.sidebar-logo was left-aligned; add auto side-margins so the 140px logo centers in
the header (desktop sidebar + mobile drawer share the element). Adds TestSidebarLogo_Centered.
2026-07-24 13:57:48 +02:00
admin 59d182a6b9 v0.167.0: REPORT — outlined logo/favicon live-verified on 9201 (0 text, viewBox unchanged) 2026-07-24 13:50:37 +02:00
admin e9a365e59c v0.167.0: outlined logo + favicon (Part 4 follow-up to v0.166.0)
Viktor pushed the text-outlined logo.svg to felhom.eu main (be9edb4) — wordmark is
now 17 real path glyphs. Swapped FelhomLogoSVG to it; stripped Inkscape's leftover
empty <text/> shells + font-* style leftovers on the paths via lxml DOM pass (glyphs
untouched, no text-to-path by CC), dropped editor namedview. Cleaned FelhomFaviconSVG
vestigial <text>. Both constants now free of <text>/font-family; viewBoxes unchanged;
palette + gradients preserved. TestLogoSVG_NoLiveText committed green.
2026-07-24 13:48:08 +02:00
admin dbd1ff0c17 v0.166.0: REPORT — live validation on 9201 (drawer markup, no customer-name, versioned URLs; Part 4 gated) 2026-07-24 13:31:23 +02:00
admin bf44216e79 v0.166.0: mobile nav off-canvas drawer + sidebar cleanup + versioned logo/favicon URLs
Mobile nav was broken — the <=768px block predated the v0.146.0 accordion and
flattened .nav-links into a horizontal overflow-x strip, clipping the accordion's
nested sub-lists. Replaced with a sticky top bar + off-canvas left drawer that
reuses the vertical sidebar (accordion untouched), plus a no-js static fallback.
Removed the sidebar customer-name span (kept on login). Added ?v={{.Version}}
cache-bust to logo/favicon URLs (Cloudflare 4h edge-cache; 0.126.1 failure mode).

Part 4 (outlined-logo constant swap) gated out per §3a: live felhom.eu main still
serves a logo.svg with live <text>/font-family; constants unchanged.

5 new tests via the real layout/CSS render; nav_accordion invariants unchanged.
2026-07-24 13:26:22 +02:00
admin 1fd070615d v0.165.1: REPORT — live rendered-page assertion on 9201 (all Group-A strings present, files absent) 2026-07-24 12:45:19 +02:00
admin a04afc367b v0.165.1: native Megosztás… button in the share modal (Web Share API)
Feature-detected navigator.share opens the OS share sheet with title+text+URL only
(no QR files: — narrow support drops the URL when given file+URL). Hidden unless
supported; Link másolása stays the universal fallback and catches the non-cancel
rejection; AbortError is silent. Template JS + tests only. 2 red-proofs verified red.
2026-07-24 12:43:14 +02:00
admin 570fb30147 v0.165.0: REPORT — live validation on 9201 (§13, all checks pass, token redacted) 2026-07-24 12:17:23 +02:00
admin 15206314ab v0.165.0: Indítópult megosztása — guest launcher via capability URL (+ optional password, QR)
Mint a 160-bit capability URL (/s/<token>) serving a standalone read-only guest
launcher: same tiles, opens apps in new tabs, no account, no admin session.
Information only, zero control — every privilege stays behind each app's own auth.

- /s/ pre-auth pass-through (after the claim gate) + session-CSRF exemption; guest
  password POST carries its own pre-auth HMAC CSRF.
- Constant-time token match; empty stored token = disabled = byte-identical mux 404.
- Optional per-share password: separate bcrypt hash + own attempt map; signed cookie
  = HMAC(token|passwordHash) keyed with web.session_secret, so rotate/change invalidates.
- Guest labels ride the v0.164.0 ruling; never expose internal state vocabulary.
- Token redacted in logs (/s/<redacted>); never in CHANGELOG/REPORT/CONTEXT.
- Admin modal: copy-link, QR (go-qrcode), set/clear password, rotate, disable.
- Tests: Groups A-G (14) + 3 red-proofs verified red.
2026-07-24 12:08:43 +02:00
admin 8e5edb2865 REPORT: v0.164.0 live-validated on 9201 (stop silent; fault alarms; stop->start clean) 2026-07-24 11:06:57 +02:00
admin c23a0f6d2d v0.164.0: deliberately stopped apps no longer alarm (banner + email)
A UI stop (Leallitas -> compose down -> StateStopped) is the user's own
action, not a fault, and must not raise the deadapp banner OR the
app_start_failed event. Filter at the single fix-3 derivation point:
extract scanDeployedAppRunStates's pure core to classifyRunStates and
change the down predicate to IsDownState(st.State) && st.State !=
StateStopped. Suppresses StateStopped from both the banner dead-list and
the notifier Down-set at once.

Rests on two invariants (recorded at the seam, README, CONTEXT):
 I1 StopStack = compose down => zero containers => StateStopped
 I2 P2 census: all catalog services unless-stopped => faults never rest
    at stopped (they surface as exited/degraded).
IsDownState unchanged; out-of-band 'compose stop' (containers remain ->
exited) still alerts. Tests +4 (notify 3->4, main 4->7), both red-proofs
verified. No template/funcmap/notifier/counter/copy change.
2026-07-24 10:50:14 +02:00
admin 77956d8df2 REPORT: v0.163.1 launcher polish — live-validated on 9201 (monogram hidden, placeholder on backups_apps/stacks) 2026-07-24 10:17:43 +02:00
admin 2c80868c63 Launcher polish: monogram reveal-on-failure + app placeholder on every icon surface (v0.163.1)
(A) The launcher monogram bled through every transparent white logo — it rendered
unconditionally under the img. Now hidden by default (.launch-mono display:none),
revealed only when the img chain fails (onerror adds .launch-tile--noimg).

(B) The /static/app-placeholder.svg default reached only app_list_row. The four
other sanctioned app-logo onerror chains now match the canonical SVG->PNG->placeholder
grammar: backups_apps (aligned row), stacks (infra->infra-logo else placeholder),
app_info hero (screenshots still vanish), deploy (keeps .LogoURL/.LogoPNGURL).

Template/CSS only; no handler/funcmap change. 5 tests + 2 red-proofs.
2026-07-24 10:14:25 +02:00
admin 7a53cb43ef REPORT: full authenticated live-validation of /launcher on 9201 (4 tiles, colors, filebrowser incl.) 2026-07-24 09:26:56 +02:00
admin ea432ca74a REPORT: v0.163.0 live-validation results (deploy healthy, placeholder + auth-gate verified live) 2026-07-24 09:22:48 +02:00
admin 4aa7d41cac REPORT: v0.163.0 Indítópult launcher + app placeholder 2026-07-24 09:17:39 +02:00
admin 987e915bf2 Indítópult launcher page + universal app placeholder icon (v0.163.0)
New /launcher page: a grid of large tappable tiles, one per openable deployed
app (subdomain presence is the single openability criterion, shared with the
Megnyitás button via the extracted Server.subdomainMap helper). Colored tiles
(deterministic slug color or .felhom.yml brand_color), white glyph/monogram
fallback, target=_blank links for operational apps, greyed unclickable tiles for
stopped ones. First sidebar item; / stays the Vezérlőpult.

Universal app placeholder: new AppPlaceholderSVG served at
/static/app-placeholder.svg, now the default FallbackIcon on app_list_row so a
logo-less app shows a placeholder instead of visibility:hidden. Brand mark is
never an app placeholder.

New Metadata.BrandColor; new funcmap tileColor/initial. 10 new test functions +
4 red-proofs. No agent coupling; MinAgent unchanged.
2026-07-24 09:16:28 +02:00
295 changed files with 40061 additions and 1665 deletions
+43
View File
@@ -0,0 +1,43 @@
---
paths: ["controller/internal/agentapi/**"]
---
# Coupling to the host agent — felhom-controller
`internal/agentapi` is **the disk seam**: the pinned-TLS client to the host agent's per-guest local
API. The controller holds no Proxmox credentials; everything disk/host/Proxmox goes through here.
## Declaring a coupled feature
Controller behaviour that depends on a specific agent version needs **all three**, or it ships broken
on an older box:
1. a `featureProbes` table row in `internal/agentapi/features.go`
2. a `Supports` gate call **at the feature's entry point** — not somewhere on the path to it
3. `MinAgent: X.Y.Z` in the CHANGELOG entry header
Rules: `felhom.eu/documentation/runbooks/publish-train-rules.md`.
## Never push a controller past the agent it depends on
The R-216 guard compared the box's agent against the **golden's** MinAgent while serving a **floor**
that could point elsewhere. Raise a floor above the vouched golden — which the day-0 runbook
recommends and a per-customer override makes trivial — and the guard checks a version it is not
serving. A box then landed on a controller needing a newer agent, and its customer was told a correct
recovery code was wrong.
**A floor above the vouched golden is HELD, with its own reason** (hub v0.97.0).
## Distinguish "could not reach" from "wrong answer"
A failed bundle FETCH must not be reported to a customer as a bad recovery code. Classify by **value**
(`ErrBundleFetch` → HTTP 502), never by error string — a string is not something a caller can branch
on. Unknown class → neutral message, never the typing message.
<!--
R-224, measured live 2026-08-05 (CAMPAIGN-11 F3/F4) with a correct current code: 0.0556 s with the
hub firewalled off and 0.0299 s with the agent stopped, against ~1.0 s for a genuine unseal — the
machine accused the customer of something it had not attempted. A green test named this exact
consequence since v0.125.0 and did not prevent it, because it asserted this package's error STRING
one layer below where the merge happened. Fixed agent v0.126.0 + controller v0.202.0.
-->
+41
View File
@@ -0,0 +1,41 @@
---
paths: ["controller/internal/backup/**", "controller/internal/appbackup/**", "controller/internal/recovery/**", "controller/internal/appexport/**", "controller/internal/quiesce/**"]
---
# Backup, recovery units and export — felhom-controller
## Assert the consequence across the whole run, not the mechanism inside one function
The R-181 recovery-unit refusal claimed *"the previous unit is untouched and NOTHING was deleted"*.
*Nothing deleted* held; **untouched was measured false** — the floor was checked ONLY in
`captureAllRecoveryUnits`, while the two dump legs wrote the bulk into the same tree first and
unguarded, so a 182,272 B tar became 2,147,666,432 B under a manifest that had not moved. A full
green suite plus three of its own red-proofs missed it, because every one asserted the mechanism
inside `captureAllRecoveryUnits`.
**The test that catches this class: fingerprint the tree before and after the whole backup run, and
compare.** Full doctrine and the other eight instances: the `felhom-testing` skill.
## Presence is not success
A timestamp recording an **attempt** must never be read as evidence of a **result**. Where a status
field travels alongside a timestamp, the verdict consults both — or the timestamp records only
successes. Ask of any timestamp: *what exactly must have happened for this to be set?* If the answer
is "we tried", it cannot answer "did it work".
**Corollary:** when a verdict changes which field it counts from, the alarm text has to change with
it. `last run 8h ago` while alarming on a six-day-old success turns a true alarm into one the
operator dismisses.
<!--
Two instances. F-CRIT-2: a phantom snapshot's ctime set tier freshness — an aborted 1-byte upload
made the tier look backed up. R-100: LastRun is written on failure, so a nightly-failing offsite
tier kept the staleness clock fresh forever.
-->
## Storage keys and paths
- Never guess a persisted key — it is `offbox`, not `offbox_target` (R-7b).
- `.fab` export/import uses strict segment validation; bundles from controller ≤0.124.0 are hollow.
- Recovery-unit restore and tier-2 copies share `appbackup`'s path primitives — change them there,
once, not per caller.
+54
View File
@@ -0,0 +1,54 @@
---
paths: ["controller/**/*.go", "controller/**/*.html", "controller/**/*.css", "controller/scripts/**"]
---
# Gates and logging — felhom-controller
## The ONE entry point
**Run `python3 controller/scripts/controller_gates.py` (from `controller/`) after ANY change in this
repo.** It runs all seven local gates — `template_id_gate`, `emoji_gate`, `native_confirm_gate`,
`offbox_rename_gate`, `app_row_dedup_gate`, `mojibake_gate`, `docker_run_volume_path_gate` — plus
`reuse_refs_check` and `instructions_gate` on the repo root, streaming each gate's own output and
exiting non-zero if any fails.
- `--fast` selects the gates that touch no network and no container runtime; today that is all of them.
- **A missing gate script is a FAILURE, never a skip.**
- **The shared `reuse_refs_check.py` and `instructions_gate.py` live in `felhom.eu/scripts/` and are
never copied here** — a copy would recreate the drift they detect; an absent sibling clone FAILS.
- **The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It is
per-clone — switch it on once with `git config core.hooksPath .githooks`, and a manual run WARNS
when this clone is unarmed. `git push --no-verify` bypasses it deliberately; **say so in the session
report when you use it** — CI re-runs the same entry point on every push and **emails the operator
on failure**, so a bypass is noticed even though it is not blocked (R-168, CLOSED 2026-08-02).
<!--
WHY A RUNNER AND NOT SEVEN INVOCATIONS (2026-08-02, R-29) — rationale, not a directive.
A census of all thirteen gates across the four repos found that every check a CLAUDE.md named was
passing, and two of the four nobody is told to run were failing. This repo's CLAUDE.md used to name
two of the seven; the other five were reachable only through a line in REUSE.md, and
docker_run_volume_path_gate.py was RED. The single-entry-point shape is the only one that
demonstrably gets run. app-catalog-felhom.eu/scripts/catalog_gates.py is the canonical version of
the runner (R-161); repo_gates.py copies it. site_gates.py is a *gate*, not a runner — do not model
new work on it.
-->
## Logging
New leveled lines use `internal/logx` — DEBUG always reaches the debug ring; stdout respects
`logging.level`. English, keys-never-values, durations on outcomes. Full rules:
`felhom.eu/documentation/runbooks/logging-conventions.md`.
## Health checks issue no block I/O
A probe that touches a wedged device enters uninterruptible sleep, survives `SIGKILL`, and cannot be
recovered until the device returns or the host reboots — so `systemctl restart` hangs too. A timeout
protects the caller's control flow and nothing else: the blocked thread remains. Liveness is decided
from `/proc` and kernel state, never by reading or writing the filesystem.
<!--
Measured, R-117 spike §6.3 (felhom.eu/documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md):
a probe stayed in D state 3m50s after kill -9; a buffered write with no fsync blocked too (O_CREAT
needs journal access); and statfs/getdents returned HEALTHY on a namespace that EIOs every byte —
fast, and wrong.
-->
+27
View File
@@ -0,0 +1,27 @@
---
paths: ["controller/internal/web/templates/**", "controller/internal/web/**/*.go", "**/*.css", "**/*.html"]
---
# UI and Hungarian copy — felhom-controller
- **All UI text is Hungarian**, Budapest timezone.
- Design tokens, badge/colour rules, the 2px/no-shadow/no-emoji/BOM hard rules and the mechanical
gate to run after each surface: **use the `felhom-ui-design` skill.**
- Template methods need **value receivers** — pointer receivers compile, pass `go vet`, pass the
suite, and then 500 at render time.
## Grep fetched pages with ASCII-only substrings
Accented Hungarian patterns get mangled through the `ssh → pct exec → bash -c` chain and return a
false `0` — which reads exactly like the banner or string being gone. Use `kezel`, `Utols`,
`Biztons`. **Never let an accented pattern gate a conclusion.**
<!--
From the 2026-07-20 remediation: an accented grep nearly produced a wrong "banner cleared" claim.
This is the "an absent line is not evidence" rule aimed at a UTF-8 transport, not at a log.
-->
## Credentials containing `!` or `'` break in heredoc-built helper scripts
History expansion eats `!!`. Use the proven inline `-d "password=$PW"` form for authed curl, and
delete any credential-bearing helper from `/tmp` (host AND guest) when done.
+104
View File
@@ -0,0 +1,104 @@
# gates — re-run this repo's gate entry point on every push, on a machine that does not care who
# pushed or what they typed.
#
# *** THIS REPORTS. IT CANNOT REFUSE. ***
#
# felhom repos push straight to `main` with no pull request, so there is no merge for a status
# check to stand at. The refusing half is `.githooks/pre-push`, which is local to a clone and which
# `git push --no-verify` skips; this half is what notices when that happened. Neither half is the
# whole thing, and both are named in felhom.eu documentation/backlog/OPEN-ITEMS.md R-168.
#
# NO `uses:` STEP ANYWHERE, deliberately: JavaScript actions need a node runtime in the runner, and
# the runner is a host-mode container with python3 and git and nothing else (see
# homelab-manifests/gitea-system/act-runner.yaml for why it is not privileged). Probe P3 measured
# that a plain `git fetch` of the pushed SHA from the in-cluster Gitea service is enough.
#
# A failing run must reach a person — a detector nobody hears is the defect R-29 filed, rebuilt one
# layer up. That is the last step, and it runs ONLY on failure.
name: gates
on: [push]
jobs:
gates:
runs-on: felhom-gates
steps:
- name: Fetch the pushed commit and the sibling clone it needs
# This repo's entry point invokes a SHARED checker that lives in the felhom.eu clone next
# door and is deliberately never copied here — so CI has to reproduce the workspace's
# sibling layout or the gate fails closed with "gate is MISSING". The sibling is also
# needed for CONTENT: this repo's REUSE.md cites a path that lives in the hub.
run: |
# Shallow, and pinned to the exact SHA that was pushed — not to the branch tip,
# which can move under us if two pushes race.
mkdir -p ws/felhom-controller
cd ws/felhom-controller
git init -q .
git remote add origin http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom-controller.git
git fetch -q --depth 1 origin "$GITHUB_SHA"
git checkout -q FETCH_HEAD
echo "checked out $(git rev-parse HEAD)"
cd .. && git clone -q --depth 1 http://gitea.gitea-system.svc.cluster.local:3000/admin/felhom.eu.git felhom.eu
echo "sibling felhom.eu present at $(cd felhom.eu && git rev-parse --short HEAD)"
- name: Run the gate entry point
# The ONLY thing CI runs. No go build, no go test, no linting, no deploy. The
# exit code IS the result: no `|| true`, no pipe that could swallow it.
run: cd ws/felhom-controller/controller && python3 scripts/controller_gates.py --fast
- name: Alarm on failure
# THE POINT OF THE WHOLE THING. Probe P5 measured that a failed run produces NO mail, NO
# notification row and NO log line from Gitea itself — a red tick in a web UI nobody watches
# is exactly the shape R-29 filed against. So the run sends its own alarm, on the project's
# existing transactional path (Resend, the same one the hub uses), and prints the provider's
# accepted id so "a message left the machine" is an observable, not an assumption.
#
# Pure python3 and urllib, NOT curl: the runner image carries python3 and git and nothing
# else on purpose, and the first version of this step died on `curl: command not found`.
# Reaching for a bigger image to send one HTTP request would have been the wrong trade.
if: failure()
env:
RESEND_API_KEY: ${{ secrets.RESEND_API_KEY }}
run: |
python3 - <<'PY'
import json, os, sys, urllib.request, urllib.error
key = os.environ.get("RESEND_API_KEY", "")
if not key:
sys.exit("ALARM FAILED: RESEND_API_KEY is empty — the alarm cannot be sent, and a "
"silent alarm is worse than none. Set the user-level Actions secret.")
repo = os.environ.get("GITHUB_REPOSITORY", "?")
sha = os.environ.get("GITHUB_SHA", "?")
run = os.environ.get("GITHUB_RUN_NUMBER", "?")
srv = os.environ.get("GITHUB_SERVER_URL", "https://gitea.dooplex.hu")
body = json.dumps({
"from": "Felhom CI <monitoring@felhom.eu>",
"to": ["admin@felhom.eu"],
"subject": "[felhom CI] gates FAILED in %s" % repo,
"text": (
"The gate entry point exited non-zero.\n\n"
"Repository : %s\n"
"Commit : %s\n"
"Run : %s/%s/actions/runs/%s\n\n"
"The failing gate names itself in the run log.\n\n"
"If the local pre-push hook was GREEN for this commit, then CI and the hook\n"
"disagree - that is a finding about the gates themselves, not about CI, and it\n"
"outranks whatever the push was for.\n"
) % (repo, sha, srv, repo, run),
}).encode()
req = urllib.request.Request(
"https://api.resend.com/emails", data=body, method="POST",
headers={"Authorization": "Bearer %s" % key,
"Content-Type": "application/json",
# Cloudflare fronts api.resend.com and BLOCKS the default
# "Python-urllib/3.x" agent with its own 403 (error 1010) — which looks
# exactly like an auth failure and is not one. Measured 2026-08-02.
"User-Agent": "felhom-ci/1.0"})
try:
with urllib.request.urlopen(req, timeout=30) as r:
print("RESEND-ACCEPTED id=%s" % json.load(r)["id"])
except urllib.error.HTTPError as e:
sys.exit("ALARM FAILED: Resend returned HTTP %s: %s" % (e.code, e.read().decode()[:300]))
PY
+82
View File
@@ -0,0 +1,82 @@
#!/bin/sh
# pre-push — refuse a push that carries a broken gate. (2026-08-02, R-29 leg (b) first half.)
#
# Runs this repo's ONE gate entry point in --fast mode: only checks that touch no network and no
# container runtime, so a push stays a push and never pulls images or starts containers. The slow
# gates stay deliberate periodic runs; a hook that takes minutes gets bypassed within a week and
# the bypass becomes the habit.
#
# BOTH LINES BELOW ARE DELIBERATE. An absent log line is not evidence a hook ran — a silent pass is
# equally consistent with "gates green" and "hook never fired", so a passing push says so out loud.
#
# HONEST LIMITS, stated so this is not mistaken for enforcement it cannot provide:
# * per-clone — core.hooksPath is local config and a clone does not carry it. Arm a clone once:
# git config core.hooksPath .githooks
# Any manual entry-point run WARNS when the clone is unarmed.
# * skippable — `git push --no-verify` bypasses this entirely. That is on purpose: an escape
# hatch that cannot be reached is one that gets removed the first time it is
# inconvenient. USING IT MUST BE STATED IN THE SESSION REPORT.
# The half that is neither per-clone nor skippable is CI — felhom.eu OPEN-ITEMS.md R-168.
#
# Measured 2026-08-02 (git 2.47.3): a relative core.hooksPath resolves correctly and the hook's cwd
# is the repo root whether `git push` is issued from the root or from any subdirectory. The
# explicit rev-parse below does not depend on that.
set -u
root=$(git rev-parse --show-toplevel 2>/dev/null) || {
echo "pre-push: FAIL - cannot resolve the repo root (git rev-parse --show-toplevel)." >&2
exit 1
}
cd "$root" || exit 1
# ── WORKSPACE-ROOT ASSERTION (2026-08-05, R-204 rider) ───────────────────────────────────────────
# Refuse a push from a clone outside the felhom workspace.
#
# WHY THIS IS A HOOK AND NOT A LINE IN A DOCUMENT: the workspace root is ALREADY written down, in
# documentation/runbooks/workspace-CLAUDE.md and in the workspace-root CLAUDE.md ("stay inside it"),
# and work drifted into a home directory anyway. A rule that has failed once as a reminder is not
# fixed by writing it down again — it has to be asserted where it can bite.
#
# A PUSH IS THE RIGHT TRIGGER, deliberately: throwaway clones under /tmp for probes and red-proofs
# never push, so nothing legitimate breaks. Reads and builds elsewhere stay unaffected.
#
# Symlinks are resolved on BOTH sides before comparison, so a symlinked path neither falsely passes
# nor falsely fails. If the workspace root does not exist on this machine the check is SKIPPED, not
# failed — this hook must not brick a legitimate clone on a different host.
#
# The only bypass is the documented `git push --no-verify`, whose use is already reportable.
FELHOM_WORKSPACE_ROOT=/mnt/5_hdd/felhom.eu
if [ -d "$FELHOM_WORKSPACE_ROOT" ]; then
ws_real=$(cd "$FELHOM_WORKSPACE_ROOT" 2>/dev/null && pwd -P) || ws_real=""
root_real=$(pwd -P) || root_real=""
if [ -n "$ws_real" ] && [ -n "$root_real" ]; then
case "$root_real/" in
"$ws_real"/*) : ;; # inside the workspace — proceed
*)
echo "pre-push: PUSH REFUSED - this clone is OUTSIDE the felhom workspace." >&2
echo " clone: $root_real" >&2
echo " expected: under $ws_real (repos live in $ws_real/git/<repo>)" >&2
echo " Work in the workspace clone, or bypass with 'git push --no-verify'" >&2
echo " and state that you did in the session report." >&2
exit 1
;;
esac
fi
fi
if ! command -v python3 >/dev/null 2>&1; then
echo "pre-push: FAIL - python3 not found, so the gates CANNOT run. This is a failure, never a" >&2
echo " pass by default. Install python3, or push with --no-verify and say so." >&2
exit 1
fi
echo "pre-push [felhom-controller]: running controller/scripts/controller_gates.py --fast ..."
python3 "controller/scripts/controller_gates.py" --fast
rc=$?
if [ "$rc" -ne 0 ]; then
echo "pre-push [felhom-controller]: PUSH REFUSED - gates exited $rc. Fix the finding above, or bypass with" >&2
echo " 'git push --no-verify' and state that you did in the session report." >&2
else
echo "pre-push [felhom-controller]: gates OK - push proceeding."
fi
exit $rc
+3064
View File
File diff suppressed because it is too large Load Diff
+88 -181
View File
@@ -1,203 +1,110 @@
# CLAUDE.md — Project Instructions for Claude Code (`felhom-controller`) # CLAUDE.md — `felhom-controller`
> Read automatically at session start. Stable orientation only — **current state lives in > Stable orientation only — **current state lives in `CONTEXT.md` and the top of `CHANGELOG.md`**,
> `CONTEXT.md` and the top of `CHANGELOG.md`**, never here. Cross-repo orientation: workspace-root > never here. Cross-repo conventions (clean-tree gate, secrets, trunk-based, artifact taxonomy):
> `/mnt/5_hdd/felhom.eu/git/CLAUDE.md`. > workspace-root `/mnt/5_hdd/felhom.eu/git/CLAUDE.md`. Path-scoped detail: `.claude/rules/`.
!!! IMPORTANT !!! ## What this repo is
- Always update CHANGELOG.md whenever you modified the code, and pushed to git!!
- IF controller feature changed (new/modify/remove) always update the relevant part of controller/README.md with the architectural change!!
## Project overview The **in-guest controller** — one per customer LXC, Docker-only, **holds NO Proxmox credentials**. It
owns the app domain: stack/deploy management, the Hungarian web UI, app-data backup, metrics,
Felhom is a managed home-server business for Hungarian customers. This repo contains the integrations, git-sync, notifications. Disk/host/Proxmox concerns are delegated to the host agent via
**felhom-controller** — the Go application that manages Docker Compose stacks inside each customer `internal/agentapi`. Whole-guest backup (PBS vzdump) is the agent's, not ours.
LXC guest via a Hungarian-language web dashboard.
Read in this order:
- **`REUSE.md`** — before writing new code (canonical helpers, patterns, traps, seams).
- `CONTEXT.md` — current project state, decisions, roadmap (update after each session).
- `controller/README.md` — full feature/architecture reference (update when features change).
- `TASK.md` — the current task to implement (if it exists).
## System context — the three-component model
The project runs **on Proxmox**, with a locked three-component model:
- **Hub** (`felhom.eu/hub/`) — operator backend on k3s.
- **Host agent** (`felhom-agent/`) — one per Proxmox host; operator-tier; owns ALL Proxmox interaction.
- **In-guest controller** (THIS repo) — one per customer LXC; **Docker-only; holds NO Proxmox
credentials**. De-privileged: disk/host/Proxmox concerns are delegated to the host agent via the
pinned local-API client (`internal/agentapi`); the controller keeps the app domain — stack/deploy
management, the Hungarian web UI, app-data backup, metrics/telemetry, integrations, git-sync,
notifications. Whole-guest backup (PBS vzdump) is the agent's.
> **Authoritative maps:** `felhom.eu/documentation/architecture/01/02/03-*.md` (topology/trust,
> controller module map, host agent) + the code-verified feature docs in
> `felhom.eu/documentation/controller/`. Match the current code, not summaries, if they drift.
**Don't confuse the two ex-"controllers":** `felhom-agent` (host, operator-tier, was **Don't confuse the two ex-"controllers":** `felhom-agent` (host, operator-tier, was
`proxmox-controller`) vs this `felhom-controller` (in-guest, was `deploy-felhom-compose`). `proxmox-controller`) vs this repo (in-guest, was `deploy-felhom-compose`).
## Layout (verified against the tree) ## Doing X → read Y
``` | Doing | Read |
controller/cmd/controller/ entry point + startup wiring (scheduler block, init-only setters) |---|---|
controller/internal/ | writing any new code | `REUSE.md` — canonical helpers, patterns, traps, seams |
agentapi/ pinned-TLS client to the host agent's per-guest local API (THE disk seam) | needing current state / roadmap | `CONTEXT.md` |
api/ REST /api/* router (writeJSON envelope, limitBody, config writes) | needing a feature or architecture reference | `controller/README.md` |
appbackup/ felhom-data paths/namespaces, DB dumps, userdata skeleton (shared primitives) | build, deploy, publish, verify a version | the **`felhom-build-deploy`** skill |
appexport/ .fab export/import bundles (password crypto, strict segment validation) | writing or reviewing a test, fixing a bug | the **`felhom-testing`** skill |
assets/ app logo/screenshot sync from the hub | UI, tokens, badges, Hungarian copy | the **`felhom-ui-design`** skill |
backup/ app-data backup manager, recovery units, tier-2 copies, offbox restic | which box may I break | `felhom.eu/documentation/runbooks/target-selection.md` |
bootstrap/ bootstrap.json ingest → controller.yaml (Day-0 + refresh) | host addresses, break-glass, node facts | `felhom.eu/documentation/operations/nodes.md` |
channelhealth/ agent-channel health checker (debounce + born-down alerting) | what version is live anywhere | ask the hub (`/hosts`, `/configs`) or the box — **never a doc** |
cloudflare/ geo-enforcement remnant (agent-delegated) | the authoritative design | `felhom.eu/documentation/architecture/01/02/03-*.md` |
config/ controller.yaml load/validate (LoadPermissive = setup-mode only)
crypto/ AES-256-GCM app.yaml secret encryption (ENC: prefix)
infra/ traefik/cloudflared/filebrowser base-stack templates
integrations/ app-to-app integrations (e.g. OnlyOffice)
mailrelay/ app-email SMTP shim → hub relay
metrics/ telemetry collection
monitor/ health checks, protected containers
notify/ hub event push (typed Notify* wrappers)
quiesce/ quiesce loop for whole-guest backup (marker + recover)
recovery/ recovery-unit restore
report/ hub report builder/pusher + pull-based config refresh
scheduler/ background jobs (Every/Daily, Budapest DST-safe)
selftest/ startup self-checks
selfupdate/ controller image self-update via the agent swap
settings/ settings.json persistence (registry, flags, corruption recovery)
setup/ first-boot setup wizard (own CSRF)
stacks/ compose ops: deploy/delete/migrate/state (THE app domain core)
sync/ git-sync of the app catalog
system/ mounts/probes (linux + permissive _other stubs)
util/ small shared helpers
web/ dashboard UI: server, auth/CSRF, handlers, funcmap, templates (Hungarian)
```
Per-package helpers/seams/traps: **`REUSE.md`** (maintained same-commit as helper changes). ## Session-critical invariants
## Conventions & cardinal rules The rest live in `REUSE.md`. These cost incidents to learn:
- **Trunk-based — no branches.** All shippable work commits directly to `main`; `main` equals what is
deployed. Report-only artifacts → `felhom.eu/documentation/` (`audits/`, `backlog/`). Risky fixes
are implemented during the supervised session itself, on `main`; if a fix can't be verified/shipped,
revert + report — never park on a branch.
- Code quality: double-check for bugs/edge cases; add debug logging; **ask rather than guess**.
- All UI text is Hungarian (Budapest timezone). Design tokens/gates: use the `felhom-ui-design`
skill; templates must pass `controller/scripts/template_id_gate.py` + `emoji_gate.py`.
- Testing doctrine (non-hollow tests, red-proofs, seams): use the `felhom-testing` skill.
- **Logging**: new leveled lines use `internal/logx` (DEBUG always reaches the debug ring; stdout
respects `logging.level`); English, keys-never-values, durations on outcomes — full rules in
`felhom.eu/documentation/runbooks/logging-conventions.md`.
- Update `REUSE.md` if you added/changed/deprecated a shared helper or pattern (same commit).
- **Coupled features** (controller behavior that depends on a specific agent version): add a
`featureProbes` table row in `internal/agentapi/features.go` + a `Supports` gate call at the
feature's entry point; declare `MinAgent: X.Y.Z` in the CHANGELOG entry header. Rules:
`felhom.eu/documentation/runbooks/publish-train-rules.md`.
> **In every repository where you make a change, update both files in that repo:**
> - **`CHANGELOG.md`** — cumulative log, newest on top.
> - **`REPORT.md`** — **overwrite** with the most recent implementation/validation summary only.
>
> **Never write secrets** into any committed file — reference them as "stored out-of-band".
## Live validation
Exercise the SERVER-SIDE PIPELINE a real user triggers, end-to-end (connect → enroll → deploy). The
forbidden shortcut is BYPASSING that pipeline (the F9 episode: raw agent guest-attach + hand-set
state). **`claude-in-chrome` is NOT available in the DooPlex environment** — the standard method is
endpoint-level: invoke the exact endpoint the UI invokes (no server logic is skipped, only
rendering) and say which method was used. Strict end-to-end UI coverage is a manual click-through.
Two traps in that method, both from the 2026-07-20 remediation:
- **Grep the fetched page with ASCII-only substrings.** Accented Hungarian patterns get mangled
through the `ssh → pct exec → bash -c` chain and return a false `0` — which reads exactly like the
banner/string being gone. Use `kezel`, `Utols`, `Biztons`; never let an accented pattern gate a
conclusion (it nearly produced a wrong "banner cleared" claim).
- **Credentials with `!` or `'` break in heredoc-built helper scripts** (history expansion eats
`!!`). Use the proven inline `-d "password=$PW"` form for authed curl, and delete any
credential-bearing helper from `/tmp` (host AND guest) when done.
## Environment & access
Claude Code runs **on DooPlex (192.168.0.180, Debian 13, user `kisfenyo`)**; repos in
`/mnt/5_hdd/felhom.eu/git/`, build dirs in `/mnt/5_hdd/felhom.eu/build/`. All repos hosted at
`gitea.dooplex.hu/admin/`. Builds are local commands; felhom-pve is one SSH hop.
| Host | Access | Role |
|------|--------|------|
| **DooPlex (this host)** | local — `/mnt/5_hdd/felhom.eu/{git,build}/` | build + push images, `sudo kubectl` |
| Demo Proxmox host `demo-felhom` | `ssh felhom-pve` (root@192.168.0.162) | `pct` into guests; live validation |
| Demo guest 9201 | `ssh felhom-pve "pct exec 9201 -- ..."` | the live demo controller (golden/bootstrap-managed) |
| felhotest (legacy) | `ssh -p 33022 kisfenyo@router.abonet.hu` | OLD /opt/docker compose mechanism |
> **Legacy: Windows workstation.** Until 2026-07-19 CC ran on Windows 11 with repos in `E:\git\`,
> and every remote command needed `SSH=/c/Windows/System32/OpenSSH/ssh.exe` (Git Bash's ssh lacks
> the Windows agent and fails silently — see `docs/vscode-ssh-fix.md`), plus `MSYS_NO_PATHCONV=1`
> for `pct exec`. Retained in case that environment is revived.
> **TEMPORARY — felhom-pve is at a remote site (until ~2026-08-02).** The home-LAN literal
> `192.168.0.162` is NOT reachable from DooPlex for the duration. Access via Tailscale:
> felhom-pve = 100.70.170.35; the `Host felhom-pve` entry in `~/.ssh/config` on DooPlex already
> points there (the direct-LAN path stays available as `Host felhom-pve-lan`). Delete this block on
> return. All documented `ssh felhom-pve` / `pct exec` workflows are unchanged. Path is **direct**
> (not DERP), ~37 ms rtt per hop. At the remote site the host is on **DHCP** and currently holds
> `192.168.0.147` (the guest holds `.104`); no Pi-hole there — the guest reaches `gitea.dooplex.hu`
> and `*.demo-felhom.eu` via public paths. **The host agent is DOWN for the duration**: its
> `localapi` binds the literal `192.168.0.162`, which no longer exists → `bind: cannot assign
> requested address`, so every agent-backed feature (storage, PBS backup, quiesce, restore-test, DR)
> is unavailable until fixed. Details + findings:
> `felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md`
External access via Cloudflare Tunnel → Traefik; Pi-hole forwards `*.demo-felhom.eu` → .162 locally.
## Build & deploy — MANDATORY after code changes
**Full runbook: use the `felhom-build-deploy` skill.** Summary (guest 9201 is bootstrap-managed —
**no compose file**; `felhom-controller-bootstrap.service` runs the tag in `/etc/felhom-controller-image`):
> **Clean-tree gate before any build:** `git status --porcelain` must be empty and
> `git rev-parse HEAD` must equal `git rev-parse origin/main` in the repo being built. An unpushed
> change does not exist — never build a dirty or unpushed tree. The `git pull` in the build step
> stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from
> elsewhere).
| Step | Command |
|------|---------|
| 1. Commit + push | `git add <explicit paths> && git commit -m "..." && git push` |
| 2. Build + push image | `cd /mnt/5_hdd/felhom.eu/build/felhom-controller && git -C /mnt/5_hdd/felhom.eu/git/felhom-controller pull && ./build.sh <VER> --push` (build.sh does NOT pull — the explicit pull is load-bearing) |
| 3. Deploy (9201) | `ssh felhom-pve "pct exec 9201 -- bash -c 'docker pull gitea.dooplex.hu/admin/felhom-controller:<VER> && echo gitea.dooplex.hu/admin/felhom-controller:<VER> > /etc/felhom-controller-image && systemctl restart felhom-controller-bootstrap.service'"` |
| 4. Verify | `ssh felhom-pve "pct exec 9201 -- docker ps --filter name=felhom-controller --format '{{.Image}} {{.Status}}'"` + container logs |
Hub build/deploy lives in `felhom.eu` (GitOps) — see that repo's CLAUDE.md / the skill. Catalog
changes (`app-catalog-felhom.eu`): commit+push; controller sync picks them up ≤15 min or via the
"Sablonok frissítése" button.
## Session-critical invariants (the rest live in REUSE.md)
- `docker compose restart` does NOT pick up new images/env — always `up -d` (`RedeployFromEnv`). - `docker compose restart` does NOT pick up new images/env — always `up -d` (`RedeployFromEnv`).
- Docker's `.State` says "running" even for unhealthy containers — `.Status` parse is the truth. - Docker's `.State` says "running" even for unhealthy containers — the `.Status` parse is the truth.
- In-memory `Deployed` flag is set BEFORE `compose up -d` (slow-pull race); reverted on failure. - In-memory `Deployed` is set BEFORE `compose up -d` (slow-pull race); reverted on failure.
- `compose up -d` exits 0 on crash-loops — post-start status check is the detection. - `compose up -d` exits 0 on crash-loops — the post-start status check is the detection.
- Env var KEYS are logged, never values. Protected stacks (traefik, cloudflared, felhom-controller) - Env var KEYS are logged, never values. Protected stacks (traefik, cloudflared, felhom-controller)
can't be stopped from the UI. cannot be stopped from the UI.
- Verify a container image HAS the healthcheck tool before using it (BusyBox wget / python3 / curl — - Verify a container image HAS the healthcheck tool before using it (BusyBox wget / python3 / curl —
catalog REUSE.md maps the families). the catalog `REUSE.md` maps the families).
- `IsRunning()` is CONCURRENCY, false during a verification restore — display MUST use
`RestoreStatus()`.
## Live validation — the fence
Exercise the SERVER-SIDE PIPELINE a real user triggers, end-to-end (connect → enroll → deploy). **The
forbidden shortcut is BYPASSING that pipeline** — the F9 episode was a raw agent guest-attach with
hand-set state, and it proved nothing.
`claude-in-chrome` is NOT available on DooPlex. The standard method is endpoint-level: invoke the
exact endpoint the UI invokes (no server logic is skipped, only rendering) and **say which method was
used**. Strict end-to-end UI coverage is a manual click-through by the operator.
Two traps in that method live in `.claude/rules/ui-hungarian.md` (ASCII-only greps; `!` in
credentials) — they load when you touch a template or stylesheet.
## Commands — one per surface
| Surface | Command |
|---|---|
| Gates (after ANY change) | `python3 controller/scripts/controller_gates.py` — from `controller/` |
| Green gate | `go build ./... && go vet ./... && go test ./...` |
| Build + deploy | the **`felhom-build-deploy`** skill — do not hand-roll it |
Guest 9201 is **bootstrap-managed — there is no compose file**;
`felhom-controller-bootstrap.service` runs the tag written in `/etc/felhom-controller-image`. Catalog
changes (`app-catalog-felhom.eu`) are picked up by controller sync ≤15 min, or via the "Sablonok
frissítése" button.
## Working with CHANGELOG.md ## Working with CHANGELOG.md
**DO NOT read the full file** — it is large and will waste context. **DO NOT read the full file** — it is large and will waste context.
- Session start: use `CONTEXT.md` + `controller/README.md` for current state.
- Session start: `CONTEXT.md` + `controller/README.md` for current state.
- Adding an entry: Read only the top ~30 lines for format, then Edit-insert after line 1. - Adding an entry: Read only the top ~30 lines for format, then Edit-insert after line 1.
- History: Grep for topics instead of reading. - History: Grep for topics instead of reading.
## End-of-session checklist ## End-of-session checklist
1. **Commit and push** all code changes 1. **Commit and push** all code changes (explicit paths; no `git add -A`).
2. **Build, push, and deploy** the new controller image (if controller code changed) 2. **Build, push, and deploy** the new controller image, if controller code changed.
3. **Update CHANGELOG.md** with what was done 3. **`CHANGELOG.md`** — always, whenever code changed and was pushed.
4. **Update CONTEXT.md** with decisions made, state and what's next 4. **`CONTEXT.md`** — decisions made, state, what is next.
5. **Update controller/README.md** if architecture or features changed 5. **`controller/README.md`** — whenever a feature was added, modified or removed.
6. **Verify** the deployment is working (check `docker ps` and logs) 6. **`REPORT.md`** — overwrite with this run's summary only.
7. **Update REUSE.md** if you added/changed/deprecated a shared helper or pattern (same commit) 7. **`REUSE.md`** — if a shared helper or pattern was added/changed/deprecated (same commit).
8. **Verify** the deployment (`docker ps` + logs).
<!--
WHY THIS FILE IS SHORT (2026-08-06, instruction-trim task).
Removed from here and rehomed, not lost:
- the `## Layout (verified against the tree)` block -> derivable by `ls internal/`; REUSE.md
carries the per-package seams and traps that the annotations were really for.
- the `!!! IMPORTANT !!!` header -> its two requirements are checklist items 3 and 5. One voice,
one place; a rule stated twice in one file is a rule that gets edited in one of them.
- the host/access table -> documentation/operations/nodes.md is the single home. The copy here
had drifted: it gave demo-felhom as plain root@192.168.0.162 (the LAN fallback, not the route),
pinned "agent 0.93.0" against the project's own no-versions-in-docs rule, and claimed no drill
VM was provisioned on demo-hp. Measured 2026-08-06: `qm list` on demo-hp shows VM 300
`drill-r50` present. felhom-agent/CLAUDE.md was right; this file was wrong.
- the "felhom-pve is back on the home LAN" block -> it was bookkeeping about a retired block; the
record is in documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md.
- the "Legacy: Windows workstation" block -> the workspace-root CLAUDE.md carries the full version.
- the gates/logging/coupling/UI paragraphs -> .claude/rules/*.md, which load when a matching file
is read instead of in every session.
Full per-block accounting: felhom.eu/documentation/audits/LEDGER-instruction-trim-2026-08-06.md
-->
+799 -1
View File
@@ -7,7 +7,642 @@
> >
> Ask Claude Code: "Please update CONTEXT.md with what we did today" > Ask Claude Code: "Please update CONTEXT.md with what we did today"
Last updated: 2026-07-24 (v0.162.0 — R-71a: the apply-bridge settle-gate) Last updated: 2026-08-14 (v0.215.0 — R-328..R-333: the disk alert that never sent)
> **2026-08-14 — v0.215.0 (R-328..R-333). Disk health, phase 1: the alert that reached nobody.**
>
> ### A severity string is a WIRE CONTRACT with the hub, not a label we choose.
>
> The hub accepts exactly `{info, warning, error, critical}` and **silently coerces anything else to
> `info`**, which `severityNotifies` then drops. `disk_health_degraded` shipped `"warn"` — one letter
> short of the contract — so **every Figyelmeztetés-level disk alert this product ever produced was
> emailed to nobody, on both legs.** Proven live side by side on 2026-08-14: `"warning"` →
> `notification_log` status **`sent`**; `"warn"` → stored `info`, **no row at all**.
> `app_start_failed` (`notifier.go` ~L546) carries the identical defect and was deliberately NOT
> changed here — it needs its own decision on whether it should notify (**R-329**).
>
> ### A drive's own PASSED verdict cannot fail on bad sectors. Do not build on it.
>
> Attributes 187/197/198 all carry `thresh: 0`; a normalized SMART value floors at 1 and can never drop
> to or below the threshold. The real drive (ST3000VX010, S/N Z6A07P2G) read `PASSED` at **352** pending
> sectors and **1001** reported-uncorrectable reads. Felhom already read the raw counters, which is the
> only reason it would have noticed at all. Evidence + fixtures:
> `felhom.eu/documentation/audits/DIAG-smart-passed-trap-2026-08-14.md`.
>
> **DECISIONS MADE, so they are not re-litigated:**
>
> - **Predicted failure is labelled „Hiba" — there is no fourth verdict word.** A fourth Hungarian word
> sharing a root with „Figyelmeztetés" would make the MORE severe state read as the milder one. Four
> labels, final: Rendben / Figyelmeztetés / Hiba / Nincs adat.
> - **Sustain is the primary rule; the count is the backstop.** Truth-table row 6 (unreadable sectors
> present again at the next check) sits ABOVE row 8 (count >= 64) because on the real drive sustain
> fires 12 Aug and the count not until 13 Aug. Row 8 exists only for a box powered off across the
> sustain window.
> - **The numbers and where they come from.** 64: the benign excursion peaked at 16 and cleared inside
> an hour; the terminal run passed 64 at 13 Aug 11:28 and never returned. **It is a judgement from ONE
> drive** — a static backstop, expected to be replaced by growth-rate detection in Phase 3. 55/60 °C:
> the operator's existing Prometheus bands on DooPlex, adopted unchanged so the two systems cannot
> disagree about the same drive. **These bands are SPINNING-DISK bands and are questionable for NVMe**
> — demo-hp's healthy Toshiba NVMe idles at **53 °C**, 2 °C below Figyelmeztetés (**R-333**).
> - **Phase 2 owns the new SMART attributes (187 Reported_Uncorrect, 199, 188).** They are a declared
> wire change, so under the G-1 gate the hub must model them in the same session. Putting them here
> would have turned a one-word severity fix into a three-repo change (**R-330**). Everything v0.215.0
> needs was already on the wire.
> - **Phase 1 state is one small record per disk, NOT a sample series.** `metrics.MetricsStore` is the
> right home for Phase 2/3 history; using it now would have put a schema migration on the critical
> path of the severity fix.
>
> **The trap this change nearly shipped, caught by a test and not by review:** the card and the check
> share one verdict function so the chip and the email can never disagree — but the check CONSUMES the
> prior and then overwrites it, so a card rendering afterwards read its own check's write and showed one
> level MORE severe than the alert. Fixed by `diskRecord.PriorSawUncorrectable`, which replays the prior
> that produced the stored verdict. The guarantee was previously asserted in a comment only.
>
> **Cadence is hourly, and it was MEASURED:** demo-hp `/disks` costs median 0.821s (min 0.805 / max
> 0.841, 10 calls, 3 physical rows) — 6x under the 5s bar. Open question deliberately NOT acted on: the
> agent runs bare `smartctl -a -j` with **no `-n standby`**, so an hourly poll would wake a spun-down
> HDD. demo-hp is all-flash so the measurement could not show it (**R-333**).
>
> **NOT live-validated:** the Fail-from-counters path has never fired on real hardware — only against
> the fixture's values in unit tests (**R-332**).
> **2026-08-08 — v0.208.0 (R-254). THE RULE, stated so it outlives this session:**
>
> ### A secret is never in a page's response body. It is fetched by an explicit act, and the act is recorded.
>
> Three instances of one pattern shipped in two days, each found by hand: the retrieval passphrase
> (R-249), an app's real first-login password (R-254 site one), and an already-deployed app's generated
> secret field (R-254 site two). Every one was "hidden" with `display:none`, `hidden`, or
> `type="password"` — **instructions a browser honours when DRAWING and nothing else.** The plaintext
> was in the bytes; a `curl` returned it; caches, history, saved pages and screen-shares had it.
>
> **The shape of the fix, now used three times:** the page carries a BOOLEAN; the value comes from a
> **POST** (so CSRF covers it and it is not re-fetchable from history) with **`Cache-Control:
> no-store`**; the reveal is **LOGGED as an act** — reading a value off markup left no trace anywhere,
> which is why nobody can say whether any of these was ever read. **Per-secret endpoints, never one
> generic "reveal any named secret"** — that would turn three narrow exposures into one lever.
>
> **And the test must assert the RAW RESPONSE BODY.** Every test that asked what the customer *sees*
> passed while the bytes carried the secret. That is precisely how this survived three times.
>
> **What is NOT this defect:** a form must carry what it submits. The pre-deploy hidden input round-trips
> a generated secret deliberately (README §318) so the saved value is the one the customer wrote down.
> The defect there was the neighbouring READONLY input on an already-deployed app, where nothing is
> submitted at all.
>
> **The gate:** `scripts/secret_in_markup_gate.py`. Name-based, all 36 templates, **blind to a secret
> arriving under a neutral page-data key** — measured, not assumed. The complementary runtime
> body-assertion covers 4 of 27 page templates; the other 23 are **R-255**.
>
> **A correction to v0.207.0's report:** it said HTML comments ship in the response body. They do not
> here — `html/template` strips them (`text/template` does not). Measured.
> **2026-08-08 — v0.207.0 (R-249, R-252, R-253). Three things the fifth walk exposed BY PASSING.**
> The walk closed R-201 (both halves) on 2026-08-07; none of the below touches the recovery path it
> proved.
>
> **R-249 — a secret was living in the page source.** `settings_security.html` rendered the retrieval
> passphrase into a `display:none` span behind a „Megjelenít" button. That toggle stops a browser
> DRAWING it and nothing else: the plaintext was in the response body of every render. Found by doing
> exactly that — it landed in a session transcript while driving the documented rebuild path.
> **THE RULE, which the codebase already stated for R and this page did not follow:** a secret is
> revealed by an XHR, never templated server-side into HTML (`escrow_handlers.go`). The page now
> carries only `HasRetrievalPassword`; the value comes from `POST /settings/retrieval-password/reveal`
> — CSRF-covered, `no-store`, and **logged as an act**, which reading it off the markup never was.
> **The test asserts the RAW RESPONSE BODY** — every test that asked what the customer *sees* passed
> while the bytes carried the secret, and that is why it survived.
> **The census found two more instances** (`app_info.html`, a real per-install app password in a
> `hidden` span; `deploy.html`, a generated secret in a `value=`) — **filed as R-254, not fixed.**
>
> **R-252 / R-253 — the two obstacles, and the rule they share.** A rebuilt box keeps its drives but
> loses their REGISTRATION, so every restore refused with a sentence naming no next step; and the
> restore list promised „a visszaállítás előbb újratelepíti" three lines above a refusal that fired
> *because* the app was not installed. **The promise was the wrong half:** reconstitution writes to
> the app's own `GetStackHDDPath`, which exists only once the CUSTOMER has chosen a drive at deploy
> time — an automatic reinstall would mean the product making that choice for them, which is the one
> decision this recovery path exists to leave with them. Both now name a reason and route to the step
> that clears it, and both notices are conditional (a healthy box is byte-identical, pinned by a test
> that fails if either becomes unconditional).
>
> **The page and the resolver ask ONE question:** `HasRestoreDestination()` reads the same
> `GetSchedulableStoragePaths()` the scratch resolver reads. A second copy of that predicate is
> exactly how a page ends up promising what the handler refuses — which is R-253 itself.
> **2026-08-07 — v0.206.0 (R-241). THE RULING, and it reversed the fix: this was a MINTING defect,
> not a screen-predicate defect.** The recovery screen was telling the truth — there genuinely was
> nothing recoverable under the key the box held, because **the box minted that key itself over the
> top of a sealed package it already knew the hub was holding**. Fixing the predicate would have
> papered over a machine quietly making its own backups unopenable.
>
> **THE RULE: a box does not create a repository key while the hub holds a sealed package for it.**
> The guard is a conjunction (package held AND no key), so a first-time box is untouched, and the
> refusal is a HOLDING state rather than a failure — the transport is still configured so the
> recovery screen can bring the tier up the moment the key arrives.
>
> **THE SECOND RULE: the fact that answers a question must be kept where the question is asked.** The
> hub-vs-local key comparison had been computed on every ACK since SLICE 3 and persisted nowhere; on
> the venue it logged the right answer thirty-five minutes before the customer looked at a screen
> that could not see it. It is now persisted and drives shape (c) of the offer.
>
> **THE THIRD RULE (the operator's, and it generalises): fix the state, do not remember that it is
> wrong.** Abandoning the old history now starts a 14-day countdown that removes the set-aside store
> and its sealed package TOGETHER, after which the offer falls silent on its own because there is
> nothing left to compare — rather than a "they decided" flag suppressing a screen over a state that
> is still wrong. The recovery offer stays reachable for the whole grace; a grace in which recovery
> is impossible is decorative.
>
> **Surface:** the full page appears once per ENTRY into the offered state, not once ever — a box
> rebuilt months later is a new situation. Three dismissal levers with three scopes, and none of them
> removes the entry point on the backups page.
>
> **Needs hub v0.98.0** for the superseded-package purge. `felhom-agent` untouched.
>
> **Two real bugs were caught by tests rather than by review** — a missing `t.Enabled` (an existing
> test) and a missing falling-edge sync that reintroduced the very defect the epoch exists to fix.
>
> **NOT built, deliberately:** the automatic 30-day abandonment (R-245, with the operator's reasoning
> recorded), and R-242's release-to-golden gate.
> **2026-08-06 — v0.205.0 (R-234).** THE RULE: **a run that skipped an app the customer selected is
> not a successful run.** The R-203 verdict block already said *"a warning beside a success is read
> as a success"* and applied it to one of the two shapes it describes — a missing declared FOLDER
> made the run `incomplete`, an app skipped ENTIRELY did not. Now both do. A selected-but-UNDEPLOYED
> app is named with what to do but does NOT move the verdict, because a box left permanently amber by
> an app somebody removed is a status nobody reads.
>
> **§7.3, MEASURED rather than assumed — and the answer was "already done".** `CaptureRecoveryUnit`
> writes compose config + a manifest (a few KB), only ENUMERATES dumps rather than creating them, is
> idempotent, and does NOT stop the app; the off-site run already calls it for every deployed stack in
> its own pre-dump phase, through `admitApp`. So there is no wait to remove for a deployed app, and
> **nothing was built**. Proven on demo-hp: a unit moved aside was RECREATED by the run.
>
> **AND THE FILED MECHANISM WAS NOT THE MEASURED CAUSE.** R-234 was filed as "the first run after a
> toggle finds no bundle and skips the app". That cannot happen for a deployed app (above). What did
> happen on 2026-08-06: the manual run was dropped by the **single-flight** while an earlier run was
> still going; `runOffboxBackup` returned nil; the handler had already said „elindult”; and the card
> then showed the PREVIOUS run's „✓ Rendben”. Fixed by taking that decision synchronously in the
> handler. **The nightly path deliberately still returns nil** — nobody asked, and it retries.
> **2026-08-05 — v0.200.0 (R-193 CLOSED).** The customer-facing recovery screen. Until now a customer
> whose machine was rebuilt had everything needed to get their data back and no way to find out — the
> only route was a command line.
>
> **IT UNLOCKS AND ONLY UNLOCKS** (operator ruling). Explains, takes the recovery code, opens the
> repository, lists what is in it (apps, dates, sizes). **Restores nothing** — restore is per-app and
> lives in the backups area; the put-back is **R-213** and its stated requirement is a
> live-versus-backup comparison.
>
> **ONE CORE, TWO CALLERS.** `backup.RecoverInstallCore` is the only fetch→unseal→compare→install path.
> `RecoverAndInstall` is now a thin CLI wrapper — exit codes and printed lines byte-identical, every
> pre-existing CLI test passed unchanged — and the handler calls the same function. Asserted from
> source by AST on BOTH sides, plus a test that the routes and the landing-page interception exist.
>
> **THE TRIGGER HAS TWO SHAPES and the second is the one that matters.** `OffsiteRecoveryOffer` = the
> hub holds a package AND (no repository password OR the tier is orphaned). The literal "no repository
> password" alone is a window that CLOSES BY ITSELF — `WriteOffboxSecrets` auto-generates one on
> re-apply (R-193's own orphaning mechanism) and hub v0.96.0's self-heal re-applies within ~1530 min.
> Shape (b) is also what the shipped move-aside requires, which is why the discard choice can reach it.
>
> **CLAIMED is part of the predicate** — a legacy-open box passes through `RequireAuth`, so without an
> explicit `authEnabled()` check the interception fired for an unauthenticated visitor. A test caught it.
>
> **„Most nem" suppresses the FULL PAGE ONLY.** The backups-area entry point is bound to
> `recoveryOffer`, never to the postpone flag.
>
> **The code:** POST body only, never logged/persisted/echoed, cleared on every path, `no-store`,
> `autocomplete=off`. **No lockout** — a ten-word phrase is not guessable and locking a customer out of
> their own data for a typo is worse; failures are logged locally without the code, and NO operator
> alert is raised (reasoning in REPORT.md §4).
>
> *Live:* demo-felhom is genuinely in shape (b), so validation needed no arrangement — `/launcher` →
> 302 `/recovery`, both mandatory sentences rendered, three wrong codes refused with the `offbox/`
> listing byte-identical and no lockout, and the code found in no file, log or ring **with a
> planted-copy positive control that first exposed a mis-aimed sweep**. **NOT proven live: a CORRECT
> code** — none was kept for demo-felhom's orphaned history and demo-hp's is operator-held.
> **2026-08-05 — v0.199.0 (R-204 item 4 / R-193).** The last of the four manual interventions the
> 2026-08-04 drill needed. **Operator ruling: automate it, and the trigger is a state the BOX
> DECLARES.** From the hub an absent off-site object has FOUR meanings — never configured,
> mid-restart, a transient config read failure, rebuilt-and-stranded — and the hub cannot tell them
> apart. The box can.
>
> **The declaration needs BOTH halves** (`backup.needsOffsiteCredential`): a fresh data area (no
> repository password) AND a hub-held recovery package (the ACK's `identity_blob_present`). Freshness
> alone is a box that never had off-site backups; dropping that condition makes the whole fleet ask
> for credentials, which is what `TestOffsiteDeclare_NeverHadOffsiteSaysNothing` catches. A merely
> DISABLED target is the customer's own choice and never declares.
>
> **The ACK field stopped being discarded.** `EscrowAutoConfirmer.Reconcile` returns early when the box
> is neither pending nor escrowed — exactly a rebuilt box — so the fact was thrown away every cycle. It
> is recorded FIRST, before every gate, via `RecordPresence`, wired in main.go and asserted by
> `TestMainWiresRecordPresence` (AST, comments dropped). Last-write-wins, not set-only, so a customer
> RESET turns the declaration back off; a nil ACK escrow records nothing.
>
> **Inert to every existing reader:** `enabled:false` + zero sizes, so the hub's `isStale` and
> `fillBand` both short-circuit; an unknown `state` string is ignored by encoding/json. **A configured
> box's report JSON is byte-identical to v0.198.0's.** The one reader that would have misread it is the
> hub's `reportHasOffsite`, tightened in hub v0.96.0 to require `enabled:true`.
>
> *Live:* both demo boxes now record `hub_escrow_identity_present=true` in settings.json (the recorder
> working on a HEALTHY box). demo-felhom 9201, arranged reversibly into the stranded shape, produced
> report id=16743 carrying `{enabled:false, state:needs_credential, quota_gb:0, repo_size_bytes:0}`;
> the single declaration was absorbed by the hub's debounce (no self-heal event) and the box was
> restored the same minute. **The hub half is felhom.eu v0.96.0.**
>
> *Rider:* `.githooks/pre-push` in all four repos now refuses a push from a clone outside
> `/mnt/5_hdd/felhom.eu`. Proven both ways against a scratch clone.
> **2026-08-05 — v0.198.0 (R-204 items 1 & 3).** The 2026-08-04 drill (R-201) passed only because a
> person was there; four manual interventions stood between a recovered key and a restored file. Two
> of the three defects are in this repo.
>
> **Item 1 — the reset code needed a restart.** `--print-reset-code` is a SEPARATE process; it
> persisted a new code while the running server kept the old one cached, so the code the customer was
> told to type was refused until the controller restarted, and nothing said so. `effectiveClaimCode`
> now calls `settings.ReloadClaimCode()` first. **The settings-vs-config precedence is unchanged** —
> the defect was freshness, not precedence. **Read-through, not a TTL, and that is the point:** a TTL
> makes the new code visible AND leaves a window in which the superseded one still works, which is
> worse than the bug. That is the mutation `TestClaimCode_SupersededByASecondMint_RefusedImmediately`
> exists to kill, and its red-proof produced exactly *"the SUPERSEDED code was accepted"*. The
> function now returns an error and **every caller fails closed**; an absent settings file is NOT an
> error. `ClaimConsumedGeneration` is deliberately NOT re-read — this process is its only writer and
> re-reading could move it BACKWARDS if a save had failed, resurrecting a consumed code.
>
> **Item 3 — the restore's default returned the wrong thing silently.** `mode=unit` restores the
> recovery unit (definition + config + DB dumps) and not the customer's files. `restoreScratchOutcomeMsg`
> now names what came back, what did not, and the next step; the wizard's intent card states its scope
> before the choice. **The size gate is untouched** and pinned unchanged by
> `TestOffboxRestore_FullPathUnchanged`. **The default stays `unit`** — all three wizard forms set
> `mode` explicitly, so a change would alter nothing visible while silently changing a mode-less POST.
>
> *Live-validated endpoint-level (no browser on DooPlex):* on demo-felhom 9201 with `restarts=0`
> across both mints, a superseded code returned „Hibás vagy lejárt kód" and the current one was
> accepted first time; on demo-hp 9201 a `privatebin` unit restore produced the scoped Hungarian
> outcome and `mode=full` without confirm revealed `full_size=6.8+KB` without restoring anything.
> demo-hp's drill scratch (`calibre-web`) was not touched.
>
> **Item 2 is the hub's** (felhom.eu v0.95.0, R-196). **Item 4 — a rebuilt box cannot obtain an
> off-site credential unaided — remains OPEN (R-193)** and was deliberately not begun.
> **2026-08-02 — v0.190.0 (R-157 mechanism A · R-170 · R-171).** Three items, one live validation
> cycle, because all three are boot behaviour and all three are proven by hard-resetting the box.
>
> **DIAGNOSE BEFORE THEORISING — and the first diagnosis was a FALSE NEGATIVE.** A hole was reasoned
> out of the v0.189.0 diff (a drive-gate-stopped app has zero containers and `desired_state: running`,
> so it now reads as a boot orphan) and confirmed on hardware BEFORE any fix was written. **Attempt 1
> produced `no boot-orphaned apps` and would have been reported as a disproof.** It was a race:
> unmounting only the parent bind is healed by the agent within ~60 s, so the drive gate's startup
> reconcile restarted the apps **one second before** the sweep looked. Holding the drive genuinely
> absent reproduced the defect immediately. **"It didn't happen this time" is not a mechanism.**
>
> **The confirmation moved the severity in BOTH directions.** The write hazard did not materialise —
> compose failed `mkdir …/userdata: permission denied` because the unbound mountpoint is
> host-root-owned and the guest is unprivileged. **That protection is ACCIDENTAL**: no code chose it,
> no test pinned it, and it is one `chown` or one privileged guest away from gone. But the harm that
> DID occur was not in the hypothesis and is real on every box: two wasted attempts and a **false
> dead-app alarm for an app the drive gate is deliberately holding**.
>
> **The fix already existed one path over.** `startGatedByMissingDrive` (the API) refuses a customer's
> start on an absent drive; the sweep bypassed it by calling `Manager.StartStack` directly.
> **`StartStack` HAS NO GATE OF ITS OWN** — carry this: every caller that is not the customer must
> decide for itself whether the app may run. New consumer-side `bootrecon.StartGate`, fail-safe
> (cannot determine ⇒ do not start).
>
> **Widening a window makes previously-unreachable overlaps reachable — a design input, not an
> afterthought.** The old T+5 s sweep never met a quiesce or an in-flight app-data operation; a 50 s
> window can. All three holders answer ONE seam because they differ only in the reason string.
>
> **A TEST REJECTED MY FIRST CONSTANT, and the comment says so.** `settle + budget + one retry` must
> fit inside `deadAppBootGrace`; 60 s gave 95 s against 90 s. The budget is 50 s **because a test said
> so** — recorded in the code rather than presented as taste. Widening the grace was rejected: it
> hides a late recovery instead of reporting one (`recordLateRecovery`).
>
> **THE FIX HAD ITS OWN DEFECT, FOUND LIVE AND NOT BY REVIEW.** The window sampled `GetStacks()` — the
> Manager's map, refreshed by the scheduler every **10 s** — every 5 s, so two identical samples could
> mean *the cache did not update*. Observed: a container removed ~5 s before the window closed was
> still in the sampled fleet and the sweep logged `no boot-orphaned apps` for an app that had none.
> `sampleBootFleet` now refreshes first. **Generalise: a settle detector is only as good as the
> freshness of what it samples — if the source is cached, refresh it, or you are watching the cache
> settle rather than the system.**
>
> **R-170:** `shouldRecreateOnBoot` reads intent with the identical three-way table; absent keeps the
> old `hasContainers` behaviour exactly; `presentStable` untouched and still load-bearing. Its comment
> argued at length FOR the count and was rewritten. Agreement pinned from BOTH sides against one
> fixture table (an import cycle prevents testing the two gates together).
>
> **Live: 6/6 hard resets** (every app back; the customer-stopped app down all six), settle times
> 10/40/10/10/15/15 s. Sharpest evidence: same app, same box — missed at 18:08:35, recovered at
> 18:18:50. R-170 proven in one reboot (calibre-web recreated, immich left stopped). 27/27 packages;
> 7 red-proofs. Detail: `REPORT.md`.
Last updated: 2026-08-02 (v0.189.0 — R-166 / D-b: the box stops guessing what the customer wanted)
> **2026-08-02 — v0.189.0 (R-166, operator decision D-b).** When an app was not running the box had
> to work out *why*, and it did so **by counting containers**: zero meant "the customer stopped it",
> some meant "something broke". A **power cut mid-compose** and an **interrupted deploy** also leave
> zero containers, so both were read as deliberate stops and stranded **silently** (R-157 mechanism
> B) — and a backup that stopped an app and died left it stopped with **nothing on disk** recording
> that it was owed a restart. The settling fact — what the customer asked for — **was written down
> nowhere**: `app.yaml` recorded *installed*, never *meant to be running*.
>
> **DECISION — one owner: the customer's action, and nothing else.** A census found **14 callers of
> `StartStack`/`StopStack`, of which exactly 2 are the customer**; the rest are quiesce, the volume
> dump, offbox reconstitution, app export/restore, the storage gate, migration and the boot
> reconciler. So the primitives are deliberately **not** writers — intent there would make a nightly
> backup indistinguishable from the customer pressing Stop. Writers: the API action switch,
> `DeployStack`, `UpdateOptionalConfig`'s redeploy branch, the `.fab` import. Intent is written
> **BEFORE** the act and a failed write **REFUSES** the act.
>
> **DECISION — absent means UNKNOWN, never "running", and this is the whole safety property.** Every
> `app.yaml` on every box predates the field, so absent is what the fleet reads on upgrade; reading
> it as running would start every deliberately-stopped app on the first boot after the upgrade. The
> legacy branch of `isBootOrphan` keeps the old container-count rule **byte-for-byte**, and its test
> asserts BOTH legacy rows together because the safety property is the pair. Backfill is
> **running-only** — "zero containers ⇒ stopped" IS the defect, so an ambiguous app stays ambiguous.
>
> **Part 2 — `backup.AppStopGuard`**, a persisted marker over every stop→work→start window (volume
> dump, offbox reconstitute, `.fab` export), in its **own** file (one file, one writer). Written
> before the stop, cleared only after a restart that **succeeded**, kept when one fails. `Recover`
> **returns** its outcome instead of using a notifier seam, because it must complete before the boot
> reconciler (`main.go` ~236) while the notifier is not built until ~307 — a seam wired after the
> fact is a seam that never fires.
>
> **THE TEST LESSON, and it is the one worth carrying:** Scenario E's first version called
> `appStop.Begin` itself, and **survived the red-proof that deleted the production call**. It proved
> the marker type, not that `DumpAppVolumesSafe` uses it. Rewritten to drive the real function with a
> simulated hard abort (an unwind that skips the restart statement, since a `defer` is not
> crash-safety — Campaign 8 fault 10). **A test that constructs the thing it is meant to prove the
> caller constructs is hollow, and its red-proof will say so if you run it.**
>
> **FOUND EN ROUTE — `SaveAppConfig` rebuilt `AppConfig` field-by-field**, the R-100 shape (v0.181.0
> shipped two live instances). The literal named five fields, so `desired_state` would have been
> dropped on **every** save across nine call sites — a customer's Stop erased by the next unrelated
> `app.yaml` write. Copy-and-overlay (`saveCfg := *cfg`) is safe by construction. **Generalise it:
> treat any field-by-field struct rebuild in a save path as a defect on sight.** Measured, not
> assumed: `app.yaml` does NOT round-trip YAML keys the struct does not model (pinned by test).
>
> **R-157: mechanism B closed, mechanism A untouched** (the sweep observes ~5 s after start and never
> re-checks) — and B's fix makes A cost more, since the sweep now has more it could recover.
> **NEW R-170:** `shouldRecreateOnBoot` (`internal/web/intermediary.go:131`) still infers a Stop from
> `hasContainers` — the same defect one gate over, for drive-backed apps. Left deliberately.
>
> **Live on 9201, three flows** (stop survives a restart; a zero-container `running` app recovered by
> name; a legacy app.yaml skipped and never inferred stopped). The **interrupted-operation half is
> IMPLEMENTED, not PROVEN-LIVE** — nobody killed the controller mid-backup on metal. 27/27 packages;
> 7 red-proofs observed FAIL then restored. Detail: `REPORT.md`.
Last updated: 2026-07-28 (v0.182.0 — R-101 + F-DIAG: the restore dialog names the last SUCCESSFUL copy)
> **2026-07-28 — v0.182.0 (R-101 + F-DIAG).** `Tier2LastRun` is the ATTEMPT clock (written on failure)
> and was rendered as „Legutóbbi másolat" in the **restore confirm dialog** — misinformation at a
> decision point: the restore fills in MISSING files, so a customer with a failing Tier-2 restored and
> silently got OLDER files. New `CrossDriveBackup.LastSuccess` + **`SuccessTracked`**; the marker is
> load-bearing because **all 7 fleet rows were pre-anchor at deploy** — without it every customer sees
> „Még nincs sikeres másolat" at once. Legacy rows migrate on first touch (`ok` adopts its time,
> `error` seeds nothing). **PART 2 — the three `record*` helpers rebuilt the WHOLE struct with only 2
> fields carried over; the naive fix would have had `recordTier2Failure` CLEAR the anchor.** Replaced
> by `tier2Update` (copy-and-overlay = safe by construction). New `fmtTimeStr` → Budapest-local dates
> in the dialog instead of raw UTC RFC3339. **F-DIAG:** 6 classes incl. an honest `unknown`, and the
> notification no longer passes `err.Error()` through raw — **LESSON: my first sanitiser was regex-only
> and leaked a bare hostname; its own test caught it. Redact KNOWN values, don't guess at shapes.**
> Live on demo-hp: rendered dialog read in the failed, healthy AND legacy states. F-OPS documented at
> `felhom.eu/documentation/runbooks/RUNBOOK-manual-guest-restore.md`.
> **2026-07-28 — v0.181.0 (R-100).** `OffboxTarget.LastSuccess` + wire field `last_success`; the hub
> (v0.80.0) anchors offsite staleness on it. **`LastRun` is written unconditionally on every run
> INCLUDING failures** — it records an ATTEMPT — so the hub's "how long since LastRun" verdict read a
> nightly-failing tier as perfectly fresh forever. The rule is the pure `offboxAnchorAfterRun(prev, at,
> runErr)`: a failure neither ADVANCES nor CLEARS the anchor (both are distinct bugs; clearing it would
> make one bad night look like never-succeeded). `LastStatus == "error" ⇒ stale` was rejected — it pages
> on every blip, the F-A1 noise mode. **TWO SILENT-WIPE SITES CLOSED** (`offboxConfigHandler` and
> `ApplyOffsiteTarget` both rebuild the target and copy runtime status field-by-field — omitting
> LastSuccess would erase the anchor on any settings save or hub re-apply). **LESSON: my first test
> modelled the rule in a local closure and stayed GREEN when production was mutated — hollow; the
> extraction to a pure function is what made the red-proof bite.** Live on demo-hp: failing run advanced
> `last_run` to 11:25:48Z while `last_success` HELD at 11:24:20Z; demo-felhom healthy → advanced. The
> settings-save preservation was proven live too. Detail: `REPORT.md` + `felhom.eu/REPORT-r100.md`.
> **2026-07-28 — v0.180.0 (F-OBS).** Source: `audits/CAMPAIGN-8-backup-restore-2026-07-27.md`.
> On a default `logging.level: info` box there was **no positive observable that `deadapp-check` had
> run**: its per-cycle line goes through `Scheduler.dbg()`, gated on `level==debug`, so on a default
> box it was never *produced* and could not even reach the always-DEBUG ring. "No alarms" was
> therefore indistinguishable from "the detector never ran" — standing rule 3's exact fallacy, and it
> undermines F-CRIT-1's fix, which is a fix to **this same detector**.
> `noteDeadAppScan()` now emits an INFO line every **20th** scan (10 min at the 30 s cadence) carrying
> scans-since-boot / evaluated / currently-down. It reports **what it saw**, not that it ran, and it
> summarises rather than floods — one line per run is 2880/day, which is what made silence attractive
> in the first place. Both bounds are pinned by test in the direction that would break them.
> **The same shape then turned up in the agent's brand-new guest-power watchdog** (v0.107.0, shipped
> hours earlier): it logged only at startup and when it acted. Fixed in agent v0.109.0 with the same
> pattern. The anti-pattern reproduces itself — which is the argument for not having dropped this part.
> Live on demo-hp at INFO on a default-level box; deployed on both boxes. Detail: `REPORT.md`.
> **2026-07-26 — v0.173.0 (R-77).** Source: `audits/DIAG-agent-channel-2026-07-26.md`.
>
> **UNRESOLVED AND DELIBERATELY DEFERRED — which file is authoritative for `local_api`?** R-77 ships
> DETECTION ONLY. `controller.yaml` and `bootstrap.json` can disagree; the controller dials
> `controller.yaml`. The obvious "fix" — reconcile from `bootstrap.json` on every boot — has a failure
> mode **as severe as the bug it fixes**: on a guest whose `controller.yaml` is correct and whose
> `bootstrap.json` is stale (a re-provision that half-completed, a hand-repaired guest, a
> setup-wizard box), auto-reconcile would clobber a WORKING channel on the next restart — fleet-wide,
> silently, at the moment of a routine deploy. R-77's position is that **naming the drift is enough**:
> it would have converted the 17.5 h outage into a specific alert on the first health cycle. The
> authority ruling is **R-78** and needs its own spike — do not resolve it opportunistically.
>
> Corollary for anyone editing `bootstrap.MaybeIngest`/`ensureLocalAPI`: `ensureLocalAPI` is the ONLY
> writer, it fires only when the endpoint is EMPTY, and `DetectEndpointDrift` must stay write-free.
> Scenario A's test asserts `controller.yaml` is byte-identical after the check, and its red-proof
> covers the auto-correcting variant precisely because that is the tempting wrong turn.
>
> **Also settled here:** the samba protected-set must mirror EVERY early return in
> `reconcileSambaAt` (currently two: `!smb.Enabled`, `!smb.UserSet`). A third would need the same
> mirror, and the doc comment above `EffectiveProtected` must be updated with it.
> **2026-07-26 — v0.172.0 (R-75).** Spike `felhom.eu/documentation/audits/SPIKE-catalog-data-paths-2026-07-26.md`;
> feature doc `felhom.eu/documentation/controller/import-and-data-paths.md`.
>
> **RULING — the import root is CANONICAL on the system drive, overriding the spike's Fork-1
> recommendation of per-drive roots.** The spike weighed sidebar clutter and per-app link ambiguity and
> concluded per-drive; the operator overruled it on an argument the spike missed: each drop-zone app has
> exactly ONE ingest bind, so on a two-drive box every import folder except the app's own would look like
> a drop-zone and silently do nothing — and because `import/*` is `class: excluded`, files stranded there
> are never backed up either. A canonical root is the only shape with no dead drop-zone. Recorded as a
> deliberate deviation, not an oversight.
>
> **Phase-0 probe changed the shape of Part 6.** The system drive is NOT a registered `StoragePath` on
> either demo box (`/mnt/felhom-drives/hdd_1` on demo-felhom; `nvme-1tb` + `Felhom-Share` on demo-hp),
> so `sharingResolvePath` REFUSES `<sysroot>/userdata/import` — verified against the real guard with a
> passing control. Registering the drive was rejected (it would make the 50 GB volume holding the
> recovery units a customer-visible drive, deploy target and wipe candidate, and `SharingDeniedRoots`
> would then deny the namespace-consistent shape anyway). **Chosen: leave it unregistered and have the
> controller write the `beolvasas` share directly** — the picker guard validates CUSTOMER-supplied paths,
> a controller-generated constant is a different trust class. No guard was weakened.
>
> Also note: `withUserdataPath` computes `USERDATA_PATH` as `<hdd>/userdata`, NOT
> `NamespaceRoot(hdd)/userdata`. For an app on the system drive those disagree
> (`/mnt/sys_drive/userdata` vs the `felhom-data` namespace). Latent — no app with a userdata bind has
> ever been deployed there — but it is a real inconsistency, left untouched here.
>
> The other three forks followed the spike unchanged: all-apps skeleton / deployed-only in the UI;
> unknown role fails OPEN while a malformed path whole-block rejects; drop-zone copy driven by the
> derived backup class.
> **2026-07-24 — v0.169.0 (disk-health card + degradation alert).** Consumes the agent's new `smart`
> field (agent v0.94.0; MinAgent floor unchanged — feature-detect by presence). **Rulings:** (1) ONE
> pure verdict fn `agentapi.DiskVerdictFor` is the shared truth for the card chip AND the 6h check — they
> can never disagree. Thresholds: FAILING→Hiba; PASSED + any(reallocated>0/pending>0/offline_unc>0/
> critical_warning>0/media_errors>0/percentage_used **≥90**)→Figyelmeztetés; PASSED clean→Rendben;
> nil/UNKNOWN→Nincs adat (never alarms). (2) **No global alert banner** — the card + email carry disk
> health; banner fatigue is a real cost, so this is deliberately NOT wired into the dead-app/alert-banner
> machinery. (3) Degradation-only notification with an in-memory baseline: first run baselines silently,
> recovery never notifies, **UNKNOWN excluded both directions** (a transient blip neither fires nor erases
> history). (4) **Controller restart re-baselines silently** (in-memory baseline lost on restart) — an
> accepted trade consistent with the health-change pattern (a real post-restart degradation still fires on
> the following 6h check once a baseline exists). (5) A **60s TTL cache** wraps the card's /disks call so
> dashboard refresh-spam can't smartctl-storm the host; the 6h check fetches FRESH (cache-independent).
> Pairs with hub +1 (allowlist `disk_health_degraded`). No new smartctl load — serialization only.
Last updated: 2026-07-24 (v0.168.0 — customer-configurable backup window "Mentési időablak")
> **2026-07-24 — v0.168.0 (customer-configurable backup window).** ONE customer setting — the window
> start W ("Mentési időablak kezdete") — drives every nightly leg at FIXED, never-stored offsets so
> misordering is impossible: DB dump at W, tier-2 at W+60m, off-box at W+105m (wrap-safe). **Design
> rulings:** offsets are DERIVED and computed everywhere, never persisted and never exposed in the UI;
> precedence is settings > controller.yaml `db_dump_schedule` > "02:30" (mirrors PasswordHash); a change
> applies WITHOUT restart via the new scheduler seam `UpdateDaily` (per-daily-job buffered `resched`
> chan + a select case in `runDailyJob`). New pure package `internal/backupwindow` holds all the time
> math (ParseHHMM/FmtHHMM/LegTimes/GateWindow/EffectiveWindow). **Disk-tier (whole-guest PBS/vzdump)
> gate:** the quiesce loop's SCHEDULED cycles run only inside [W+2h, W+6h) (wall-clock Europe/Budapest),
> with a safety valve — last successful backup older than cadence+24h (or none) runs regardless, so a
> box only ever on outside its window never starves. **Manual "Mentés most"/TriggerNow is NEVER gated**
> (bypasses runOnce). The `quiesce.Backend.Due` seam now also returns the backup age (from the agent's
> own `/backup/due`); the agent, its cadence, and `/backup/due` are untouched. Window read fresh each
> poll (WindowStartFn) so runtime changes take effect. Cadence defaults to 24h controller-side (the
> response carries no cadence). Backup page gets a "Mentési időablak" card (time input + derived rows +
> the "kb. W+2hW+6h között" rendszermentés line); POST /backups/window (RequireAuth+CsrfProtect).
> **2026-07-24 — v0.167.0 (outlined logo + favicon — Part 4 unblocked).** Viktor pushed the
> text-outlined `logo.svg` to felhom.eu `main` (`be9edb4`); the wordmark is now 17 real `<path>`
> glyphs. `FelhomLogoSVG` swapped to it; Inkscape's leftover **empty `<text/>` shells + font-* leftovers
> on the paths** were stripped via an lxml DOM pass (glyphs untouched — CC did NOT do text-to-path),
> editor `<sodipodi:namedview>` dropped. `FelhomFaviconSVG` vestigial `<text>` removed. Both constants:
> **0 `<text`, 0 `font-family`**; viewBoxes unchanged; palette + 14 gradients preserved. **Gotcha logged:
> Inkscape "Object→Path" leaves empty `<text/>` shells AND copies `style="…font-family:…"` onto the
> resulting `<path>`s — a search for `svg:text` misses them (elements are `<text>`, no prefix); grep
> `<text` and `font-family`.** Also: `serveLogoHandler`/`serveFaviconHandler` serve a **hub-synced file
> first** (`assetsSyncer.Resolve`) and fall back to the constant only if none is on disk — on 9201 the
> constant is what's live (verified). Still open (separate follow-up): website + hub serve their own
> non-outlined logo copies; login.html stylesheet link still unversioned.
> **2026-07-24 — v0.166.0 (mobile nav = off-canvas drawer; sidebar cleanup; ?v= on logo/favicon).**
> Mobile nav was broken: the ≤768px block predated the v0.146.0 accordion and flattened `.nav-links`
> into a horizontal `overflow-x` strip, clipping the accordion's nested sub-lists (they share the
> `.nav-links` class). **Decision: mobile nav = a sticky top bar + off-canvas left drawer that REUSES
> the vertical sidebar (Option A).** The accordion handler is untouched and works inside the drawer;
> a `no-js` html-class fallback renders the sidebar static inline so nothing dead-ends without JS.
> Options B (separate mobile menu) and C (exclude nested lists from the strip) were rejected. z-index
> ladder topbar 800 < backdrop 900 < drawer 950 < modal 1000; `100dvh`; reduced-motion disables the
> slide; focus-trap deliberately omitted (navigations reset state). **Sidebar customer-name removed**
> (logo only); `{{.CustomerName}}` stays in base data + login subtitle. **Logo policy decision: the
> wordmark must be OUTLINED paths, never live `<text>`** — under `<img>` secure static mode only
> locally-installed fonts resolve, so `font-family` in the SVG renders a fallback font everywhere.
> **Part 4 (swap `FelhomLogoSVG`/`FelhomFaviconSVG` to the outlined master) is GATED OUT** — §3a check
> against live felhom.eu `main` (`be9edb44`) found `website/assets/logo.svg` still has `<text>`/
> `font-family`; the outlined master is Viktor's manual Inkscape push, still pending. Only the `?v=`
> cache-bust (logo/favicon/login-logo, Cloudflare 4h edge-cache — the 0.126.1 failure mode) shipped
> from the logo work. Follow-up: when Viktor pushes the outlined asset, ship Part 4 (swap constants +
> clean the favicon's vestigial `<text>` nodes). Separately, the website + hub still serve their own
> non-outlined logo copies — propagation is a distinct follow-up.
> **2026-07-24 — v0.165.1 (native "Megosztás…" in the share modal, Web Share API).** The share modal
> gains a feature-detected `navigator.share` button (OS share sheet → Messenger/WhatsApp/email),
> sending **title + text + URL only**. Hidden unless supported; "Link másolása" stays the universal
> fallback (and catches the non-cancel rejection); `AbortError` (user cancel) is silent. **Ruling: the
> QR is NOT attached** (no Web Share Level-2 `files:`) — file-share support is narrow and several
> targets drop the URL when handed file+URL, leaving an unscannable QR picture in a chat; the QR's job
> (physical cross-device scanning) is already served by the modal image (mobile long-press). Template
> JS + tests only; the OS sheet interaction is an operator manual check (not endpoint-testable).
> **2026-07-24 — v0.165.0 (Indítópult megosztása — guest launcher via capability URL).** The admin
> launcher gets an "Indítópult megosztása" button that mints a **capability URL**
> (`https://<host>/s/<token>`, 160-bit `crypto/rand` token) serving a standalone, read-only guest
> launcher — same tiles, opens apps in new tabs — with **no account and no admin session**. **Security
> ruling: the link grants INFORMATION ONLY, ZERO CONTROL** — app names + public URLs; every privilege
> stays behind each app's own auth and the controller admin password. The token IS the secret (160-bit
> entropy is the whole defence for the GET — never rate-limited, never logged, `subtle.ConstantTimeCompare`
> only; an empty stored token = sharing OFF, matches nothing, so a wrong/disabled token is byte-identical
> to the mux default 404). Optional per-share password is a SEPARATE credential (own bcrypt hash, own
> attempt map — NEVER the admin ones); one pass mints a cookie = HMAC(`token|passwordHash`) keyed with
> the persisted `web.session_secret`, so rotate-token OR change-password invalidates all cookies for free.
> **Part-2 secret decision: REUSED `web.session_secret`** (persisted + box-scoped + stable — the SAME
> secret the claim pre-auth CSRF already trusts; not per-boot, not claim-generation-scoped → the reuse
> branch), so no `ShareCookieSecret` field was added. **Design rulings recorded:** member accounts are
> **superseded** by this capability-URL model; **per-member tile visibility is PARKED under the SSO arc.**
> Guest state labels ride the v0.164.0 invariants: `StateStopped` ⇒ "A tulajdonos leállította"; any
> other non-clickable state ⇒ "Átmenetileg nem elérhető" (guests never see stopped/exited/degraded/
> unhealthy). Accepted residuals (documented, no code action): link-preview crawlers fetch once and see
> app names (noindex prevents indexing); reverse-proxy/CF access logs may hold the path (ops-tier); the
> modal link carries the request Host, so a LAN-IP admin session yields a LAN-IP link. New dep:
> `github.com/skip2/go-qrcode`. Tests: Groups AG (14 tests) + 3 red-proofs verified red.
> **2026-07-24 — v0.164.0 (stopped ≠ fault).** Operator finding on 9201: a UI stop (Leállítás) raised
> the global "Telepített alkalmazás nem fut: … (stopped)" banner on every page AND fired the
> `app_start_failed` email. RULING: **a deliberate user action must not alarm anywhere.** One-line
> filter at the single fix-3 derivation point — `scanDeployedAppRunStates`'s pure core extracted to
> `classifyRunStates([]stacks.Stack)`, down predicate now
> `stacks.IsDownState(st.State) && st.State != stacks.StateStopped`. `StateStopped` is dropped from
> BOTH the banner dead-list and the notifier Down-set (⇒ no banner, no event, clean tracker). Rests on
> **two invariants that MUST both hold for this suppression to be correct:** **I1** — the UI stop path
> `Manager.StopStack` runs `docker compose down` → containers removed → a deployed stack with zero
> containers aggregates to `StateStopped` (refreshStatusLocked). **I2** — the P2 restart-policy census
> (2026-07-21, 53 templates / 78 services) found every catalog service on `unless-stopped`, so a crash
> never rests at `stopped` — faults surface as `exited`/`degraded`/`restarting`/`unhealthy`. **If
> either invariant changes, revisit this suppression.** `IsDownState` UNCHANGED (other callers rely on
> stopped=down). Out-of-band `docker compose stop` (containers remain → `StateExited`) still alerts —
> correct, tampering is reportable. The `stopped_by_user` intent flag was considered and PARKED (only
> adds value against out-of-band stops, which should keep alerting). Tests +4 (notify 3→4, main 4→7),
> both red-proofs verified. No template/funcmap/notifier/counter/copy change.
> **2026-07-24 — v0.163.1 (launcher polish).** Two v0.163.0 live findings fixed. RULE recorded:
> **every app-logo surface ends in a visible placeholder** (`SVG → PNG → /static/app-placeholder.svg`,
> infra rows → `infra-logo.svg`) — the four sibling `onerror` chains (`backups_apps`, `stacks`,
> `app_info` hero, `deploy`) now match `app_row.html`; `app_info` screenshots deliberately still
> vanish on error. And the **launcher monogram is launcher-only AND failure-only**: hidden by default,
> revealed when the tile's img chain fails (`onerror` adds `.launch-tile--noimg`) — it was bleeding
> through every transparent white glyph. Template/CSS only; no handler/funcmap change. 5 tests + 2
> red-proofs. [[launcher-v0163-2026-07-24]]
> **2026-07-24 — v0.163.0 (Indítópult app launcher + universal placeholder icon).** New
> customer-facing `/launcher` page: the FIRST sidebar item (above Vezérlőpult), a grid of large
> tappable tiles for openable deployed apps. `/` stays the Vezérlőpult — the launcher is ADDITIVE.
> Design rulings recorded here:
> - **(a) The felhom brand mark is NEVER an app placeholder** — brand = platform identity only. The
> logo-less fallback everywhere is the new generic `AppPlaceholderSVG` (a 2×2 app-grid glyph,
> `/static/app-placeholder.svg`), now the DEFAULT `FallbackIcon` on `app_list_row` (was
> `visibility:hidden`). On the launcher tile the fallback is the **monogram**, not the placeholder.
> - **(b) A launcher tile exists ⟺ a „Megnyitás" button would** — subdomain presence (env `SUBDOMAIN`
> > `.felhom.yml` subdomain > `protectedStackSubdomains`) is the single openability criterion. The
> controller stack is excluded by name. The subdomain assembly was extracted to
> `Server.subdomainMap` (3 callers: dashboard, Alkalmazások, launcher; priority byte-unchanged).
> - **(c) Colored-tile + mono-glyph design.** `tileColor` = validated `.felhom.yml` `brand_color`
> (`#rgb`/`#rrggbb`, new `Metadata.BrandColor`, omitempty) OR a deterministic FNV-1a-of-slug HSL
> (fixed S/L, hue per app). Invalid `brand_color` silently falls back to the hash color (the one
> §8 exception to no-silent-failure — cosmetic). `tileColor` returns `template.CSS` (we
> validate/compute in Go; html/template's CSS filter mangles a legit `hsl()` from a func pipeline).
> - **(d) `/` remains the Vezérlőpult.** No role/auth gating — member-role gating is a future arc
> (ROADMAP: member role → launcher becomes the member landing page). No catalog app sets
> `brand_color` yet (curation parked).
> No agent coupling; MinAgent unchanged. 10 new test functions + 4 red-proofs (all observed FAIL then
> restored). Gates green (app_row_dedup / template_id / emoji).
> **2026-07-24 — v0.162.0 (R-71a), SHIPPED + deployed BOTH boxes (demo-felhom 9201 + demo-hp 9201 > **2026-07-24 — v0.162.0 (R-71a), SHIPPED + deployed BOTH boxes (demo-felhom 9201 + demo-hp 9201
> via G1 break-glass), clean+healthy, settle-gate GO line captured on both.** B live note: both > via G1 break-glass), clean+healthy, settle-gate GO line captured on both.** B live note: both
@@ -1315,6 +1950,169 @@ Last updated: 2026-06-13 (v0.60.0 backlog-Medium cleanup)
--- ---
## THE TWO RULES THE RECOVERY JOURNEY LEANS ON (v0.203.0, 2026-08-06)
> **1. A credential the hub stages is collected by the box, not waited for.** The reconcile that
> collects runs on a tick for exactly as long as the box's own declaration says it needs one — and
> stops the instant a target exists. It is driven from `OffboxReportStatus().State`, the same statement
> the hub acts on, so the two can never disagree about whether a retry is wanted.
>
> **2. A mount Felhom itself made is not "something else".** Enrolment mounts a drive twice — the
> managed path and a raw `/mnt/<name>` on the host — and the host survives a guest rebuild while the
> guest's registry does not. The claimed check forgives a non-managed mount **only when corroborated**
> by the same device also being mounted under the managed path. **A genuinely foreign mount is still
> refused, and that fence has its own test.**
**Why both are stated here rather than left in the code:** each was a dead end that kept the unaided
recovery journey failing, and each looked correct in isolation. R-218's declaration half shipped and
worked while nothing consumed what it asked for; R-220's check was right about foreign disks and wrong
about our own. **Neither is a bug in the thing it guards — both are about what runs, and when.**
Two things that must not be "simplified" back:
- **The settle gate stays.** The retry goes through `ReconcileWhenSettled`, so the day-0 floor race is
unchanged. A retry that skipped it would trade one defect for another.
- **The R-220 exemption is corroborated, never a prefix.** Widening it to any `/mnt/*` path offers a
disk another system is using for formatting — the red-proof shows exactly that.
## THE UNLOCK PATH'S RULE (v0.202.0, 2026-08-06) — state it before changing anything there
> **On the recovery unlock path the customer is blamed only after a real attempt REFUSED their code.
> Every other outcome — including one that cannot be classified — says something else.**
This is the rule, and it outlives the bug that produced it. It was learned twice, because fixing it
once was not enough:
- **v0.201.0** stopped an agent that is too OLD from being reported as a wrong code (R-216).
- **v0.202.0** found the same defect through a different door: an agent that is **stopped**, and a hub
that cannot be **reached**, still fell through to a message about the code. Measured with a
**correct** code at 0.0299 s and 0.0556 s, against ~1.0 s for a real unseal — the machine accused the
customer of something it had not tried (R-224).
- And the inverse: the one message that says *"check your ten words"* was unreachable on any box that
had re-escrowed, which is exactly the box a customer has just recovered (R-226).
**How it is enforced.** `agentapi.ClassifyRecoveryFailure` maps the failure to one of five classes
**from the value, never the text**; the typing message is reachable from **one** of them
(`RecoveryAskedAndRefused`, i.e. HTTP 400, i.e. the bundle was fetched and `age` refused it); and the
zero value is `RecoveryUnknown`, which renders **neutral**. **The safe default is the load-bearing
part** — an unrecognised status must not fall into an accusation.
**Two things that are deliberately NOT how it works, and must not be "fixed" into it:**
1. **Elapsed time is never a classifier.** It is what diagnosed this, it is logged for the operator,
and that is all. A duration guard would be a second thing that can be wrong.
2. **The error's TEXT is never read.** A string match is a defect waiting for a rewording. When the
distinction was not available as a value, the **agent was changed to provide one**
(`escrow.ErrBundleFetch` → HTTP 502, agent v0.126.0, `MinAgent 0.126.0`) rather than parsed for.
**The coupling degrades safely and silently:** an agent below 0.126.0 answers 400 for both causes, so
`FeatureRecoveryFailureClass` withholds the refusal reading and the 400 becomes neutral. The gate
blocks nothing; it only decides whether the customer may be told to check their typing.
## CAMPAIGN 11 — what changed in v0.201.0 (2026-08-05)
**The off-site key recovery is a COUPLED feature and now declares it.** It needs agent **0.125.0**
(`POST /escrow/recover-offsite-password`). `FeatureOffsiteKeyRecovery` has a `featureProbes` row, a
`featureMinAgent` row and a `Supports` gate at the unlock entry point.
**That gate FAILS CLOSED — alone in that table.** The package default is fail-open, and that default
is what produced R-216: an agent that could not answer 404'd, the unlock was attempted anyway, and the
customer was told their correct recovery code was wrong. Anything but `SupportYes` now says *the
machine* cannot ask yet, and no attempt is made. Do not "fix" it back to the package default.
**The box declares `needs_credential` until the TIER WORKS, not until a key exists.** The old
short-circuit on "a repository password is present" is deleted: installing one is the recovery
screen's whole job, so it made succeeding at recovery switch off the mechanism that delivers the
coordinates to use it. A disabled target still short-circuits at the first line (Scenario E).
**The unlock finishes the job**: place the key → bring the tier up (`offsiteapply.Bridge.Reconcile`,
wired via `SetRecoveryTierUp`) → list. Without the middle step the promised listing can never render on
shape (a), because no key ⇒ no target ⇒ no inventory.
**Four messages, not one.** Wrong code (the only one mentioning typing) · the machine cannot ask ·
the store could not be read / the connection details have not arrived · the code belongs to a RETAINED
earlier package. The last one is driven by the ACK's `superseded_present`/`superseded_at` (hub
v0.97.0) and **promises nothing** — no read path for a superseded package exists.
**Still open from the campaign:** R-214 (console pairing banner), R-220 (drives unenrollable after a
rebuild), R-221 (a rebuilt box cannot run the escrow ceremony), R-223 (the Day-0 manifest still vouches
agent 0.120.0 — operator decision).
## R-203 (v0.197.0) — the namespace-root contract, and what `ok` now means
**The contract, in one line:** `appbackup`'s path helpers (`UserdataDir`, `PrimaryBackupPath`,
`RecoveryUnitPath`, `AppDataDir`) take a **NAMESPACE ROOT**. Anything that came out of `HDD_PATH` or a
`StoragePath` is a **DRIVE path** — put it through `appbackup.NamespaceRootFor(drive, systemDataPath)`
first. `UserdataDir(bareDrivePath)` still compiles and is still wrong; five callers proved it.
**The rule now has ONE expression.** `NamespaceRootFor` / `IsEnrolledDrive` in `appbackup`;
`backup.Manager.namespaceRoot` and `stacks.Manager.inGuest` delegate. There were two copies before and
**they differed** — one compared without `filepath.Clean`, the other with it.
**Why it was invisible:** on an enrolled drive the namespace root IS the drive path. The two diverge
only on the system-data fallback, which `paths.go:26` names as a supported arrangement.
**`last_status` gains `incomplete`.** A run that could not capture a directory an app declares
MANDATORY is not a successful run. **Not `error`** — the rest of the run worked, so `SnapshotCount`
and `LastSuccess` still record what WAS captured. It reaches the operator via the existing
`backup_run_failures` digest (a new event type is a two-repo change; the hub drops unlisted types).
The Hungarian customer warning is unchanged; the page renders `! Hiányos`.
**Still open, and NOT fixed here:** `resolveAbs` resolves `RootHDD` and `RootUserdata` against the same
root. Both callers now pass the namespace root so the export and the backup agree with each other, but
whether `${HDD_PATH}` should mean the namespace root on the system drive touches every deployed app's
binds and needs a decision, not a patch.
**Blast radius, measured before changing anything:** exactly one app in the fleet had
`HDD_PATH == system_data_path` (`calibre-web` on demo-hp, the R-201 drill fixture). Its data was
migrated and its sentinel re-verified byte-identical.
## R-203 (2026-08-04) — a MANDATORY userdata directory can be absent from the off-site snapshot while the run says `ok`
Found live on demo-hp while staging the R-201 drill, and it **halted that drill**.
`NamespaceRoot(drivePath, inGuestDrive)` (`appbackup/paths.go:28-33`) appends the `felhom-data` segment
**when the drive IS the system data path**`m.namespaceRoot` = `NamespaceRoot(drivePath, drivePath !=
m.systemDataPath)` (`backup/backup.go:331`). The deploy-time bind does not: `${USERDATA_PATH}` is
`<HDD_PATH>/userdata` (`stacks/classify_binds.go:14`).
With `system_data_path: /mnt/sys_drive` and `calibre-web` deployed at `HDD_PATH=/mnt/sys_drive`:
live bind (files land here): /mnt/sys_drive/userdata/media/books ← exists
capture set looked for: /mnt/sys_drive/felhom-data/userdata/media/books ← does not
**The same compose used BOTH roots**`${IMPORT_PATH}` resolved *with* the segment,
`${USERDATA_PATH}` *without*. The run logged one `[WARN] mandatory data path missing on disk, skipped
from offsite`, then `0 mandatory path(s)` and **`backup OK: 3 app(s), 3 snapshot(s)`**, with
`last_status: ok`. Nothing customer-visible or hub-visible said the directory was dropped.
**Not established:** whether `HDD_PATH == system_data_path` is a supported deploy. It was accepted
(HTTP 202) one call after the NAS path was correctly refused (R-108). **Either branch is a defect**
broken resolution, or a missing refusal.
**Two things a fix must do:** make the two roots one function, and make a skipped **MANDATORY** path a
customer/hub-visible signal rather than a container-log WARN. `opengist`/`privatebin` declare no
mandatory userdata paths and are unaffected.
## R-200 (v0.195.0) — the offsite key recovery diagnostic
`--recover-offsite-check` is a `docker exec` escape hatch (the `--print-reset-code` shape), NOT a page
or an API a browser can reach. R comes from **STDIN** — never argv, never `ps`, never shell history,
never a transcript. It asks the agent (>= v0.125.0) to fetch this host's sealed bundle and open it,
then reports whether the recovered repository password matches the on-disk one **by sha256**.
docker exec -i felhom-controller /usr/local/bin/felhom-controller --recover-offsite-check < /path/to/code
**IT COMPARES AND NEVER INSTALLS.** `CheckOffsiteKeyRecoverable` must stay free of any write — if a
future change makes it place the recovered password, it stops being a diagnostic and needs the drill's
supervision (that is link 9, R-200's remaining half). Pinned by
`TestCheckOffsiteKeyRecoverable_WritesNothing`, whose red-proof is adding the install call.
Exit codes are load-bearing: **0** match, **2** a clean MISMATCH, **1** a step failed. A mismatch is a
finding about the system; a failure is a finding about the run, and they must never share a status.
**Proven live on demo-felhom 2026-08-04** — recovered sha256 == on-disk sha256 == the hub's stored
hash. Nothing customer-facing ships with it: no card, no form, no preview.
## About Viktor (project owner) ## About Viktor (project owner)
- Works at Deutsche Telekom (Budapest), building Felhom.eu as a side business - Works at Deutsche Telekom (Budapest), building Felhom.eu as a side business
+423 -74
View File
@@ -1,92 +1,441 @@
# REPORT — felhom-controller v0.162.0 (R-71a: the apply-bridge settle-gate) # REPORT — controller v0.215.0 → v0.216.0: disk-health severity ladder, escalation, and the alert that never sent
**Date:** 2026-07-24 · **Task:** R-71(a) — structural fix for the F10 day-0 update-vs-bridge race **Date:** 2026-08-14 · **Task class:** Implementation · **Repos touched:** `felhom-controller` (code),
(consume-then-persist not crash-safe). **Scope:** `felhom-controller` only, ordering-only. No agent `felhom.eu` (documentation only — no hub code, no manifest bump, no ArgoCD sync)
(0.93.0 stays), no hub. MinAgent unchanged.
## The defect (DIAG-f10) ---
A fresh box boots below the operator floor (ISO 0.153.0 < floor 0.156.0). The apply-bridge consumes ## 1. Confirmed baselines used (as read at the start of the run)
the hub's single-use offsite password at boot; ~35 s later the managed **auto-floor update** replaces
the container mid-install → the new 0.156.0 process finds no installed key → `consume`**404**
offsite tier dead until an operator Re-issue. Recurs on **every** onboarding whose ISO floor lags the
managed floor; demo-felhom escaped by timing alone. Layering: the v1.25.0 golden≥floor build gate
PREVENTS the trigger for fresh installs, R-71c (hub) HEALS a burn, **this (a) removes the systematic
trigger for every restart shape.**
## What shipped (ordering only — consume/install/persist internals + the 404-no-oracle contract + the Consumer UNTOUCHED; R-71(b) stays rejected-by-design) | Repo | `main` @ start | Version | → Shipped |
|------|----------------|---------|-----------|
| felhom-controller | `3e3ee94b7bbe6b66663c468e22aa86616365a45a` | v0.214.0 | **v0.215.0**, then **v0.216.0** (a defect found live in v0.215.0 — §14) |
| felhom.eu | `e0b56c976f8e4a7352309d78754fec448dd55f99` | n/a (docs only) | n/a |
1. **Seam** `offsiteapply.SettleProvider.SettleState() (version, floor string, updateRunning, Both trees verified clean (`git status --porcelain` empty, `HEAD == origin/main`) before any build.
floorKnown bool)` + `SettleFunc` adapter (seams.go) — a thin adapter over the self-updater's own The controller hash matched the spec's stated baseline exactly. `MinAgent` stays **0.129.0** — no
knowledge (`GetFloor()`/`IsUpdateRunning()`) in main.go. No second floor-fetch path. agent change; every field read here has been on the wire since agent v0.94.0/v0.95.0.
2. **`Bridge.AwaitSettle`** — polls every 10 s BEFORE the 3-minute Reconcile context is created (the
deferral never eats the reconcile budget). Release table:
- `updateRunning` → wait (the swap's restart supersedes us).
- `floorKnown && version < floor` → wait (auto-floor update imminent — do NOT burn the password);
log `deferring offsite apply: managed update to floor <F> pending (we are <V>)`.
- `floorKnown && at/above floor` → **GO on the first poll, zero sleep** (the B invariant).
- `!floorKnown` past the 90 s sub-bound → GO + WARN (a hub that can't serve a floor can't serve a
consume → no burn risk).
- overall 5-min bound → GO + WARN (`R-71c self-heal is the belt`).
`ReconcileWhenSettled` runs the gate, then Reconcile under a fresh 3-min context.
3. **main.go** — the bridge goroutine MOVED to after the self-updater is constructed (so the adapter
can read it). Wired ONLY when an updater exists; otherwise `Settle` is nil = reconcile immediately
(no updater → no floor-update to race).
4. **Bounds** are named constants with rationale comments. The floor is in-memory
(report-ACK-derived), **NOT persisted** — so on any restart it is unknown until the first report
ACK. Source-confirmed latency: the startup report fires ~5 s after boot and `SetFloor` runs
synchronously in its ACK handler → floor normally known in ~510 s (≤~45 s across 3×15 s report
retries); the 90 s sub-bound is headroom over that worst case.
## Tests — all green (`go test ./...` clean, `go vet ./...` clean, gofmt clean) ---
Injectable clock (no real sleeps), fake `SettleState`, recorded `Consumer` ## 2. Files created / modified
(`internal/offsiteapply/settlegate_test.go`):
- **A** below-floor defers (0 consumes while below) then GOes at floor → exactly one consume. **felhom-controller**
- **B** update-running defers then GOes when it finishes. - `controller/internal/agentapi/diskverdict.go` — modified (14-row ladder, `DiskPrior`, `UncorrectableSectors`, `TemperatureFailC`)
- **C** floor-unknown GOes at the 90 s sub-bound + the WARN line. - `controller/internal/agentapi/diskverdict_test.go` — modified
- **D** perpetually-below GOes at the 5-min overall bound + the WARN line. - `controller/internal/agentapi/diskverdict_ladder_test.go`**created**
- **E** (B) at/above-floor GOes on **poll 1** with **zero** sleep consumed. - `controller/internal/notify/notifier.go` — modified (severity, `DiskAlert`, `DiskAlertKind`, `Severity()`, 5 message shapes)
- nil-provider → immediate reconcile; cancelled gate → skips reconcile (no consume). - `controller/internal/notify/disk_health_test.go` — rewritten
- `controller/internal/web/disk_health_state.go`**created** (persistence + decision)
- `controller/internal/web/disk_health.go` — modified
- `controller/internal/web/disk_health_test.go` — rewritten
- `controller/internal/web/server.go` — modified (seam signature)
- `controller/cmd/controller/main.go` — modified (6h → 1h)
- *(v0.216.0)* `controller/internal/web/disk_health.go` + `disk_health_test.go` — the R-335 dedup fix and its test
- `CHANGELOG.md`, `CONTEXT.md`, `REUSE.md`, `controller/README.md`, `REPORT.md`
## Red-proofs — four, each OBSERVED failing then restored **felhom.eu** (documentation only)
- `documentation/audits/DIAG-smart-passed-trap-2026-08-14.md`**created**
- `documentation/audits/fixtures/smart-ST3000VX010-failing-2026-08-14.json`**created** (raw `smartctl -a -j`, verbatim)
- `documentation/audits/fixtures/smartd-history-sdg-2026-08-14.txt`**created** (406 `smartd` journal lines)
- `documentation/architecture/00-capability-map.md`, `documentation/backlog/ROADMAP.md`, `documentation/backlog/OPEN-ITEMS.md` — modified
| # | Break | Observed FAIL | ---
|---|-------|---------------|
| A | gate removed (early `return` in AwaitSettle) | `TestSettle_BelowFloorDefersThenGoes`: "expected the gate to defer while below floor (>=2 sleeps), got 0" — below-floor consumes immediately (the F10 burn) |
| B | `updateRunning` branch removed | `TestSettle_UpdateRunningDefersThenGoes`: mid-swap box GOes immediately (log shows GO, deferral line absent) |
| C | floor sub-bound branch removed | `TestSettle_FloorUnknownGoesAfterSubBound` + red-proof: "release near the sub-bound 1m30s, waited **5m0s**" — floor-unknown drags to the overall bound |
| D | overall bound check removed | `TestSettle_PerpetuallyBelowFloorGoesAtOverallBound`: **20 s test timeout** — perpetual below-floor loops forever in `AwaitSettle` |
## Ship / honesty ## 3. Commits pushed to `main`
The deferral paths ship **unit-proven + red-proofed, NOT live-fired** — their precondition is now | Repo | Hash | What |
structurally prevented by the v1.25.0 golden≥floor build gate, which is the point. **Layering: gate |------|------|------|
prevents, (a) defers, (c) heals.** R-71 → SHIPPED with (a)+(c); (b) recorded rejected-by-design. | felhom.eu | `848de81` | Part 0 — fixtures + findings doc |
| felhom-controller | `bb50e12` | Parts 13 — ladder, severity, persisted state + tests |
| felhom-controller | `c24f192` | Group L strengthened to two post-restart checks |
| felhom-controller | `34d83f5` | Part 4 — cadence 6h → 1h (measured) |
| felhom-controller | `8144a70` | Part 5 — CHANGELOG / CONTEXT / README / REUSE |
| felhom.eu | `767960b` | Part 5 — capability map, ROADMAP, register rows R-328…R-334 |
| felhom-controller | `90f2545` | **v0.216.0** — R-335, one physical disk evaluated once per run |
| felhom.eu | `fa4748d` | R-335 register row |
## Build + deploy + LIVE (2026-07-24) ---
Image `gitea.dooplex.hu/admin/felhom-controller:0.162.0` (sha `e6e1146f…`, 145M) built + pushed. ## 4. Per-test results — all twelve groups
Deployed to **both** boxes, clean start, healthy:
| Group | Scenario | Test | Result |
|-------|----------|------|--------|
| A | real drive, 2nd observation | `TestDiskCheck_RealDrive_HibaAndOneCriticalEvent` + `TestLadder_RealDrive_ReachesHiba` | **PASS** |
| B | the transient that cleared | `TestDiskCheck_FirstSightingIsWarnOnly` | **PASS** |
| C | sustained → Hiba | `TestDiskLadder_SustainDrivesTheEscalation` + `TestLadder_SustainIsWhatFires` | **PASS** |
| D | recovered, silent | `TestDiskCheck_RecoveryIsSilentAndClearsState` | **PASS** |
| E | flap damping | `TestDiskCheck_FlapDamping` | **PASS** |
| F | escalation beats damping | `TestDiskCheck_EscalationBeatsDamping` | **PASS** |
| G | still getting worse | `TestDiskCheck_RealertWhenStillWorsening` | **PASS** |
| H | below both bars | `TestDiskCheck_NoRealertBelowBothBars` | **PASS** |
| I | heat | `TestLadder_Temperature` + `TestDiskCheck_TemperatureShape` | **PASS** |
| J | no data never alarms | `TestDiskCheck_UnknownNeverAlarmsNorErasesPrior` + `TestLadder_UnknownNeverAlarms` | **PASS** |
| K | severity routes | `TestNotifyDiskHealthDegraded_SeverityRoutes` | **PASS** |
| L | state survives restart (seam) | `TestDiskCheck_StateSurvivesRestart_ProductionPath` | **PASS** |
Supporting: `TestLadder_CountBackstopBoundary`, `TestLadder_ZeroPriorIsFailSafe`,
`TestUncorrectableSectors`, `TestDegradedAttributes_NamesFailCounters`,
`TestNotifyDiskHealthDegraded_{WarnShape,FailShapes,CopyDiscipline}`,
`TestDiskAlertDecision_Table`, `TestDiskState_CorruptFileFallsBackToNoPrior`,
`TestDiskCheck_DisappearedDiskIsForgotten`, `TestDiskCheck_UnreachableAgentIsInert` — all PASS.
---
## 5. Red-proof outcomes — all twelve, individually
Each mutation was applied by script, **asserted present in the source before the run** (the harness
aborts with `MUTATION-NOT-APPLIED` if the target text is absent), the named test run, and the file
reverted with `git checkout --`. The tree was confirmed clean after the sweep.
| # | Mutation applied | Target test | Outcome |
|---|------------------|-------------|---------|
| A | remove truth-table row 6 (the sustain rule) | `TestDiskCheck_RealDrive_HibaAndOneCriticalEvent` | **RED-PROOF PASSED — FINDING, see below** |
| B | make row 9 return `Fail` | `TestDiskCheck_FirstSightingIsWarnOnly` | failed as required |
| C | pass a zero `DiskPrior` in `RunDiskHealthCheck` | `TestDiskLadder_SustainDrivesTheEscalation` | failed as required |
| D | let Rendben fall through the silence guard | `TestDiskCheck_RecoveryIsSilentAndClearsState` | failed as required |
| E | compare against last **observed** verdict, not last **alerted** | `TestDiskCheck_FlapDamping` | failed as required |
| F | let damping cover escalations | `TestDiskCheck_EscalationBeatsDamping` | failed as required |
| G | remove the re-alert branch | `TestDiskCheck_RealertWhenStillWorsening` | failed as required |
| H | make cooldown/doubling an **OR** instead of an AND | `TestDiskCheck_NoRealertBelowBothBars` | failed as required |
| I | remove truth-table rows 3 **and** 13 | `TestLadder_Temperature`, `TestDiskCheck_TemperatureShape` | failed as required |
| J | let UNKNOWN delete the prior record | `TestDiskCheck_UnknownNeverAlarmsNorErasesPrior` | failed as required |
| K | restore `severity := "warn"` | `TestNotifyDiskHealthDegraded_SeverityRoutes` | failed as required |
| L | skip loading the persisted state | `TestDiskCheck_StateSurvivesRestart_ProductionPath` | failed as required |
### A thirteenth red-proof, added after the deploy (R-335)
| # | Mutation applied | Target test | Outcome |
|---|------------------|-------------|---------|
| M | delete the `if seen[key] { continue }` dedup guard | `TestDiskCheck_SameDiskTwiceIsEvaluatedOnce` | failed as required |
Observed failure: `first sighting of an aliased disk must be silent, got 1: [{Label:felhom-backup … Kind:2 Sectors:8}]`
— i.e. `Kind:2` is `DiskAlertFailSectors`, a **Hiba on a first sighting of 8 sectors**. Reverted.
### FINDING — red-proof A passed, and it is the spec's mutation that is at fault, not the code
The task specified group A's red-proof as *"remove truth-table row 6 → verdict is Warn"*. **That
mutation cannot fail a test built on the real drive's values**: the real drive carries **352**
unreadable sectors, so with row 6 deleted it still reaches Hiba via **row 8** (count ≥ 64). The test
correctly stayed green, so the mutation proves nothing about row 6.
This was anticipated while writing the tests and is documented in the test's own comment rather than
discovered afterwards. **Row 6 is genuinely pinned**, by two tests that hold the counters at **8**
(far below the 64 backstop) and vary *only* the prior:
- `TestLadder_SustainIsWhatFires` (agentapi) — same `SmartSummary`, `DiskPrior{}` → Warn,
`DiskPrior{SawUncorrectable:true}` → Fail.
- `TestDiskLadder_SustainDrivesTheEscalation` (web) — the event-level twin.
Both were run under the row-6-deleted mutation and **both failed**, as recorded:
`SAME 8 sectors, now sustained = 2 (Figyelmeztetés), want Fail/Hiba` and
`severity = "warning", want critical` / `chip = "Figyelmeztetés", want Hiba`. So the invariant is
covered; only the spec's chosen mutation was invalid.
### A second finding, from building red-proof L
The first version of the Group L seam test ran **one** check after the restart and **passed under the
mutation** — because a controller that has forgotten its state is also silent on its first check. The
test was strengthened to run **two** checks (commit `c24f192`), after which the mutation fails. This
is the exact shape §10 warns about, caught by running the red-proof rather than assuming it.
---
## 6. Test count and suite state
- **Before:** 1391 test functions (at `3e3ee94`) · **After:** 1414 (+23)
- `go build ./... && go vet ./... && go test ./...`**all green**, no failures, no skips introduced.
- `python3 controller/scripts/controller_gates.py`**all 11 gates OK.**
---
## 7. Cadence measurement (Part 4)
Measured on **demo-hp** (Tier 0, disposable), through `fetchDisks`' real path — the agent local API
`GET /disks`, not the 60 s card cache. Ten consecutive calls, all **HTTP 200**:
- **demo-felhom (guest 9201, `ssh felhom-pve`)** — `Up (healthy)`:
``` ```
05:50:44 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply 0.840558 0.817000 0.815783 0.831992 0.809066
05:50:54 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.156.0 (we are 0.162.0), no managed update running 0.832899 0.824788 0.804886 0.812539 0.833547 (seconds)
```
- **demo-hp (guest 9201, G1 break-glass; secret extracted file→file + shredded)** — `Up (healthy)`:
```
05:52:53 [INFO] [offsite-apply] settle-gate: awaiting floor knowledge (first report ACK) before offsite apply
05:53:03 [INFO] [offsite-apply] settle-gate: GO — at/above floor 0.156.0 (we are 0.162.0), no managed update running
``` ```
**Honest accounting of the B live leg:** both above-floor boxes GOed correctly, but NOT literally on | min | median | max | disk count |
the first poll — because the floor is **in-memory, not persisted**, it is unknown at the very first |-----|--------|-----|------------|
poll (the report ACK hasn't landed) so the gate logged `awaiting floor knowledge`, then GOed on the | **0.804886 s** | **0.820894 s** | **0.840558 s** | **3 physical rows** across 2 devices (SanDisk X600 M.2 SATA SSD; Toshiba KXG50PNV1T02 NVMe, counted twice as `c11-scratch` + `felhom-backup`) |
next poll (~10 s later) the instant the floor became known and confirmed at/above. This is exactly
the documented finding, not a regression: the gate correctly refused to consume while the floor was **Branch taken: median < 5 s → `6*time.Hour` → `1*time.Hour`.** The median is ~6× under the bar. The
unknown, and added ~10 s to a background reconcile (invisible to any user). The B zero-wait-when- detection argument is the real one: the observed benign excursion lasted about **one hour**, so a
floor-known invariant is unit-proven (test E); the ~10 s live wait is the report-ACK latency the 90 s 6-hourly sampler can land either side of it and then catch the terminal run half a day late.
sub-bound was sized against. The important live proof holds: **the gate did not burn the one-time
password before the managed-update picture was clear.** No spin-up signature appeared in the timings (uniform ~0.82 s; demo-hp is all-flash), so the
measurement did not suggest the spun-down-drive concern. That question is recorded as an Observation
below and deliberately **not acted on**.
---
## 8. Live validation
Deployed to **demo-hp guest 9201** via the bootstrap path (`docker pull`
`/etc/felhom-controller-image``systemctl restart felhom-controller-bootstrap.service`).
```
gitea.dooplex.hu/admin/felhom-controller:0.215.0 Up 19 seconds (healthy) # 06:23Z
gitea.dooplex.hu/admin/felhom-controller:0.216.0 Up 6 seconds (healthy) # after the R-335 fix
```
### Leg 1 — no over-correction (the load-bearing check)
Method: **endpoint-level** — authenticated `GET /dashboard` on the real controller
(`https://felhom.enkisfelhom.hu/dashboard`, HTTP 200, 44 036 bytes), i.e. the exact endpoint the UI
invokes; only rendering is skipped. No browser is available on DooPlex.
Card contents, parsed from the response body:
| Disk | Chip | Class | Temp |
|------|------|-------|------|
| KXG50PNV1T02 NVMe TOSHIBA 1024GB | **Rendben** | `state-text-run` | 53 °C |
| KXG50PNV1T02 NVMe TOSHIBA 1024GB | **Rendben** | `state-text-run` | 53 °C |
| SanDisk X600 M.2 2280 SATA 128GB | **Rendben** | `state-text-run` | 44 °C |
`Figyelmeztetés` = 0, `Hiba` = 0, `Nincs adat` = 0, `state-text-warn` = 0, `state-text-crit` = 0.
**No healthy disk was over-corrected.**
**Positive observable, at deploy:**
`[INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s)` — the new cadence is
in force, not merely compiled.
**Positive observable, per cycle** — two full hourly cycles observed after the deploy, from the
container log:
```
2026/08/14 06:23:13 [INFO] [scheduler] Registered periodic job: disk-health-check (every 1h0m0s)
2026/08/14 07:23:13 [INFO] [scheduler] Running job: disk-health-check
2026/08/14 07:23:14 [INFO] [web] disk-health check complete: 3 disk(s) evaluated, 0 alert(s)
2026/08/14 07:23:14 [INFO] [scheduler] Job disk-health-check completed (took 849ms)
2026/08/14 08:23:13 [INFO] [scheduler] Running job: disk-health-check
2026/08/14 08:23:14 [INFO] [web] disk-health check complete: 3 disk(s) evaluated, 0 alert(s)
2026/08/14 08:23:14 [INFO] [scheduler] Job disk-health-check completed (took 843ms)
```
`grep -c disk_health_degraded` over the whole container log: **0**. Both cycles ran (849 ms / 843 ms,
matching the §7 measurement), evaluated every disk, and emitted nothing. **Zero alerts from a check
that demonstrably ran** — not silence.
Persisted state written by the first cycle (`/opt/docker/felhom-controller/data/disk-health-state.json`,
428 bytes, on the `felhom-controller-data` docker volume, so it survives container recreation):
```json
{"version": 1, "disks": {
"path:/var/lib/vz": {"verdict": 1, "saw_uncorrectable": false, ...},
"uuid:91d2dc2d-2d28-4929-9bdd-3e11fa2f41ae": {"verdict": 1, "saw_uncorrectable": false, ...}}}
```
`verdict: 1` is `DiskVerdictOK` for both, `saw_uncorrectable: false`, never alerted.
**Reading those two artefacts against each other is what exposed R-335** — see §14.
### Leg 2 — the severity fix arrives (the point of the task)
Two synthetic `disk_health_degraded` events pushed for customer `demo-hp` **through the real hub
event endpoint** (`POST https://hub.felhom.eu/api/v1/event`), from the guest's own controller using
its own hub credentials — the genuine controller→hub path, not a hand-crafted operator call. Both
returned `HTTP 200 {"ok":true}`. The hub DB was read with its `-wal` and `-shm` copied alongside
`hub.db` (a `hub.db`-only read is stale).
**As STORED by the hub (`events`):**
| id | severity pushed | severity STORED |
|----|-----------------|-----------------|
| 2964 | `warning` | **`warning`** |
| 2965 | `warn` | **`info`** ← coerced |
**`notification_log` rows for those two events:**
| id | event_type | severity | channel | status | error |
|----|-----------|----------|---------|--------|-------|
| 689 | `disk_health_degraded` | `warning` | `operator` | **`sent`** | *(none)* |
| — | *(the `"warn"` push)* | — | — | **NO ROW EXISTS** | — |
**That pair is the proof.** The identical event, differing only in one word of the severity string,
is the difference between *delivered to the operator* and *stored as an informational notice and
delivered to nobody*. This is the first time this leg has been observed end to end.
Only the operator leg fired because **demo-hp has no `customer_notifications` row at all** (no
customer email, no `enabled_events`), so no customer row was possible for either push — verified
directly, not assumed. **One real email was sent to the operator**, as the task anticipated.
---
## 9. NOT yet live-validated — stated explicitly
**The Fail-from-counters path has never fired on real hardware.** Everything in §4/§5 exercises it
against the committed fixture's values in unit tests only. The live legs above prove the *negative*
(no false alert on three healthy disks) and the *severity wire* (end to end, through the hub) — they
do **not** prove a live disk reaching Hiba. The fixture tests must not be read as a live proof.
Tracked as **R-332 (WATCHING)**. Closing condition: a live disk reaching Hiba from counters, or a
deliberate injection through the real pipeline (agent `/disks` → controller check → hub event) — not
a hand-set verdict.
**One item originally listed here has since been proven live** and is no longer part of this gap: the
**persisted state surviving a controller restart**. The v0.215.0 → v0.216.0 redeploy destroyed and
rebuilt the container, and the new one read back a `changed_at` written by the previous version rather
than re-baselining — see §14. What remains unproven is the stronger half: an already-**alerted** disk
not re-alerting after a restart, which needs a disk that has actually alerted. The drive that produced the fixture lives in DooPlex, which is Tier 2 and never a
drill target; the demo boxes are all-flash and healthy.
---
## 10. Teardown
**This run provisioned nothing** — no VM, no guest, no hub customer record, no storage. Nothing was
formatted, mounted, unmounted, repaired or written on any monitored disk; the only write is the
controller's own `disk-health-state.json` inside its data volume.
Disposition of what the run did create:
- **Two synthetic hub events (`events` id 2964, 2965) and one `notification_log` row (id 689)** on the
live hub. **Left in place deliberately.** Both messages are self-labelling
(`"R-328 severity probe (…) - synthetic, no real disk fault"`), and deleting rows from the
production hub DB is a riskier act than leaving two clearly-marked probe rows. Named here so they
are not mistaken later for a real disk fault on demo-hp.
- **One real operator email** resulting from row 689.
- A local copy of `hub.db`/`-wal`/`-shm` in the session scratchpad only (not committed, not exported).
---
## 11. Register rows
| Row | State | Owner |
|-----|-------|-------|
| **R-328** — the severity drop: `"warn"` coerced to `info`, emailed to nobody | **CLOSED** (controller v0.215.0), proven live side by side | CC |
| **R-329**`app_start_failed` carries the identical defect | READY — **not fixed here**; needs a decision on whether it should notify at all | Viktor |
| **R-330** — Phase 2: collect SMART attrs 187/199/188 + persist samples | READY — a declared wire change, hub models it in the same session under G-1 | CC |
| **R-331** — Phase 3: growth-rate detection; revisit the static 64 | READY, blocked on R-330 | CC |
| **R-332** — the Fail path has never fired on real hardware | **WATCHING** | CC |
| **R-333** — NVMe temperature bands; agent `smartctl` has no `-n standby` | READY (S each) | Viktor decides (a); CC does (b) |
| **R-334** — released with no golden carrying it (gate waiver) | READY — now applies to **v0.216.0** | CC bakes; **Viktor vouches** |
| **R-335** — one physical disk walked twice per run, sustaining against itself | **CLOSED** (controller v0.216.0) | CC |
`smartd`-on-DooPlex-alerts-nobody is recorded in `DIAG-smart-passed-trap-2026-08-14.md` §8 as the
same shape one layer out.
---
## 12. Observations — noticed, NOT acted on
1. **`app_start_failed` has the identical severity defect** (`notifier.go` ~L546, `"warn"`). Left
untouched per scope. It needs a prior decision — should a stopped app email the customer at all? —
because flipping the string alone converts a silent event into a mail flood on a crash-looping box.
**R-329.**
2. **The 55/60 °C bands are spinning-disk bands being applied to NVMe, and this is close to biting.**
Adopted unchanged from the operator's Prometheus config by explicit decision — but demo-hp's
**healthy** Toshiba NVMe idles at **53 °C**, i.e. **2 °C below Figyelmeztetés and 7 °C below Hiba**,
and NVMe routinely passes 60 °C under sustained write with no fault. As shipped, a healthy customer
NVMe under load can be reported as **Hiba** — the single worst outcome this feature can produce, and
the one leg 1 exists to guard. Not changed here because the threshold is a stated, settled operator
decision; flagged rather than overridden. **R-333(a) — recommend splitting the bands by device
class, or dropping them for NVMe and relying on `critical_warning`.**
3. **The agent runs bare `smartctl -a -j` with no `-n standby`**
(`felhom-agent/internal/storage/hostops.go:368`), so every poll wakes a spun-down drive, and 6h → 1h
multiplies that by six. Recorded, not acted on, per the task's instruction. demo-hp is all-flash so
the measurement could not reveal it. Mitigating datum from the fixture: the failing drive logged
only **3375 load cycles in 60505 power-on hours** (~one per 18 h), so this duty cycle barely spins
down at all. **R-333(b).**
4. **`source ~/.config/credentials` prints two recovery codes to the terminal.** The file contains
hyphenated keys (`R_DEMO-FELHOM`, `R_DEMO-HP`) that bash cannot assign, so sourcing it emits
`command not found` errors **containing the secret values**. Anything that sources that file leaks
them into logs, scrollback and transcripts. Not a code defect and out of scope; worth quoting
values from it by other means, or renaming the keys.
5. **`golden_currency_gate.py` has no waiver parser.** Its own failure text says *"record a waiver in
`OPEN-ITEMS.md` — never a bypass"*, but nothing reads such a waiver, so the only way past it is the
bypass it warns against. See §13.
---
## 13. Deviations, stated plainly
- **`git push --no-verify` was used once**, on the `felhom.eu` docs push (`767960b`), and only there.
Cause: `golden_currency_gate.py` correctly convicts the fact that controller **v0.215.0 is released
and no golden carries it** (newest bake 0.214.0), so a *newly installed* machine would receive
0.214.0 — without the severity fix. A golden bake was out of the task's scope, and its second half
(vouching in the hub's day-0 artifact manifest) is operator-password-gated, so CC cannot complete it;
a baked-but-unvouched golden is worse than none. Recorded as **R-334** with the bake+vouch owners
named. CI re-runs the same entry point and will mail the operator. The running fleet is unaffected.
- **One pre-existing test changed meaning by design:** `TestDiskVerdictFor`'s
`critical_warning>0 → warn` case is now `→ fail` (truth-table row 4 — NVMe's own critical flag is a
device declaration, not a drifting counter). `TestDiskHealthCheck_DegradationOnce` and its siblings
were rewritten into the scenario groups because they encoded the pre-v0.215.0 single-alert behaviour
the task deliberately replaces (Scenario C).
---
## 14. R-335 — a defect in v0.215.0, found live, fixed as v0.216.0
**How it was found.** Not by a test and not by review: by reading the release's own **positive
observable** against the release's own **persisted artefact**. The hourly check logged *"3 disk(s)
evaluated"*; `disk-health-state.json` held **two** records. Two artefacts that should have agreed did
not.
**Cause.** demo-hp's `c11-scratch` and `felhom-backup` are the same physical NVMe (`/dev/nvme0n1`) and
resolve to the same `diskKey`, so one disk was walked twice in a single run.
**Why it mattered.** `RunDiskHealthCheck` writes a disk's new record before the next entry reads it, so
the **second** copy of an aliased disk consumed the **first** copy's write as its prior. The disk
therefore **sustained against itself and reached Hiba on a first sighting** — defeating truth-table
row 6, the single rule separating a one-hour benign excursion from a false critical alert — and would
have emitted **two identical events** for one drive.
**Severity in practice: latent, not active.** Nothing fired on demo-hp because all three entries are
healthy with zero counters. But any aliased disk developing one pending sector would have gone
straight to Hiba, which is precisely the outcome §8 leg 1 exists to prevent. Aliasing is not exotic —
it is the *normal* shape whenever a box has two PVE storage entries on one physical device.
**Fix (v0.216.0, `90f2545`).** Each `diskKey` is evaluated once per run. Both entries stay marked
`seen`, so neither is mistaken for a disappeared disk, and the card still renders **both** storage
rows — the dedup is about state and alerts, not display. Pinned by
`TestDiskCheck_SameDiskTwiceIsEvaluatedOnce`, red-proof run and reverted (§5).
**Deployed:** `gitea.dooplex.hu/admin/felhom-controller:0.216.0 Up 6 seconds (healthy)`.
**Confirming cycle on v0.216.0 — CONFIRMED LIVE, 09:31:35Z:**
```
live image: gitea.dooplex.hu/admin/felhom-controller:0.216.0 Up About an hour (healthy)
2026/08/14 09:31:35 [INFO] [web] disk-health check complete: 2 disk(s) evaluated, 0 alert(s)
grep -c disk_health_degraded: 0
```
**`2 disk(s) evaluated` now matches the 2 persisted records.** The count and the artefact agree, which
is the disagreement that exposed R-335 in the first place. Still zero alerts, still both card rows.
### The redeploy also proved persistence live — a gap §9 had listed as unproven
The 0.215.0 → 0.216.0 redeploy **replaced the container**, and the state file came back intact:
```json
"path:/var/lib/vz": {"verdict": 1, "changed_at": "2026-08-14T07:23:14.640216851Z", ...}
"uuid:91d2dc2d-…": {"verdict": 1, "changed_at": "2026-08-14T07:23:14.640216851Z", ...}
```
That `changed_at` was written by **v0.215.0's first cycle at 07:23Z**, before the container was
destroyed and rebuilt. The v0.216.0 container read it back and preserved it rather than stamping a
fresh time — so the new container **loaded the pre-restart record instead of silently re-baselining**.
That is Scenario L observed on real hardware, not just through the production-path unit test, and it
is exactly the behaviour that was impossible before v0.215.0 (the baseline was in-memory).
It also incidentally confirms the unchanged-verdict path: `changed_at` is preserved across four checks
and two controller versions because the verdict never changed, rather than being churned every cycle.
**What this still does NOT prove:** these disks are healthy and were never alerted, so the stronger
half — *an already-ALERTED disk not re-alerting after a restart* — remains unit-tested only. R-332
stands.
**Process note, recorded because it nearly cost the fix.** The red-proof harness reverts with
`git checkout --`, which restores to `HEAD`. Running a red-proof against an **uncommitted** fix
therefore *deletes the fix* along with the mutation — which happened here and was caught only by
re-grepping the source afterwards. Commit the fix before red-proofing it, or snapshot outside git.
+45 -7
View File
@@ -14,7 +14,11 @@
| `PrimaryBackupPath` / `RecoveryUnitPath` / `RecoveryUnitComposePath` / `RecoveryUnitManifestPath` | controller/internal/appbackup/paths.go | `(nsRoot[, stackName]) string` | All backup dir layout | Take the NAMESPACE ROOT, not a bare drive path | | `PrimaryBackupPath` / `RecoveryUnitPath` / `RecoveryUnitComposePath` / `RecoveryUnitManifestPath` | controller/internal/appbackup/paths.go | `(nsRoot[, stackName]) string` | All backup dir layout | Take the NAMESPACE ROOT, not a bare drive path |
| `AppDBDumpPath` / `AppVolumeDumpPath` / `AppDataDir` | controller/internal/appbackup/paths.go | `(nsRoot, stackName) string` | Per-app dump/data dirs | Same nsRoot contract. `AppDataDir`'s final segment is the app's real appdata dir NAME — NOT always the stack name (paperless-ngx → `paperless`); resolve via `AppDataDirNames` first (F-S2/F-S3) | | `AppDBDumpPath` / `AppVolumeDumpPath` / `AppDataDir` | controller/internal/appbackup/paths.go | `(nsRoot, stackName) string` | Per-app dump/data dirs | Same nsRoot contract. `AppDataDir`'s final segment is the app's real appdata dir NAME — NOT always the stack name (paperless-ngx → `paperless`); resolve via `AppDataDirNames` first (F-S2/F-S3) |
| `AppDataDirNames` / `AppDataBindsPresent` | controller/internal/appbackup/paths.go | `(hddPath, stackName string, hddMounts []string) []string` / `(hddPath, hddMounts) bool` | Resolve the real `appdata/<name>` dir(s) from compose `${HDD_PATH}` binds (F-S2/F-S3) | `hddMounts` = ParseComposeHDDMounts shape. Deduped+sorted; falls back to `[stackName]` when no appdata bind. Tier-2 (`backup.Manager.tier2AppDataName`) refuses N>1; migrate (`stacks.Manager.ResolveAppDataDirNames`) loops N. `BindsPresent` drives the WARN-on-missing-declared-dir | | `AppDataDirNames` / `AppDataBindsPresent` | controller/internal/appbackup/paths.go | `(hddPath, stackName string, hddMounts []string) []string` / `(hddPath, hddMounts) bool` | Resolve the real `appdata/<name>` dir(s) from compose `${HDD_PATH}` binds (F-S2/F-S3) | `hddMounts` = ParseComposeHDDMounts shape. Deduped+sorted; falls back to `[stackName]` when no appdata bind. Tier-2 (`backup.Manager.tier2AppDataName`) refuses N>1; migrate (`stacks.Manager.ResolveAppDataDirNames`) loops N. `BindsPresent` drives the WARN-on-missing-declared-dir |
| `UserdataDir` / `EnsureUserdataSkeleton` / `EnsureDirOwned` | controller/internal/appbackup/userdata.go | `(nsRoot)` / `(path, gid int)` | userdata/ tree w/ 2775 setgid gid-1000 convention | Linux-only chown via build-tag twin userdata_linux.go | | `UserdataDir` / `ImportDir` / `EnsureUserdataSkeleton` / `EnsureDirOwned` | controller/internal/appbackup/userdata.go | `(nsRoot)` / `(nsRoot)` / `(nsRoot, dirs []string)` / `(path, gid int)` | userdata/ tree w/ 2775 setgid gid-1000 convention. **R-75:** `ImportDir` is the CANONICAL drop-zone (`<nsRoot>/userdata/import`) and callers MUST resolve it against the SYSTEM namespace, never an app's HDD_PATH — use `stacks.Manager.GetImportRoot()`. `EnsureUserdataSkeleton` now takes the dir set: build it with `BuildUserdataSkeleton(DeriveUserdataDirs(stacksDir))`, or via `Manager.EnsureUserdataSkeleton` / `web.Server.ensureUserdataSkeleton`. | Linux-only chown via build-tag twin userdata_linux.go. **The set MUST stay sorted**`fbNeedsRecreate` force-recreates FileBrowser on any byte diff and the naive map-order derivation measured 20/20 distinct (SPIKE P6). `UserdataSkeletonCarry()` is the old hardcoded list, retained forever so derivation can only ADD (zero removals). |
| `BuildUserdataSkeleton` / `UserdataSkeletonCarry` / `DeriveUserdataDirs` | appbackup/userdata.go, stacks/skeleton_derive.go | `([]string)` / `()` / `(stacksDir)` | catalog-derived userdata skeleton (R-75) | Derives `${USERDATA_PATH}` binds only — `${IMPORT_PATH}` is NOT part of a drive skeleton (one root, system drive, `Manager.EnsureImportRoot`). Do NOT wire the catalog sync to `SyncFileBrowserMounts`. |
| `appbackup.ValidateRelPath` / `ValidRoot` | controller/internal/appbackup/classify.go | `(root, path)` / `(root)` | THE single path-safety refusal set for every `${VAR}`-relative catalog path | Shared by `backup:` and `data_paths:`. **Do not write a second path validator.** |
| `stacks.ValidateDataPaths` | controller/internal/stacks/datapaths.go | `(entries, binds, appName, logger)` | `data_paths:` annotation validation | ASYMMETRIC on purpose (Fork-3): malformed PATH ⇒ whole-block reject (data handling, `backup:` precedent); unknown ROLE ⇒ fails OPEN, one WARN (presentation, `Lifecycle` precedent). |
| `web.fileBrowserLink` / `importFolderLink` | controller/internal/web/filebrowser_link.go | `(domain, sourceName, relPath)` | FileBrowser Quantum deep link | Template read out of the shipped router (SPIKE P2). **`url.PathEscape` per segment — NEVER `QueryEscape`** (space→`+` is a literal plus in a path). Let `html/template` do the attribute escaping; do not pre-escape. |
| `HumanizeBytes` | controller/internal/appbackup/appdata.go | `(b int64) string` | Human byte sizes | Exported canonical; private clones exist (§6) | | `HumanizeBytes` | controller/internal/appbackup/appdata.go | `(b int64) string` | Human byte sizes | Exported canonical; private clones exist (§6) |
| `stablePathForName` / `agentWhere` | controller/internal/web/intermediary.go | `(name/registeredPath) string` | Map registry stable path `/mnt/felhom-drives/<n>` ↔ raw agent mount | Registry stores STABLE path; agent ops take the RAW mount — always convert | | `stablePathForName` / `agentWhere` | controller/internal/web/intermediary.go | `(name/registeredPath) string` | Map registry stable path `/mnt/felhom-drives/<n>` ↔ raw agent mount | Registry stores STABLE path; agent ops take the RAW mount — always convert |
| `offsiteRestoreRootFor` | controller/internal/backup/offbox_verify_copies.go | `(drivePath string) string` | THE only place `backups/offsite-restore` is spelled | `offboxRestoreScratchDir` builds on it — the listing/delete surface MUST resolve byte-identical paths to what the restore wrote. Do not re-hardcode the segments (they were open-coded in 3 places before v0.147.0) | | `offsiteRestoreRootFor` | controller/internal/backup/offbox_verify_copies.go | `(drivePath string) string` | THE only place `backups/offsite-restore` is spelled | `offboxRestoreScratchDir` builds on it — the listing/delete surface MUST resolve byte-identical paths to what the restore wrote. Do not re-hardcode the segments (they were open-coded in 3 places before v0.147.0) |
@@ -53,6 +57,10 @@
| Symbol | File | Short signature | Use for | Gotchas | | Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---| |---|---|---|---|---|
| `backup.ErrOffboxSealedPackageHeld` + `IsOffboxSealedPackageHeld` + `sealedPackageHeld` + `OffboxAwaitingRecoveryKey` (R-241, v0.206.0) | controller/internal/backup/offbox.go | sentinel; `(error) bool`; `() bool`; `() bool` | **THE MINT GUARD** — a box never creates a repository key while the hub holds a sealed package for it | **The guard is a CONJUNCTION** (package held AND no key present). Widening it to "never mint" leaves a first-time box unable to start, waiting for a package that will never exist — pinned by `TestR241_ScenarioB_FirstTimeBoxStillMints`. **The refusal is a HOLDING state, not a failure:** `ApplyOffsiteTarget` catches the sentinel and still writes the transport, so `/recovery`'s synchronous tier-up (R-219) can bring the tier up the instant the key arrives; returning the error instead leaves `needsOffsiteCredential` true and the hub re-staging a consumed credential for ever. `OffboxAwaitingRecoveryKey` is **DERIVED, never stored** — and **`t.Enabled` is load-bearing in it**: a customer who switched off-site OFF is not awaiting anything (the Scenario-E carve-out `needsOffsiteCredential` makes two functions above; the first draft omitted it and an existing test caught it). A nil settings store reads as "no package held" — a transient read failure must never become a permanently-held tier |
| `settings.HubEscrowKeySHA256` + `SetHubEscrowKeySHA256` / `GetHubEscrowKeySHA256`, and `OffsiteRecoveryOffer` **shape (c)** (R-241, v0.206.0) | controller/internal/settings/settings.go, controller/internal/backup/offbox.go | `(sha, checkedAt string) error` / `() (string, string)` | **THE DISCRIMINATOR the recovery screen asks** — does the hub hold a package for a key other than the one we use? | **The comparison was ALREADY computed on every ACK since SLICE 3 and persisted nowhere** — that is R-241's second half. Wire the recorder in `main.go`'s `EscrowAutoConfirmer` literal or shape (c) reads an empty hash for ever and the fix ships INERT (pinned by `TestMainWiresRecordEscrowKeyHash`). **§7.2 staleness, decided:** a KNOWN DIFFERENCE offers **however old the reading** — age is deliberately NOT gated on, because gating makes a box offline from the hub silently stop offering; an **ABSENT hash falls back to (a)/(b)** and does NOT offer, because `""` is the hub positively saying its package seals no key (legacy hash-less escrow), not an unknown. `CheckedAt` is for diagnosis, never a gate |
| `backup.AbandonStatus` / `AbandonSweep` / `CancelAbandon` / `ClearAbandonPurgeIfConfirmed` / `ExtendAbandon` / `StopAbandon` + `AbandonGraceDays` (R-241, v0.206.0) | controller/internal/backup/offbox_abandon.go | see file | **The 14-day abandonment countdown** — the ONLY thing in the product that deletes a customer's off-site history | **BOTH HALVES OR NEITHER.** The set-aside store and the sealed package that protects it are two halves of one thing; removing only one leaves a package that opens nothing, or ciphertext nobody can decrypt. Not atomic across two machines, so it is a **two-phase commit**: delete the store, set `AbandonPurgeRequested`, and keep declaring it until the hub's ACK stops reporting a superseded package — the confirmation rides the SAME ACK as the request. **The countdown starts in `ResetOrphanedRepo`, NOT in the shared `resetOrphanedRepo`** — the helper is also the UNCLAIMED auto-reset, where nobody decided anything. **The recovery offer stays reachable for the whole grace** (a grace in which recovery is impossible is decorative). **Drive it with `SetOffboxClock`, never a shortened live timer** (§7.4). A transport failure leaves the countdown DUE so tomorrow retries; the operator levers REFUSE rather than no-op when nothing is running or the store is already gone |
| `settings.SyncRecoveryOfferEpoch` / `PostponeRecoveryNoticeForEpoch` / `OptOutRecoveryRemindersForEpoch` + `web.recoveryBannerCookie` (R-241, v0.206.0) | controller/internal/settings/settings.go, controller/internal/web/recovery_handlers.go | `(offered bool, now) (RecoveryOfferView, error)` | **The offer EPOCH** — "once per entry into the offered state", not once ever | **Sync the epoch FIRST and UNCONDITIONALLY in `recoveryInterrupts`.** The first draft returned early when the offer was false, so the FALLING edge was never recorded, `RecoveryOfferActive` stayed true through a settled period, and the next entry counted as a continuation — **the exact defect the epoch exists to fix, reintroduced inside the fix**. Dismissals are recorded against the epoch they were made in, so a fresh entry resets them **by arithmetic**, with nothing to clear. **Three levers, three scopes, and NONE removes the entry point on `/backups/remote`:** the banner cookie is a browser SESSION cookie (no MaxAge — cleared on login) and persists nothing; the reminder opt-out is durable but silences the BANNER ONLY; "most nem" suppresses the full page only |
| `atomicWrite` | controller/internal/backup/recovery_unit.go | `(path, data, perm) error` | Atomic file writes (backup pkg) | tmp+rename; no dir creation, no fallback | | `atomicWrite` | controller/internal/backup/recovery_unit.go | `(path, data, perm) error` | Atomic file writes (backup pkg) | tmp+rename; no dir creation, no fallback |
| `writeFileAtomic` | controller/internal/bootstrap/bootstrap.go | `(path, b) error` | controller.yaml writes from bootstrap | Always 0600 (holds local-api token + hub key) | | `writeFileAtomic` | controller/internal/bootstrap/bootstrap.go | `(path, b) error` | controller.yaml writes from bootstrap | Always 0600 (holds local-api token + hub key) |
| `writeConfig0600` | controller/internal/api/router.go | `(path, body) error` | config writes via API | ALWAYS chmods 0600 even pre-existing (F8); direct-write fallback on bind-mount EBUSY (non-atomic!) | | `writeConfig0600` | controller/internal/api/router.go | `(path, body) error` | config writes via API | ALWAYS chmods 0600 even pre-existing (F8); direct-write fallback on bind-mount EBUSY (non-atomic!) |
@@ -68,6 +76,11 @@
| `infra.SambaContainerName` / `SambaPassdbVolume` / `SambaPassdbMount` | controller/internal/infra/samba.go | consts | single source of truth for the samba container identity | the compose renderer interpolates them; stacks/backup/monitor read them. The CONTAINER name (`felhom-samba`) is NOT the stack name (`samba`) — `EffectiveProtected` needs the container one | | `infra.SambaContainerName` / `SambaPassdbVolume` / `SambaPassdbMount` | controller/internal/infra/samba.go | consts | single source of truth for the samba container identity | the compose renderer interpolates them; stacks/backup/monitor read them. The CONTAINER name (`felhom-samba`) is NOT the stack name (`samba`) — `EffectiveProtected` needs the container one |
| `sambaWriteAtomic` | controller/internal/stacks/samba.go | `(path, data, mode) error` | samba smb.conf/compose writes | tmp+**fsync**+rename (the only one of these that fsyncs). Fourth atomic-write helper in the tree — see §6 | | `sambaWriteAtomic` | controller/internal/stacks/samba.go | `(path, data, mode) error` | samba smb.conf/compose writes | tmp+**fsync**+rename (the only one of these that fsyncs). Fourth atomic-write helper in the tree — see §6 |
| `Loop.writeMarker` / `Recover` | controller/internal/quiesce/quiesce.go | `(m Marker)` / `()` | Quiesce crash-safety | Marker written BEFORE stopping stacks; Recover restarts stranded stacks at boot | | `Loop.writeMarker` / `Recover` | controller/internal/quiesce/quiesce.go | `(m Marker)` / `()` | Quiesce crash-safety | Marker written BEFORE stopping stacks; Recover restarts stranded stacks at boot |
| `quiesce.TieredBackend` + `Loop.resolveDueTiers` / `quiesceAndPollTiers` | controller/internal/quiesce/tiers.go, quiesce.go | `Tiers/DueFor/StartBackupFor/BackupStatusFor`; `resolveDueTiers(ctx) ([]dueTier,bool,error)` | THE R-82 multi-tier backup schedule — several whole-guest tiers (local daily + PBS weekly) reconciled into ONE quiesce window | **Both tiers due ⇒ ONE stop/start pair**, never two (two = two app outages for one night). Tiers run SEQUENTIALLY (vzdump holds a guest lock) and the app stays down until the LAST tier snapshots — resuming earlier loses app-consistency on the DR tier. Order is fast-first (agent advertises primary first) or downtime blows up. `ErrTiersUnsupported` (route 404) ⇒ pre-R-82 agent ⇒ degrade to the untargeted path and **STILL BACK UP** — never read it as "nothing due". |
| `quiesce.failureBreaker` + `Loop.dropBackedOffTiers` / `noteTierFailure` / `noteTierSuccess` | controller/internal/quiesce/breaker.go, quiesce.go | `blocked/recordFailure/recordSuccess(target, now)`; `backoffFor(n) time.Duration` | **R-88** — a tier whose backups keep failing stops re-quiescing. Backoff 15m→30m→1h→2h→4h (cap), reset on success | **It gates the QUIESCE, not the backup** — the harm was never the failing backup, it was the app outage taken to attempt it, so backed-off tiers are dropped from the due set BEFORE any stack is stopped. **Per TARGET** — a broken offsite tier must never suppress a healthy local one (`TestBreaker_OneFailingTierDoesNotSuppressAHealthyOne`). **Never permanent** — the cap bounds the retry INTERVAL, it never stops retrying; a latched breaker is a silent backup outage, worse than the loop it replaces. **`TriggerNow` is never gated** (it already bypasses due-ness and the window gate), though a manual run still RECORDS its outcome. **`stillRunning` is NOT a failure** — a first full offsite snapshot legitimately runs for hours. State is **in-memory on purpose**: a restart forgets the backoff and re-attempts, which is the cheap direction to fail. Log the deferral ONCE when armed, never per tick. |
| `quiesce.TierNotifier` + `Loop.SetTierNotifier` / `noteTierFailure` / `noteTierSuccess` | controller/internal/quiesce/breaker.go, quiesce.go | `BackupFailed(tier,msg,err)` / `BackupRecovered(tier,msg)`; `SetTierNotifier(n)` INIT-ONLY | **R-97a** — the whole-guest backup tier reports its outcome to the hub | A **seam, not an import** — quiesce keeps no dependency on `internal/notify` (same reason `windowStartFn` is injected). Wired by a setter because main.go builds the notifier AFTER the loop; `nil` = unprovisioned guest, not an error. **Edge-triggered:** failure fires only when the breaker ARMS (`n == 1`), never per retry — the cadence is 15m/30m/1h/2h/4h and an event per attempt is an inbox nobody reads. Recovery rides `recordSuccess`'s existing bool. **Event types are OPERATOR-ONLY** (`whole_guest_backup_failed`/`_recovered`, hub >= v0.78.0) — NOT `backup_failed`, which has a customerMessages entry AND sits in live `enabled_events`, so it would email the CUSTOMER about a backup they cannot act on. `WholeGuestBackupDetails.Tier` is load-bearing: the hub keys its per-tier cooldown on it. |
| `quiesce.Loop.SuppressedStacks` + `markQuiesced` / `markUnquiesced` | controller/internal/quiesce/suppress.go | `() map[string]bool` (nil-safe on a nil *Loop) | **R-97b** — an app THIS controller stopped for a backup is not a fault | Consumed at the SINGLE derivation point `classifyRunStates` (which computes both the banner dead-list and the notifier Down-set — keep it one place). **Cycle-keyed, not state-based:** v0.164.0's `!= StateStopped` filter cannot see an app caught MID-RESTART (`starting`/`unhealthy`), which is how BookStack alarmed on 2026-07-27. The window (`quiesceAlarmGrace` = 180 s, derived from the deploy flow's 120 s health timeout and Mealie's 60 s start_period) **EXPIRES** — permanent suppression turns a loud false alarm into a silent real one. Open-ended while the cycle runs (a first offsite snapshot legitimately takes hours). |
| `agentapi.BackupTiers` / `BackupDueFor` / `StartBackupFor` / `BackupStatusFor` | controller/internal/agentapi/backup_tiers.go | `(ctx[, target]) (…, error)` | The per-tier agent surface (agent >= v0.97.0) | `targetQuery("")` returns an EMPTY suffix so an untargeted call hits the pre-R-82 route byte-for-byte. `BackupTiers` maps a 404 to `ErrTiersUnsupported` — the documented ROUTE-PROBE capability signal, NOT a `featureProbes` row (the loop needs the tier LIST, not a yes/no). |
### Compose ops / stack lifecycle ### Compose ops / stack lifecycle
@@ -78,7 +91,13 @@
| `Manager.PersistUnitRedeployConfig` (R-47, v0.153.0) | controller/internal/stacks/deploy.go | `(name, env map[string]string) error` | the PERSIST half of `RedeployFromEnv` — app.yaml + locked fields + in-memory flags, **starts nothing** | **TRAP: the restore paths must use THIS, never `RedeployFromEnv`.** RedeployFromEnv ends in a full `up -d`, which before the replay IS the H4 race. RedeployFromEnv is now literally this + the unchanged up-and-report tail | | `Manager.PersistUnitRedeployConfig` (R-47, v0.153.0) | controller/internal/stacks/deploy.go | `(name, env map[string]string) error` | the PERSIST half of `RedeployFromEnv` — app.yaml + locked fields + in-memory flags, **starts nothing** | **TRAP: the restore paths must use THIS, never `RedeployFromEnv`.** RedeployFromEnv ends in a full `up -d`, which before the replay IS the H4 race. RedeployFromEnv is now literally this + the unchanged up-and-report tail |
| `Manager.StartStackServices` (R-47, v0.153.0) | controller/internal/stacks/manager.go | `(name string, services []string) error` | scoped `compose up -d <svc>...` — the DB-only window a dump is replayed in | **REFUSES an empty list** (argument-less `up -d` is a FULL start — the one silent fall-through that would reintroduce the race). No `logPostStartStatus`: the app containers are absent on purpose. Never `RestartStack` here — it is a full up in disguise | | `Manager.StartStackServices` (R-47, v0.153.0) | controller/internal/stacks/manager.go | `(name string, services []string) error` | scoped `compose up -d <svc>...` — the DB-only window a dump is replayed in | **REFUSES an empty list** (argument-less `up -d` is a FULL start — the one silent fall-through that would reintroduce the race). No `logPostStartStatus`: the app containers are absent on purpose. Never `RestartStack` here — it is a full up in disguise |
| `appbackup.DBServiceNames` / `dbTypeForImage` (R-47, v0.153.0) | controller/internal/appbackup/dbservices.go | `(composePath string) ([]string, error)` | naming the compose SERVICE(s) holding a database, sorted | yaml.v3 `services:` MAP parse — **never a line scan** (immich's top-level `immich_ml_cache:` / `immich_postgres_data:` volume keys look exactly like services). `dbTypeForImage` is shared with `DiscoverDatabases`, which is what makes "a dump exists ⇒ a service can be named" hold. An error means CANNOT-TELL, never "no database" — callers refuse when a dump exists | | `appbackup.DBServiceNames` / `dbTypeForImage` (R-47, v0.153.0) | controller/internal/appbackup/dbservices.go | `(composePath string) ([]string, error)` | naming the compose SERVICE(s) holding a database, sorted | yaml.v3 `services:` MAP parse — **never a line scan** (immich's top-level `immich_ml_cache:` / `immich_postgres_data:` volume keys look exactly like services). `dbTypeForImage` is shared with `DiscoverDatabases`, which is what makes "a dump exists ⇒ a service can be named" hold. An error means CANNOT-TELL, never "no database" — callers refuse when a dump exists |
| `Manager.StartStack/StopStack/RestartStack/UpdateStack` | controller/internal/stacks/manager.go | `(name string) error` | Lifecycle | Protected stacks refuse stop; all funnel through composeExec | | `Manager.StartStack/StopStack/RestartStack/UpdateStack` | controller/internal/stacks/manager.go | `(name string) error` | Lifecycle | Protected stacks refuse stop; all funnel through composeExec. **NOT writers of desired state (R-166)** — 14 call sites, only 2 are the customer; recording intent here would make a nightly backup indistinguishable from the customer pressing Stop. Use `SetDesiredState` at the intent point instead |
| `Manager.SetDesiredState` / `DesiredStateOf` / `BackfillDesiredState` (R-166, v0.189.0) | controller/internal/stacks/desiredstate.go | `(name, desired string) error` / `(Stack) string` / `() int` | THE customer-intent record — `app.yaml` `desired_state`, tri-state `""`/`running`/`stopped` | **ONE OWNER: the customer's action.** Writers are the API action switch, `DeployStack`, `UpdateOptionalConfig`'s redeploy branch, and the `.fab` restore adapter — nothing else, ever. **`""` (absent) means UNKNOWN, never "running"**: every pre-v0.189.0 app.yaml reads absent, so treating it as running would start every deliberately-stopped app on upgrade. Write intent BEFORE the act and REFUSE the act if it fails (§8.2). Backfill is **running-only** — never infer `stopped` from zero containers, that inference IS the defect |
| `Manager.DriveLive` (R-171, v0.190.0) | controller/internal/stacks/deploy.go | `(hddPath string) bool` | is an app's data drive a live mountpoint RIGHT NOW | Wraps the **same** `isMountPoint` seam the userdata belt uses (`manager.go`) — never write a second liveness check, the two would drift invisibly. The system/local path is legitimately not a mountpoint and returns true |
| `bootrecon.StartGate` (R-171, v0.190.0) | controller/internal/bootrecon/bootrecon.go | `MayStart(stack) (bool, reason)` | THE one question the boot sweep asks before starting anything | **Fail-safe: cannot determine ⇒ return FALSE.** One seam for all three holders (absent drive · quiesce · an in-flight app-data operation) because they differ only in the reason string. Implemented in `main.go` (`bootDriveGate`) reusing `quiesce.SuppressedStacks()`, `AppStopGuard.HeldStacks()` and `Manager.DriveLive` — never re-derive any of them. Held apps go to `Result.HeldByDrive`, **never** `StillDown` (that is the dead-app alarm's bucket) |
| the boot settle window (R-157 A, v0.190.0) | controller/cmd/controller/main.go | `bootReconcileSample` / `StableFor` / `Budget` | sample the fleet until it stops changing, then sweep ONCE | **settle + budget + one `DefaultRetryDelay` must stay under `deadAppBootGrace`** — pinned by `TestBootWindow_CommonCaseFitsInsideTheDeadAppGrace`, which is why the budget is 50 s and not 60 s. Sampling is READ-ONLY; sweeping per sample would never see a settled fleet (the sweep's own StartStack changes it). A late recovery is REPORTED (`recordLateRecovery`), never hidden by widening the grace |
| `backup.AppStopGuard` (`Begin`/`End`/`Recover`) (R-166, v0.189.0) | controller/internal/backup/appstop_marker.go | `(opID, reason, stacks) error` / `()` / `() *AppStopRecovery` | THE crash marker for stop→work→start windows (volume dump, offbox reconstitute, `.fab` export) | Its **own** file (`appstop-state.json`), never quiesce's — one file, one writer. **A `defer` is NOT the mechanism** (Campaign 8 fault 10: SIGKILL runs no defer); the marker is. Written BEFORE the stop, cleared ONLY after a restart that succeeded; a FAILED restart deliberately KEEPS it. `Recover` RETURNS its outcome rather than notifying, because it must complete before the boot reconciler while the notifier does not exist yet |
| `backup.ErrStartRefused` + `AppStopRecovery.Refused`/`Alarming()` (R-174, v0.191.0) | controller/internal/backup/appstop_marker.go | `errors.Is(err, ErrStartRefused)` / `() bool` | THE refusal-vs-failure split in the app-stop crash recovery | **A gated starter's refusal is NOT a restart failure.** `Recover`'s starter MUST be the gated `gatedAppStopStarter` (cmd/controller/main.go), never the raw `stacks.Manager` — that was the v0.189.0 defect, which started apps onto ABSENT drives at boot (R-171 one path over). A refusal goes to `Refused` (marker KEPT, silent), a real error to `Failed` (marker kept, ALARMS). Collapsing them routes a deliberate hold into `NotifyBackupFailed`, a customer-enabled type — the R-171 false alarm again. `main.go` must guard the notify with `Alarming()`, not `!= nil` |
| `Manager.DeleteStack` / `RemoveStack` | controller/internal/stacks/delete.go | `(name, removeHDDData[, backupPaths])` | THE guarded removal paths | Orphan/protected/deploying/running checks + ProtectedHDDPaths filter before any RemoveAll | | `Manager.DeleteStack` / `RemoveStack` | controller/internal/stacks/delete.go | `(name, removeHDDData[, backupPaths])` | THE guarded removal paths | Orphan/protected/deploying/running checks + ProtectedHDDPaths filter before any RemoveAll |
| `resolveContainerState` / `aggregateState` | controller/internal/stacks/manager.go | `(dockerState, dockerStatus)` / `([]ContainerInfo)` | State classification | `.State` says "running" even when unhealthy — `.Status` parse is the fix | | `resolveContainerState` / `aggregateState` | controller/internal/stacks/manager.go | `(dockerState, dockerStatus)` / `([]ContainerInfo)` | State classification | `.State` says "running" even when unhealthy — `.Status` parse is the fix |
| `Manager.logPostStartStatus` | controller/internal/stacks/manager.go | `(name, stackDir, env)` | Async post-start verification | compose up exits 0 on crash-loops; this is the detection. Goroutine + 3s, never blocks | | `Manager.logPostStartStatus` | controller/internal/stacks/manager.go | `(name, stackDir, env)` | Async post-start verification | compose up exits 0 on crash-loops; this is the detection. Goroutine + 3s, never blocks |
@@ -89,7 +108,8 @@
| `Manager.ClassifiedBinds` + `StackDataProvider.GetStackClassifiedBinds` | controller/internal/stacks/metadata.go, appbackup/appdata.go | `(name) ([]appbackup.ClassifiedBind, bool)` | Per-stack classification through the REAL LoadMetadata validate path | The wired seam Task 3 consumes; LoadMetadata is the SINGLE validation choke point (bad block → nil + one ERROR → legacy) | | `Manager.ClassifiedBinds` + `StackDataProvider.GetStackClassifiedBinds` | controller/internal/stacks/metadata.go, appbackup/appdata.go | `(name) ([]appbackup.ClassifiedBind, bool)` | Per-stack classification through the REAL LoadMetadata validate path | The wired seam Task 3 consumes; LoadMetadata is the SINGLE validation choke point (bad block → nil + one ERROR → legacy) |
| `backup.Manager.DumpAppVolumesSafe` | controller/internal/backup/backup.go | `(stackName) error` | Volume tar of a live app | Stops → dumps → restarts; surfaces BOTH errors (app may be left stopped). Check `GetDockerVolumes()!=0` + `IsProtectedStack` BEFORE calling — it stops the stack before its own volume check (see `runVolumeDumps`) | | `backup.Manager.DumpAppVolumesSafe` | controller/internal/backup/backup.go | `(stackName) error` | Volume tar of a live app | Stops → dumps → restarts; surfaces BOTH errors (app may be left stopped). Check `GetDockerVolumes()!=0` + `IsProtectedStack` BEFORE calling — it stops the stack before its own volume check (see `runVolumeDumps`) |
| `backup.Manager.ListRestorePoints` | controller/internal/backup/restore_points.go | `(stackName) ([]RestorePoint, bool)` | Restorable keep-side backups (the /api/backup/snapshots payload) | ONE point per app (the current unit); tier always 1 — never list Tier-2 (not restorable via /backup/restore) | | `backup.Manager.ListRestorePoints` | controller/internal/backup/restore_points.go | `(stackName) ([]RestorePoint, bool)` | Restorable keep-side backups (the /api/backup/snapshots payload) | ONE point per app (the current unit); tier always 1 — never list Tier-2 (not restorable via /backup/restore) |
| `backup.Manager.RestoreTier2Files` | controller/internal/backup/tier2_restore.go | `(stackName) (filesRestored int, err error)` | In-place ADDITIVE-ONLY class-C file restore from the recorded Tier-2 copy (`POST /backup/tier2/restore`) | Never overwrites/deletes live files; refusals (Hungarian) before any stop; source = recorded `DestinationPath`, never re-selected | | `backup.Manager.RestoreTier2Files` | controller/internal/backup/tier2_restore.go | `(stackName) (filesRestored int, err error)` | In-place ADDITIVE-ONLY class-C file restore from the recorded Tier-2 copy (`POST /backup/tier2/restore`) | Never overwrites/deletes live files; refusals (Hungarian) before any stop; source = recorded `DestinationPath`, never re-selected. **C9-F1 (v0.183.0): reads `hdd/` + `userdata/` ONLY — never `recovery-unit/`.** For 43 of 53 catalog apps that is a guaranteed no-op, so it now refuses with `ErrTier2NoRestorableData` BEFORE stopping the app. Ask `Tier2RestoreCoverage` first |
| `backup.Manager.Tier2RestoreCoverage` | controller/internal/backup/tier2_restore.go | `(stackName) (Tier2Coverage{Legs, HasUnit}, error)` | Answers what a Tier-2 restore CAN and CANNOT return for an app, from the RECORDED copy on disk | **C9-F1.** `Legs` = subtrees the restore reads; `HasUnit` = the copy also holds DB dumps + volume tarballs it will NEVER read. Use it to refuse up front and to decide whether the success message must disclose uncovered data. Judged from the copy, not the catalog, so a retemplated app is judged by what it actually has |
| `Manager.acquireRunning`/`releaseRunning`, `acquireMigrating` | controller/internal/backup/backup.go, controller/internal/stacks/migrate.go | `() error` | Single-flight for long ops | Copy this mutex-flag pattern for any new long-running manager op | | `Manager.acquireRunning`/`releaseRunning`, `acquireMigrating` | controller/internal/backup/backup.go, controller/internal/stacks/migrate.go | `() error` | Single-flight for long ops | Copy this mutex-flag pattern for any new long-running manager op |
### Secrets hygiene ### Secrets hygiene
@@ -101,10 +121,13 @@
| `SaveAppConfig` / `LoadAppConfigDecrypted` | controller/internal/stacks/deploy.go | `(stackDir, cfg, encKey, sensitiveVars)` | app.yaml persistence | Encrypts only `SensitiveEnvVars(meta)`; never write app.yaml directly | | `SaveAppConfig` / `LoadAppConfigDecrypted` | controller/internal/stacks/deploy.go | `(stackDir, cfg, encKey, sensitiveVars)` | app.yaml persistence | Encrypts only `SensitiveEnvVars(meta)`; never write app.yaml directly |
| `generateValue` / `randomAlphanumeric` | controller/internal/stacks/deploy.go | `(spec "password:N\|hex:N\|base64key:N\|static:v")` | Auto-generated secrets | crypto/rand-backed; reuse the spec grammar | | `generateValue` / `randomAlphanumeric` | controller/internal/stacks/deploy.go | `(spec "password:N\|hex:N\|base64key:N\|static:v")` | Auto-generated secrets | crypto/rand-backed; reuse the spec grammar |
| `Manager.GenerateSecretForField` | controller/internal/stacks/deploy.go | `(stackName, envVar) (string, bool)` | Replacement value for a RESETTABLE secret from its catalog `generate` spec (O4 restore path via `backup.SetSecretGenerator`) | REFUSES `data_key` fields, spec-less and non-secret fields; never log the value | | `Manager.GenerateSecretForField` | controller/internal/stacks/deploy.go | `(stackName, envVar) (string, bool)` | Replacement value for a RESETTABLE secret from its catalog `generate` spec (O4 restore path via `backup.SetSecretGenerator`) | REFUSES `data_key` fields, spec-less and non-secret fields; never log the value |
| `reconcileRestoreSecrets` | controller/internal/backup/restore_unit.go | `(nonSecretEnv, recoveredSecrets, secretNames, dataKeyNames)` | Recovery-unit restore env merge | Units are secret-FREE by design; secrets come from live app.yaml | | `reconcileRestoreSecrets` | controller/internal/backup/restore_unit.go | `(nonSecretEnv, unitSecrets, guestSecrets, secretNames, dataKeyNames)` | Recovery-unit restore env merge | **Precedence: UNIT WINS over guest** (the unit's secrets match the data being restored; the guest's are merely newest). Pure — new sources arrive as ARGUMENTS. Fail-closed data-key gate lives here |
| `stacks.PortableSecretEnvVars` | controller/internal/stacks/deploy.go | `(meta) []string` | **THE D5 secret boundary**: which secrets may travel on a customer drive | `type: secret` travels, `type: password` NEVER, minus the `nonPortableSecrets` code register. Withholding the password class is what licenses plaintext — do not relax one without the other |
| `buildUnitAppYaml` / `readUnitEnv` | controller/internal/backup/{recovery_unit,restore_unit}.go | `(info) []byte` / `(path, portableNames)` | The ONE place the unit's app.yaml is written / split back | Split is driven by the MANIFEST's portable names, never guessed from key names; write 0600; empty `portableNames` = schema-1 unit ⇒ everything is plain config |
| `EncryptFile` / `DecryptFile` / `IsEncryptedFAB` | controller/internal/appexport/crypto.go | password-based file crypto | .fab export bundles | scrypt-derived AES+HMAC keys | | `EncryptFile` / `DecryptFile` / `IsEncryptedFAB` | controller/internal/appexport/crypto.go | password-based file crypto | .fab export bundles | scrypt-derived AES+HMAC keys |
| `maskRepoURL` | controller/internal/sync/sync.go | `(url) string` | Logging git URLs | Strips embedded credentials | | `maskRepoURL` | controller/internal/sync/sync.go | `(url) string` | Logging git URLs | Strips embedded credentials |
| `metrics.RedactLine` | controller/internal/metrics/redact.go | `(s string) string` | ANY log line shipped off-box (issue context, log tails) | Masks password/passwd/secret/token/api-key/authorization/bearer values + 64-hex; apply BEFORE the line leaves the box — controller-side redaction is authoritative | | `metrics.RedactLine` | controller/internal/metrics/redact.go | `(s string) string` | ANY log line shipped off-box (issue context, log tails) | Masks password/passwd/secret/token/api-key/authorization/bearer values + 64-hex; apply BEFORE the line leaves the box — controller-side redaction is authoritative |
| `settingsRetrievalPasswordRevealHandler` | controller/internal/web/handlers.go | `POST /settings/retrieval-password/reveal` | **THE PATTERN for showing a secret in the UI** — an XHR that returns only the value | **Never template a secret into a page and hide it with CSS.** `display:none` / `hidden` / `type="password"` stop a browser DRAWING the value; the plaintext is still in the response body, so a `curl` of the page returns it, and it reaches caches, history and any screen-share of the source. R-249 shipped exactly that for two months and was found by it landing in a transcript. The page carries a **boolean** (`HasRetrievalPassword`); the value comes from a POST (CSRF-covered, uncacheable) and the reveal is **logged as an act**. `escrow_handlers.go` states the same rule for R. **Test on the RESPONSE BODY** — a test asserting what the customer *sees* cannot see this class at all. **Both R-254 sites are now FIXED the same way**`POST /apps/<slug>/initial-credentials/reveal` (re-reads the container, never a cached copy) and `POST /stacks/<name>/auto-field/reveal` (authorised on the field being a `type: secret` auto-field of that stack). **Per-secret, never one generic reveal-any-named-secret endpoint.** The PRE-DEPLOY hidden input is deliberate and untouched — a form must carry what it submits (README §318). Enforced by `scripts/secret_in_markup_gate.py`, whose measured blind spot (a secret under a neutral page-data key) is in its docstring; runtime body-assertion covers 4 of 27 pages — R-255. |
### Storage registry + mount detection ### Storage registry + mount detection
@@ -136,12 +159,18 @@
| `Client.AddNetStorage/ListNetStorage/RemoveNetStorage` | controller/internal/agentapi/client.go | NAS mounts (A1) | Network storage | Password passes through to agent's 0600 cred file; controller NEVER persists it | | `Client.AddNetStorage/ListNetStorage/RemoveNetStorage` | controller/internal/agentapi/client.go | NAS mounts (A1) | Network storage | Password passes through to agent's 0600 cred file; controller NEVER persists it |
| `agentapi.StatusError` | controller/internal/agentapi/client.go | `{Path, Code}` typed non-2xx GET error | Distinguishing HTTP statuses from transport errors (`errors.As`) | NEVER string-match agent error text — the capability probe keys on `Code==404` | | `agentapi.StatusError` | controller/internal/agentapi/client.go | `{Path, Code}` typed non-2xx GET error | Distinguishing HTTP statuses from transport errors (`errors.As`) | NEVER string-match agent error text — the capability probe keys on `Code==404` |
| `SupportCache.Supports` / `Client.Supports` | controller/internal/agentapi/features.go | `(ctx, prober, Feature) SupportState` | Agent-capability gate for COUPLED features (route probe, TTL 5m) | 404 ⇒ No; transport/5xx ⇒ Unknown (NEVER refuse on Unknown). New coupled feature = new `featureProbes` row + gate call at the entry point + `MinAgent:` in the CHANGELOG header (publish-train-rules.md). Web layer: `Server.netFeatures` through the `netAgent` seam | | `SupportCache.Supports` / `Client.Supports` | controller/internal/agentapi/features.go | `(ctx, prober, Feature) SupportState` | Agent-capability gate for COUPLED features (route probe, TTL 5m) | 404 ⇒ No; transport/5xx ⇒ Unknown (NEVER refuse on Unknown). New coupled feature = new `featureProbes` row + gate call at the entry point + `MinAgent:` in the CHANGELOG header (publish-train-rules.md). Web layer: `Server.netFeatures` through the `netAgent` seam |
| `agentapi.DiskVerdictFor` / `DiskVerdict.Label` / `DegradedAttributes` / `UncorrectableSectors` / `DiskPrior` / `TemperatureFailC` | controller/internal/agentapi/diskverdict.go | `(*SmartSummary, DiskPrior) DiskVerdict` | THE shared disk-health verdict (card chip + hourly check) — v0.169.0, 14-row ladder v0.215.0 | Pure — no clock, no I/O; history arrives as `DiskPrior`. nil/UNKNOWN → `DiskVerdictUnknown` (Nincs adat, NEVER alarms, row 1 is first for that reason). **Never trust `smart_status.passed`**: attrs 187/197/198 carry `thresh: 0`, so it cannot fail on unreadable sectors. A zero `DiskPrior` is the fail-safe (first sighting can only reach Figyelmeztetés). **Four labels, no fifth** — predicted failure is „Hiba". Do NOT recompute the verdict inline anywhere, and do NOT re-literal 60 °C — use `TemperatureFailC` |
| `Server.resolveBackupTargetState` / `backupTargetView` | controller/internal/web/backup_target_offer.go | `(ctx)` → state / `*BackupTargetView` (nil = render nothing) | The whole-system backup-target answer: healthy · degraded-never-configured · **TargetAbsent** (configured, drive gone) · unknown | Test seams `Server.tiersFn` + `Server.disksFn` (nil → the real client). **`degradedMessageFor` is the ONE place that decides customer copy** — add a state there, never in a template. `backupTargetView` returns **nil** for healthy AND unknown so a template typo cannot decorate a working box. R-112: this state had NO consumer for two releases; the render is server-side on `backups.html`, and the seam test drives `backupsHandler` and asserts rendered HTML |
| `Server.cachedDisks` / `RunDiskHealthCheck` | controller/internal/web/disk_health.go | `(ctx)` | Card fetch (60s TTL) / the hourly degradation check | Card uses the 60s TTL cache (anti-smartctl-storm); the CHECK fetches FRESH (`fetchDisks`). Test seams: `Server.disksFn` (source) + `Server.diskNotifyFn(notify.DiskAlert)` (sink). State is PERSISTED (v0.215.0) — a restart no longer re-baselines |
| `diskAlertDecision` / `diskAlertKindFor` / `Server.priorFor` / `Server.cardPriorFor` | controller/internal/web/disk_health_state.go | pure + `(key) agentapi.DiskPrior` | Whether an observation emits, and which message shape | Compares against the **last ALERTED** verdict, not the last observed — that is what collapses a flap to one alert. Re-alert needs doubling **AND** 24h (an AND). **`priorFor` is for the CHECK, `cardPriorFor` for the CARD** — they differ by one observation and mixing them makes the chip read one level more severe than the email |
| `diskRecord` / `writeDiskState` / `Server.loadDiskStateLocked` | controller/internal/web/disk_health_state.go | `disk-health-state.json` in `cfg.Paths.DataDir` | Persisted per-disk observation + alert history | Atomic tmp+rename (the `selfupdate.SaveState` shape, copied not imported). Missing file = normal; corrupt = LOG and fall back to no-prior, **never fatal**. Written ONCE per check run. Keyed by `diskKey`. **One record per disk, NOT a sample series** — history is Phase 2/3 in `metrics.MetricsStore` |
### Notifications / hub sync ### Notifications / hub sync
| Symbol | File | Short signature | Use for | Gotchas | | Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---| |---|---|---|---|---|
| `Notifier.PushEvent` | controller/internal/notify/notifier.go | `(eventType, severity, message, details)` | Hub events | Async goroutine, 3 attempts/3s backoff. NEW event types MUST be added to hub `allowedEventTypes` or POST /event 400s; hub only emails `warning`/`error` from this path | | `Notifier.PushEvent` | controller/internal/notify/notifier.go | `(eventType, severity, message, details)` | Hub events | Async goroutine, 3 attempts/3s backoff. NEW event types MUST be added to hub `allowedEventTypes` or POST /event 400s. **SEVERITY IS AN EXACT WIRE CONTRACT: `{"info","warning","error","critical"}` and nothing else.** The hub silently COERCES any other string to `"info"` (`hub/internal/api/handler.go`, the ingest severity switch) and `severityNotifies` (`hub/internal/notify/dispatcher.go`) emails only warning/error/critical — so a typo'd severity is stored and delivered to NOBODY, with no error anywhere. **`"warn"` is not a severity.** It shipped on `disk_health_degraded` (fixed v0.215.0, R-328) and is STILL live on `app_start_failed` (R-329) |
| `notify.DiskAlert` / `DiskAlertKind` / `DiskAlertKind.Severity()` | controller/internal/notify/notifier.go | `NotifyDiskHealthDegraded(DiskAlert)` | The disk-health alert payload + its five Hungarian message shapes | The notifier owns customer copy — pass a `DiskAlert`, never a pre-formatted string, or Hungarian scatters across packages. `Severity()` is the ONE mapping kind→hub severity and is exported so any package can assert the contract instead of duplicating the literal |
| `Notifier.Notify*` convenience methods | controller/internal/notify/notifier.go | typed wrappers (backup/DB/storage/channel/DR…) | Standard events | Add a typed wrapper rather than raw PushEvent calls | | `Notifier.Notify*` convenience methods | controller/internal/notify/notifier.go | typed wrappers (backup/DB/storage/channel/DR…) | Standard events | Add a typed wrapper rather than raw PushEvent calls |
| `report.BuildReport` / `Pusher.Push` | controller/internal/report/builder.go + pusher.go | periodic hub report | Box→hub reporting | ACK carries `config_version``ConfigRefresher.Reconcile` | | `report.BuildReport` / `Pusher.Push` | controller/internal/report/builder.go + pusher.go | periodic hub report | Box→hub reporting | ACK carries `config_version``ConfigRefresher.Reconcile` |
| `report.Trigger` (`NewTrigger`/`Fire`/`Run`) | controller/internal/report/trigger.go | `Fire()` after a hub-relevant user action | THE out-of-cycle report push (v0.139.0) — fire via `api.Router.reportPushNow` / `web.Server.reportTriggerNow`, both nil-safe | Coalesce-and-eventually-fire (trailing edge; quiet 2s, min spacing 15s). NEVER add retries (Pusher owns them); NEVER reuse the `internal/sync` REFUSE-debounce for hub pushes (a refused fire loses the update until the next cycle). Fire only AFTER a successful local commit | | `report.Trigger` (`NewTrigger`/`Fire`/`Run`) | controller/internal/report/trigger.go | `Fire()` after a hub-relevant user action | THE out-of-cycle report push (v0.139.0) — fire via `api.Router.reportPushNow` / `web.Server.reportTriggerNow`, both nil-safe | Coalesce-and-eventually-fire (trailing edge; quiet 2s, min spacing 15s). NEVER add retries (Pusher owns them); NEVER reuse the `internal/sync` REFUSE-debounce for hub pushes (a refused fire loses the update until the next cycle). Fire only AFTER a successful local commit |
@@ -152,13 +181,20 @@
| `bootstrap.MaybeIngest` / `RefreshConfig` | controller/internal/bootstrap/bootstrap.go | bootstrap.json → controller.yaml | Day-0 + refresh | Overwrites controller.yaml, NEVER settings.json | | `bootstrap.MaybeIngest` / `RefreshConfig` | controller/internal/bootstrap/bootstrap.go | bootstrap.json → controller.yaml | Day-0 + refresh | Overwrites controller.yaml, NEVER settings.json |
| `api.GracefulSelfRestart` | controller/internal/api/selfrestart.go | `(logger)` | Controller self-restart | Detached exit; bootstrap unit re-runs the image | | `api.GracefulSelfRestart` | controller/internal/api/selfrestart.go | `(logger)` | Controller self-restart | Detached exit; bootstrap unit re-runs the image |
| `Settings.AddPendingEvent/DrainPendingEvents` | controller/internal/settings/settings.go | offline event queue | Events while hub unreachable | — | | `Settings.AddPendingEvent/DrainPendingEvents` | controller/internal/settings/settings.go | offline event queue | Events while hub unreachable | — |
| `Manager.SetUnitNotify` + `UnitSpace` (R-158/R-167, v0.191.0) | controller/internal/backup/recovery_unit.go | `(func(stack string, err error, *UnitSpace))` | THE per-app Tier-1 recovery-unit capture failure alert — fires PER APP from `captureAllRecoveryUnits`, loop continues | **OPERATOR-TIER** (`recovery_unit_capture_failed`, in the hub's `operatorOnlyEvents`). **NEVER route it to `backup_failed`** — that type is in `DefaultEnabledEvents` and carries Hungarian copy, so it emails the CUSTOMER about a failure they cannot act on (D-c; R-158's own proposal said `backup_failed` and D-c overrides it). `UnitSpace` is **nil when the target filesystem is unreadable** and renders as *"unavailable"*, never as zeros — "0 GB free" and "we could not look" are opposite diagnoses. No controller-side cooldown: the hub owns it |
| `Manager.beginRunSummary` / `noteFailure` / `noteAttempted` / `emitRunSummary` / `SetRunSummaryNotify` (R-182, v0.194.0) | controller/internal/backup/runsummary.go | `(kind, runID) func()` / `(app, leg, reason)` / `(RunSummary)` | **THE per-run operator digest.** One `backup_run_failures` event at the end of a run listing every failed app, its leg and its reason — emitted ONLY when something failed | **The RECORD and the NOTIFICATION are different things and must stay so.** The per-app `recovery_unit_capture_failed` event is the record (hub routes it *record-only*, stored + logged every time); this digest is the notification. Before R-182 one event was both, and did neither: nine arrived, two were mailed, seven vanished before `LogNotification`. **Lifetime is `admissionSet`'s exactly** — absent collector means "no run in flight", never a stale answer. **A refusal is noted ONCE, inside `admitApp` where the verdict is taken**, not at the three legs that consult it: R-181's one-verdict-covers-all-three contract makes per-leg noting produce "2 of 1 apps failed". **Deliberate skips (disconnected / decommissioned) must NEVER be noted** — they have their own alert and a nightly digest about an unplugged drive is an ignored digest. **A clean run emits NOTHING**; silence is safe only because the hub's deadline check (`monitor/deadline.go:396,417`) raises a missed backup from report freshness independently — if that is ever weakened this design loses its footing. **`run_id` is unique per real run** (so the hub's 1-h cooldown cannot collapse a manual run into the nightly one) and **deliberately EMPTY on the periodic refresh sweep**, which must stay under that cooldown or a polled status page becomes a mail flood |
| `Manager.admitApp` / `beginAdmissionRun` / `decideAdmission` / `estimatedWriteBytes` (R-181, v0.193.0) | controller/internal/backup/admission.go | `(stackName) bool` / `() func()` | **THE reserve gate. Call it before ANY per-app backup write** — one verdict per app per run, covering the DB dump, the volume dump and the unit capture (all three write under one per-app root) | **Decided LAZILY at the app's first write, never once at run start** — app A's dump can put app B under the reserve, so a run-start verdict reads a disk that no longer exists. **Never re-decided between an app's own legs**: that is exactly the split R-181 closed (bulk written, capture refused). **Reset per run** via the closer `beginAdmissionRun` returns. **Must sit ahead of `DumpAppVolumesSafe`**, which stops the stack as its first act — a refusal decided inside it has already bounced the app. Fires **exactly one** `unitNotify` per refused app per run. Nil admission set (periodic status refresh) → decides fresh, which is still once per app per sweep. Wiring pinned by an **AST walk** in `TestAdmission_IsWiredIntoEveryProductionWriteLeg`, not `strings.Contains` |
| `Manager.floorVerdict` + `FloorUsedPercent`/`FloorFreeGiB` / `ErrCaptureFloor` / `floorReason` (R-165 B2 v0.192.0, size term R-181 v0.193.0) | controller/internal/backup/recovery_unit.go | `(*UnitSpace, estGiB float64) (*UnitSpace, floorReason)` | The pure two-question predicate behind `admitApp`: is the filesystem already below the reserve (`floorHeadroom`), and would THIS app's write take it below (`floorSize`)? | **Headroom is about the FILESYSTEM, never a per-unit cap** — a size cap is R-163 rebuilt inside one volume; the size term bounds the *delta*, not the unit. **REFUSES, never deletes:** nothing here is generational (a unit is one fixed path per app, a DB dump one fixed name), so pruning could only destroy a DIFFERENT app's only local copy — **never repurpose `pruneStalePrimaryDirs`**, which removes ORPHANED dirs from an app that moved drives and has no notion of age. Two terms (97% / 1 GiB) in `fillwatch`'s shape, deliberately BEYOND its critical band (95% / 2 GiB) so the customer is always warned first — pinned by `TestFloorSitsBelowTheCriticalWarningBand`. **`estGiB == 0` degrades to headroom-only on purpose** — refusing an app with no history makes the FIRST backup the one that can never happen. A nil reading neither refuses nor warns (§8.4). Inject `unitSpaceFn` in tests rather than manufacturing occupancy on a real disk |
| `fillwatch.Watcher` (`New`/`SetNotify`/`Check`) (R-167, v0.191.0) | controller/internal/fillwatch/fillwatch.go | `(statePath, logger, targetsFn, usageFn)``Check() error` | THE customer fill warning — warns BEFORE a filesystem fills, per FILESYSTEM (never per app: one full disk holding ten apps would fire ten times) | Emits the **pre-existing** `disk_warning`/`disk_critical` pair, which was allowlisted + copy'd + default-enabled with **no producer in any repo** until now — do NOT mint a new type beside it. **Two threshold terms, whichever trips first** (85% / 5 GiB; critical 95% / 2 GiB) because a percentage alone lies at both ends of this fleet's size range. **Edge-triggered on ESCALATION ONLY**, state persisted; de-escalation is silent and re-arms. Hysteresis dead zone between clear (75% / 7 GiB) and warn — pinned by `TestThresholdsKeepTheirHysteresisGap`. **A nil usage read is NEVER a warning** (§8.4). The hub has **no `customerMessages` entry** for either type on purpose — an entry would override the dynamic message and discard the drive label + free space |
### Scheduler / time / UI ### Scheduler / time / UI
| Symbol | File | Short signature | Use for | Gotchas | | Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---| |---|---|---|---|---|
| `Scheduler.Every` / `Daily` | controller/internal/scheduler/scheduler.go | `(name, interval/"HH:MM", fn)` | ALL background jobs | Daily is Europe/Budapest, DST-safe (`nextDailyRun` avoids Add(24h)); register in main.go block (§5) | | `Scheduler.Every` / `Daily` | controller/internal/scheduler/scheduler.go | `(name, interval/"HH:MM", fn)` | ALL background jobs | Daily is Europe/Budapest, DST-safe (`nextDailyRun` avoids Add(24h)); register in main.go block (§5) |
| `getBudapestLocation` | controller/internal/scheduler/scheduler.go | `() *time.Location` | Local-time math | web has its own `getTimezone` (§6) | | `Scheduler.UpdateDaily` | controller/internal/scheduler/scheduler.go | `(name, "HH:MM") bool` | Retime a daily job at runtime (no restart) | Per-job buffered `resched` chan + select case in `runDailyJob`; false (WARN) on invalid time / unknown-or-non-daily name; read `Schedule` under the mutex in the loop |
| `backupwindow.*` (LegTimes / GateWindow / EffectiveWindow / ParseHHMM / FmtHHMM / Valid) | controller/internal/backupwindow/backupwindow.go | pure `string``int` | Backup-window arithmetic (v0.168.0) | Offsets (W+60m/W+105m, gate W+2h..W+6h) are CONSTANTS — derived, never stored; wrap-safe modulo 1440; `EffectiveWindow(settings, yaml)` = settings>yaml>"02:30" |
| `getBudapestLocation` | controller/internal/scheduler/scheduler.go | `() *time.Location` | Local-time math | web has its own `getTimezone` (§6); quiesce has its own `budapestLocation` (window gate) — 3rd copy, see §6 |
| `Server.templateFuncMap` | controller/internal/web/funcmap.go | template.FuncMap | ALL template functions | `stateColor` outputs v2 suffixes `run/progress/warn/neutral/off`; stopped = NEUTRAL not red (operator-approved); `stateLabel` copy is frozen byte-identical (unit-tested) | | `Server.templateFuncMap` | controller/internal/web/funcmap.go | template.FuncMap | ALL template functions | `stateColor` outputs v2 suffixes `run/progress/warn/neutral/off`; stopped = NEUTRAL not red (operator-approved); `stateLabel` copy is frozen byte-identical (unit-tested) |
| `timeAgoStr` | controller/internal/web/funcmap.go | `(s RFC3339 string) string` | Ago-format for STRING timestamps | Exists because `timeAgo(time.Time)` 500'd on strings (v0.93 bug) | | `timeAgoStr` | controller/internal/web/funcmap.go | `(s RFC3339 string) string` | Ago-format for STRING timestamps | Exists because `timeAgo(time.Time)` 500'd on strings (v0.93 bug) |
| `Server.baseData` / `executeTemplate` | controller/internal/web/handlers.go + server.go | page-data plumbing | New pages | baseData injects nav/alerts/version; templates must pass `controller/scripts/template_id_gate.py` + `controller/scripts/emoji_gate.py` | | `Server.baseData` / `executeTemplate` | controller/internal/web/handlers.go + server.go | page-data plumbing | New pages | baseData injects nav/alerts/version; templates must pass `controller/scripts/template_id_gate.py` + `controller/scripts/emoji_gate.py` |
@@ -215,6 +251,7 @@
| `Server.sambaAddrFn` (func seam) | controller/internal/web/server.go (field) + sharing_handlers.go `sambaLANAddress()` | nil → `stackMgr.SambaLANAddress()` | The web-side half of the connect card. Tests inject a COUNTED fn — the fresh-per-render assertion is what stops anyone memoizing a DHCP lease | | `Server.sambaAddrFn` (func seam) | controller/internal/web/server.go (field) + sharing_handlers.go `sambaLANAddress()` | nil → `stackMgr.SambaLANAddress()` | The web-side half of the connect card. Tests inject a COUNTED fn — the fresh-per-render assertion is what stops anyone memoizing a DHCP lease |
| `Manager.guestNetExecFn` (func seam) + `GuestGateway()` / `GuestNetSnapshot()` | controller/internal/stacks/manager.go (field) + guestnet.go | nil → `docker exec felhom-samba <args>` — ONE seam for all R-66 guest-netns reads (route/link/addr/resolv.conf); tests script canned outputs per argv | guestnet_test.go. **The netns door rule:** the controller's OWN netns is the docker bridge, so any in-process read (`net.Interfaces`, `/proc/net/route`, its own `/etc/resolv.conf` = 127.0.0.11) is the S-2 wrong answer — guest-net reads MUST go through the samba (`network_mode: host`) exec door. Megosztás off ⇒ door closed ⇒ "" / per-item error strings; NEVER substitute an in-process value. Same S-5 law as SambaLANAddress: live per render, never cached/persisted. Parsers (`parseDefaultRoute`, `parseGuestInterfaces`, `parseResolvConf`) are pure + separately pinned | | `Manager.guestNetExecFn` (func seam) + `GuestGateway()` / `GuestNetSnapshot()` | controller/internal/stacks/manager.go (field) + guestnet.go | nil → `docker exec felhom-samba <args>` — ONE seam for all R-66 guest-netns reads (route/link/addr/resolv.conf); tests script canned outputs per argv | guestnet_test.go. **The netns door rule:** the controller's OWN netns is the docker bridge, so any in-process read (`net.Interfaces`, `/proc/net/route`, its own `/etc/resolv.conf` = 127.0.0.11) is the S-2 wrong answer — guest-net reads MUST go through the samba (`network_mode: host`) exec door. Megosztás off ⇒ door closed ⇒ "" / per-item error strings; NEVER substitute an in-process value. Same S-5 law as SambaLANAddress: live per render, never cached/persisted. Parsers (`parseDefaultRoute`, `parseGuestInterfaces`, `parseResolvConf`) are pure + separately pinned |
| `buildFileBrowserPaths` + `fbPathDeps` (R-67, v0.160.0) | controller/internal/web/handlers.go | pure assembly of one FileBrowser sync pass: (mount lines, config source paths) from the registry, with per-kind gates | filebrowser_network_test.go. **Two storage classes, two DIFFERENT gates:** drives keep the drive-absent gate + userdata scoping + skeleton (byte-identical to pre-R-67 — tested); network shares bind the share ROOT `:rslave` with the STUB gate instead (`classifyFSPath`; stub ⇒ excluded from mounts AND sources — an exposed stub swallows uploads the real mount later shadows; idle autofs / unknown ⇒ include, fail open). NEVER call `EnsureUserdataSkeleton` toward a network path (red-proven); never force-wake an idle trigger in the sync (doctrine) | | `buildFileBrowserPaths` + `fbPathDeps` (R-67, v0.160.0) | controller/internal/web/handlers.go | pure assembly of one FileBrowser sync pass: (mount lines, config source paths) from the registry, with per-kind gates | filebrowser_network_test.go. **Two storage classes, two DIFFERENT gates:** drives keep the drive-absent gate + userdata scoping + skeleton (byte-identical to pre-R-67 — tested); network shares bind the share ROOT `:rslave` with the STUB gate instead (`classifyFSPath`; stub ⇒ excluded from mounts AND sources — an exposed stub swallows uploads the real mount later shadows; idle autofs / unknown ⇒ include, fail open). NEVER call `EnsureUserdataSkeleton` toward a network path (red-proven); never force-wake an idle trigger in the sync (doctrine) |
| `Settings.RefuseAsAppNamespace` (R-108, v0.187.0) | controller/internal/settings/settings.go | `(path) (refuse bool, hungarianReason string)` — may an app's DATA NAMESPACE live here? | **THE single predicate for every placement surface** (deploy POST `api/router.go`, per-app migrate list + `handleStorageMigrateApp`, `handleStorageDecommission` mode=migrate TARGET). **Network storage is refused** because an app's namespace root IS its backup root (`namespaceRoot` returns a non-system drive path as-is → `<HDD_PATH>/backups/primary/<stack>/`), and on a share that lands inside FileBrowser's share-ROOT `download:true` bind — which CANNOT be narrowed (R-67 `:rslave` = automount wake; and apps on a share store at `<share>/<app>`, so there is no `userdata/` to scope to and creating one would write Felhom convention onto a customer's NAS). **DISTINCT from `refuseNetworkLifecycle`** — that asks "may a DRIVE lifecycle op run on this path" and is applied to the op's SUBJECT; this asks "may an app live here" and is applied to a placement TARGET. Migrate needs BOTH. **FAILS CLOSED:** `/mnt/felhom-drives` holds both kinds, so a path prefix cannot classify — `Kind` exists only on a REGISTERED path, therefore an unregistered path under that root is un-classifiable and REFUSES. Empty path = SSD-resident = allowed; nil receiver refuses. network_app_namespace_test.go, 4 red-proofs |
| `Server.guestGatewayFn` / `guestNetFn` (func seams) | controller/internal/web/server.go (fields) + sharing_handlers.go accessors | nil → `stackMgr.GuestGateway` / `stackMgr.GuestNetSnapshot` | network_card_test.go — the counted-fn freshness test (2 renders ⇒ 2 resolves) is what stops anyone memoizing a DHCP lease; the Hálózati név row is gated on `smb.Enabled` (red-proven: gate dropped ⇒ \\FELHOM rendered while samba is down) | | `Server.guestGatewayFn` / `guestNetFn` (func seams) | controller/internal/web/server.go (fields) + sharing_handlers.go accessors | nil → `stackMgr.GuestGateway` / `stackMgr.GuestNetSnapshot` | network_card_test.go — the counted-fn freshness test (2 renders ⇒ 2 resolves) is what stops anyone memoizing a DHCP lease; the Hálózati név row is gated on `smb.Enabled` (red-proven: gate dropped ⇒ \\FELHOM rendered while samba is down) |
| `sambaEnsureState.consumeIfRunning()` | controller/internal/web/samba_ensure_job.go | serve-once `snapshot()` for terminal `running` only | `/sharing/status` carries a job EDGE (`phase`) and a service LEVEL (`running`) in one envelope — never let a level reach the phase channel, and never re-serve a consumed edge: the client answers `phase=="running"` with `location.reload()`, so both mistakes produce an infinite page reload (S-1/S-4, DIAG-sharing-2026-07-20.md). `failed`/`needs_password`/in-flight are NOT consumed | | `sambaEnsureState.consumeIfRunning()` | controller/internal/web/samba_ensure_job.go | serve-once `snapshot()` for terminal `running` only | `/sharing/status` carries a job EDGE (`phase`) and a service LEVEL (`running`) in one envelope — never let a level reach the phase channel, and never re-serve a consumed edge: the client answers `phase=="running"` with `location.reload()`, so both mistakes produce an infinite page reload (S-1/S-4, DIAG-sharing-2026-07-20.md). `failed`/`needs_password`/in-flight are NOT consumed |
| `infra.SambaHostInterface` | controller/internal/infra/samba.go | the guest LAN nic name (`eth0`) | Single source for smb.conf's `interfaces =`, the container's `FELHOM_IFACE`, and the LAN-address read — if they name different nics, the service and the address the page prints drift apart | | `infra.SambaHostInterface` | controller/internal/infra/samba.go | the guest LAN nic name (`eth0`) | Single source for smb.conf's `interfaces =`, the container's `FELHOM_IFACE`, and the LAN-address read — if they name different nics, the service and the address the page prints drift apart |
@@ -230,6 +267,7 @@
| `Manager.execFn` (func seam) + `restartPolicyLookup` / `inspectRestartPolicyFn` (R-51, v0.156.0) | controller/internal/stacks/manager.go | nil → real `exec.Command` / `docker inspect -f {{.HostConfig.RestartPolicy.Name}}` | `scriptedDocker` in controller/internal/stacks/degraded_test.go drives the WHOLE production path (docker ps → aggregateState → docker inspect) — an aggregateState-only test proves the function, not the caller. Policy answers are cached per container+state and pruned to the live `docker ps` set; a FAILED inspect is deliberately never cached (a hiccup must not pin a container to "unknown") and reads as SUPERVISED, i.e. fail-closed — the opposite of `IsDownState`'s fail-open, because there the state is ambiguous while here a member is known dead | | `Manager.execFn` (func seam) + `restartPolicyLookup` / `inspectRestartPolicyFn` (R-51, v0.156.0) | controller/internal/stacks/manager.go | nil → real `exec.Command` / `docker inspect -f {{.HostConfig.RestartPolicy.Name}}` | `scriptedDocker` in controller/internal/stacks/degraded_test.go drives the WHOLE production path (docker ps → aggregateState → docker inspect) — an aggregateState-only test proves the function, not the caller. Policy answers are cached per container+state and pruned to the live `docker ps` set; a FAILED inspect is deliberately never cached (a hiccup must not pin a container to "unknown") and reads as SUPERVISED, i.e. fail-closed — the opposite of `IsDownState`'s fail-open, because there the state is ambiguous while here a member is known dead |
| `bootrecon.StackProvider` (R-52, v0.156.0) | controller/internal/bootrecon/bootrecon.go | `*stacks.Manager` (GetStacks/StartStack/RefreshStatus) | `fakeStacks` counts StartStack per app; the load-bearing assertion is the NEGATIVE — a zero-container stack (a UI Stop = `compose down` = containers removed) must record **0** starts, while a boot orphan (containers present, Exited) records exactly 1. `Reconciler.sleep` is injected so the 30 s gap costs nothing | | `bootrecon.StackProvider` (R-52, v0.156.0) | controller/internal/bootrecon/bootrecon.go | `*stacks.Manager` (GetStacks/StartStack/RefreshStatus) | `fakeStacks` counts StartStack per app; the load-bearing assertion is the NEGATIVE — a zero-container stack (a UI Stop = `compose down` = containers removed) must record **0** starts, while a boot orphan (containers present, Exited) records exactly 1. `Reconciler.sleep` is injected so the 30 s gap costs nothing |
| `bootReconcileFn` + `runBootReconcile` (package-main seam, v0.156.0) | controller/cmd/controller/main.go | `bootrecon.New(mgr, logger).Run` | controller/cmd/controller/bootrecon_wiring_test.go. **The wiring itself is asserted by an AST walk** over `func main()`, not a `strings.Contains` — the substring version passed its own red-proof because a commented-out call still contains the string. Comments are not callers | | `bootReconcileFn` + `runBootReconcile` (package-main seam, v0.156.0) | controller/cmd/controller/main.go | `bootrecon.New(mgr, logger).Run` | controller/cmd/controller/bootrecon_wiring_test.go. **The wiring itself is asserted by an AST walk** over `func main()`, not a `strings.Contains` — the substring version passed its own red-proof because a commented-out call still contains the string. Comments are not callers |
| `classifyRunStates` (pure fix-3 derivation, v0.164.0) | controller/cmd/controller/main.go | `([]stacks.Stack, quiesced, failedRestart map[string]bool, now time.Time)``(dead []web.DeadApp, states []notify.AppRunState)` | classify_runstates_test.go. **THE single fix-3 rule: down = `(IsDownState(st.State) || st.CrashLooping(now)) && !userStopped && !quiesced`.** C9-F2 (v0.183.0) added the crash-loop term: `restarting` is NOT in `IsDownState` and must not be — adding it alarms on every deploy and update fleet-wide — so a SUSTAINED restarting run (`stacks.crashLoopAfter` = 5 m, above the 120 s deploy timeout, Mealie's 60 s start_period AND R-97b's 180 s grace) becomes down instead. `now` is injected so the threshold is a testable contract. A deliberate UI stop (`compose down` → zero containers → StateStopped, I1) must not alarm — banner OR email — while faults (Exited/Degraded) alarm byte-identically; I2 (P2 census: all catalog services `unless-stopped`) is why a crash never rests at stopped. **Do NOT touch `IsDownState`** (other callers rely on stopped=down) and do NOT filter in `buildDeadAppAlerts`/`NotifyAppStartFailures` — one derivation point. If I1 or I2 changes, revisit the suppression |
| `report.SetPendingControllerLog` / `SetControllerLogSource` | controller/internal/report/selftail.go | ACK-armed consume-once self-log pull (the logtail.go shape) | selftail_test.go; source = `logBuffer.Lines`, wired once in main.go | | `report.SetPendingControllerLog` / `SetControllerLogSource` | controller/internal/report/selftail.go | ACK-armed consume-once self-log pull (the logtail.go shape) | selftail_test.go; source = `logBuffer.Lines`, wired once in main.go |
| `util.ParseVersion` / `util.Version.Compare` | controller/internal/util/version.go | THE one semver comparator (house rule: never a second) — selfupdate aliases it; agentapi's MinAgent comparison uses it | rejects pre-release/dev/latest (callers fall back, never trust); numeric compare (0.100 > 0.81) | | `util.ParseVersion` / `util.Version.Compare` | controller/internal/util/version.go | THE one semver comparator (house rule: never a second) — selfupdate aliases it; agentapi's MinAgent comparison uses it | rejects pre-release/dev/latest (callers fall back, never trust); numeric compare (0.100 > 0.81) |
| `agentapi.AgentVersionReporter` + `featureMinAgent` | controller/internal/agentapi/features.go | version-first Supports (v0.82.0 header channel); probe = fallback for header-less agents | a coupled feature adds BOTH a featureProbes row AND a featureMinAgent row; v0.116.0: `SupportsWithSource` also reports HOW the verdict was reached (version/probe-cache/probe) for the gate log line | | `agentapi.AgentVersionReporter` + `featureMinAgent` | controller/internal/agentapi/features.go | version-first Supports (v0.82.0 header channel); probe = fallback for header-less agents | a coupled feature adds BOTH a featureProbes row AND a featureMinAgent row; v0.116.0: `SupportsWithSource` also reports HOW the verdict was reached (version/probe-cache/probe) for the gate log line |
@@ -283,7 +321,7 @@ Cross-repo edges:
| dir-size ×6 | controller/internal/stacks/delete.go `getDirSizeBytes`/`getDirSizeHuman`; controller/internal/backup/tier2.go `dirSizeBytes` (du -sb); controller/internal/appexport/estimate.go `dirSize`+`duBytes`; controller/internal/appexport/export.go `calcDirSize`; controller/internal/web/handlers.go `dirSizeHuman` | | dir-size ×6 | controller/internal/stacks/delete.go `getDirSizeBytes`/`getDirSizeHuman`; controller/internal/backup/tier2.go `dirSizeBytes` (du -sb); controller/internal/appexport/estimate.go `dirSize`+`duBytes`; controller/internal/appexport/export.go `calcDirSize`; controller/internal/web/handlers.go `dirSizeHuman` |
| timeAgo switch body ×2 | controller/internal/web/funcmap.go `timeAgo` vs `timeAgoStr` (identical formatting logic) | | timeAgo switch body ×2 | controller/internal/web/funcmap.go `timeAgo` vs `timeAgoStr` (identical formatting logic) |
| CSRF ×2 | controller/internal/web/csrf.go (session HMAC) vs controller/internal/setup/csrf.go (cookie double-submit) — intentional (pre-auth wizard) but unlabeled | | CSRF ×2 | controller/internal/web/csrf.go (session HMAC) vs controller/internal/setup/csrf.go (cookie double-submit) — intentional (pre-auth wizard) but unlabeled |
| Budapest timezone loader ×2 | controller/internal/scheduler/scheduler.go `getBudapestLocation` vs controller/internal/web/funcmap.go `getTimezone` | | Budapest timezone loader ×3 | controller/internal/scheduler/scheduler.go `getBudapestLocation` vs controller/internal/web/funcmap.go `getTimezone` vs controller/internal/quiesce/quiesce.go `budapestLocation` (v0.168.0 window gate — Budapest wall-clock, kept local to avoid a scheduler↔quiesce import edge) |
| JSON writers ×5, 3 envelope shapes | api `writeJSON`; web `writeDiskJSON`, `jsonResponse`/`jsonError`, `writeDebugJSON` | | JSON writers ×5, 3 envelope shapes | api `writeJSON`; web `writeDiskJSON`, `jsonResponse`/`jsonError`, `writeDebugJSON` |
| Safe-name validators ×4 | controller/internal/web/validate.go `validStackName`; controller/internal/api/router.go `validStackParam` (same body — api↔web import cycle); controller/internal/backup/offbox.go `isSafeStackName`; controller/internal/appexport/validate.go `ValidateSegment` (strictest) | | Safe-name validators ×4 | controller/internal/web/validate.go `validStackName`; controller/internal/api/router.go `validStackParam` (same body — api↔web import cycle); controller/internal/backup/offbox.go `isSafeStackName`; controller/internal/appexport/validate.go `ValidateSegment` (strictest) |
| DB wait/import ×2 | controller/internal/appbackup/dbdump.go `waitDBReady`/`ImportDump` vs controller/internal/appexport/restore.go `waitForDB`/`importDBDump` | | DB wait/import ×2 | controller/internal/appbackup/dbdump.go `waitDBReady`/`ImportDump` vs controller/internal/appexport/restore.go `waitForDB`/`importDBDump` |
+505 -51
View File
@@ -154,14 +154,79 @@ backups, monitoring and notifications. All Proxmox/disk operations are delegated
action block right; used by the dashboard installed-apps list, the Távoli mentés toggle list action block right; used by the dashboard installed-apps list, the Távoli mentés toggle list
and the Visszaállítás restore-to-verify/.fab lists; the backups-apps expander header is and the Visszaállítás restore-to-verify/.fab lists; the backups-apps expander header is
ALIGNED to the same grammar (own markup — it carries the toggle). Protected infra stacks ALIGNED to the same grammar (own markup — it carries the toggle). Protected infra stacks
**The off-site restore list is keyed on the STORE (v0.204.0, R-237):** `offsite_restore_list.go`
builds it from `OffsiteInventoryList` (the repository's own snapshot tags), NOT from deployed +
offsite-toggled apps. A rebuilt box has neither and used to be shown nothing to restore while its
snapshots sat in the repository. Installed-ness is a property OF a row (it changes what restoring
implies), never a filter on it; an unreadable store renders as UNKNOWN and keeps the action; the
`felhom-offbox` and `_shares` marker tags are never offered as apps.
**Its two preconditions now name a reason AND a route (v0.207.0, R-252/R-253):** a rebuilt box keeps
its drives but loses their REGISTRATION, so the page renders a notice — *„Előbb csatold vissza az
adatmeghajtót"*, linking to `/storage` — whenever `HasRestoreDestination()` is false, asked through
the backup manager's own predicate so page and resolver read the same `GetSchedulableStoragePaths()`.
And the not-installed row no longer promises *„a visszaállítás előbb újratelepíti"*: reconstitution
writes to the app's own `GetStackHDDPath`, which exists only after the customer picks a drive at
deploy time, so the copy says to install it first and links to `/stacks/<app>/deploy`. **Both notices
are conditional** — a healthy box renders exactly as before, pinned by a test that fails if either
becomes unconditional.
**A run that skipped a selected app is `incomplete` (v0.205.0, R-234):** the off-site verdict now
counts `missingUnprotected` beside `mandatoryGaps` — an app the customer selected that is DEPLOYED
but has no recovery unit is not protected, so the run is not `ok`. A selected-but-UNDEPLOYED app is
named with what to do and does NOT move the verdict (a permanently amber box is a status nobody
reads); a disconnected/decommissioned drive has its own signal. The manual „Távoli mentés most”
also refuses SYNCHRONOUSLY when a run is already in flight, instead of answering „elindult” and
leaving the previous run's verdict on the card.
**The box does not mint a repository key over a sealed package (v0.206.0, R-241):**
`WriteOffboxSecrets` auto-generated on ONE input — does the password file exist — while its two
neighbours in the same file both consulted `GetHubEscrowIdentityPresent()`. A rebuilt box's
credential self-heal therefore wrote a fresh key over the package the hub was holding for it, and
the recovery screen then correctly reported that nothing was recoverable under the key the box
held. The guard is a CONJUNCTION (a package held AND no key present), so a first-time box mints
exactly as before; the refusal is a HOLDING state that still writes the transport and declares
`offsite.state=awaiting_recovery_key`, so `/recovery` can bring the tier up the moment the key
arrives. `--abandon-status` / `--abandon-extend=N` / `--abandon-stop` are the operator levers.
**The recovery offer has a third shape (v0.206.0, R-241):** shape (c) — *the hub holds a package
for a key OTHER than the one we are using*. Shapes (a) "no key at all" and (b) "a run proved the
repo will not open" are proxies for that question and have each been wrong in opposite directions.
The comparison was already computed on every report ACK and persisted nowhere; it now lives in
`settings.HubEscrowKeySHA256`. A KNOWN difference offers however old the reading; a hash never
learned falls back to (a)/(b).
**Abandoning the old history is a finishable thing (v0.206.0, R-241):** the customer's confirmed
set-aside starts a **14-day countdown**, visible on Távoli mentés and reversible by recovering with
the code, at the end of which the set-aside store AND the hub's retained sealed package are removed
TOGETHER (two-phase commit; the hub half is `PurgeSupersededEscrowForCustomer`, hub v0.98.0).
Afterwards shape (c) has nothing to compare and the offer ends because the state is right. The
recovery screen appears **once per ENTRY into the offered state** (an epoch), the reminder bar is
per-visit, the durable opt-out silences the bar only, and **none of the three removes the entry
point on Távoli mentés**. While a recovery is outstanding, „Helyreállítási kód létrehozása" is
UNAVAILABLE — creating a new code would put the earlier history beyond every shipped path.
**A code for an EARLIER package is now recognised as correct (v0.214.0, R-311; MinAgent 0.129.0).**
When the supplied code does not open the package the hub currently holds, the agent tries the
RETAINED ones and answers 422 if one opens; the screen then says the code is *correct*, names the
supersession date, says the earlier package is kept and that the CURRENT backups are unaffected, and
routes to support. It deliberately promises **no restore** — there is no in-product route to a
set-aside store (R-312), and the retained package may itself predate the repository-password field.
On an agent older than 0.129.0 the screen keeps the earlier hedged wording, which was honest: until
something looked, a correct-but-earlier code and a mistype really were indistinguishable.
(traefik/cloudflared/filebrowser) get curated Hungarian display identity from the (traefik/cloudflared/filebrowser) get curated Hungarian display identity from the
`inframeta.go` map (name + description + generic `/static/infra-logo.svg` fallback icon); `inframeta.go` map (name + description + generic `/static/infra-logo.svg` fallback icon);
filebrowser is the only infra stack with a customer link (`files.<domain>`). filebrowser is the only infra stack with a customer link (`files.<domain>`).
**Enforcement:** `scripts/template_id_gate.py` (JS element-ID integrity) + `scripts/emoji_gate.py` **Universal app placeholder (v0.163.0):** `app_list_row` now DEFAULTS its fallback icon to the
(no emoji) + `scripts/native_confirm_gate.py` (zero native confirm/prompt) + embedded `AppPlaceholderSVG` (a 2×2 app-grid glyph, served at `/static/app-placeholder.svg`), so a
`scripts/app_row_dedup_gate.py` (row markup single-sourced) + `scripts/mojibake_gate.py` logo-less app shows a placeholder on every list surface instead of a hidden icon; infra rows still
(no double-encoded UTF-8 in templates/Go sources) — run after any template change; the Go override with the server glyph. The felhom brand mark is never an app placeholder (brand = platform
`TestNoEmojiInTemplates` mirrors the emoji gate. identity only).
**Enforcement — one entry point (2026-08-02):** run `python3 scripts/controller_gates.py` from
`controller/` after any template change. It is THE runner and invokes every gate:
`template_id_gate.py` (JS element-ID integrity), `emoji_gate.py` (no emoji),
`native_confirm_gate.py` (zero native confirm/prompt), `app_row_dedup_gate.py` (row markup
single-sourced), `mojibake_gate.py` (no double-encoded UTF-8 in templates/Go sources),
`docker_run_volume_path_gate.py` (every `docker … -v` mount reviewed), and `reuse_refs_check.py`
on the repo root. It exits non-zero if any gate does, and a missing gate script is a FAILURE, not
a skip. `--fast` (what `.githooks/pre-push` runs) selects the gates that touch no network and no
container runtime — today all of them. The Go `TestNoEmojiInTemplates` mirrors the emoji gate.
Why a runner: of this project's gates, only the ones named by a `CLAUDE.md` entry point ever got
run — the 2026-08-02 census found the two unnamed ones red, one for nineteen days.
- **Europe/Budapest timezone** — All scheduled jobs, timestamps, and UI labels use Hungarian timezone. - **Europe/Budapest timezone** — All scheduled jobs, timestamps, and UI labels use Hungarian timezone.
### Module Map ### Module Map
@@ -180,7 +245,8 @@ backups, monitoring and notifications. All Proxmox/disk operations are delegated
| **System** | `internal/system/` | System info (`/proc`), CPU collector, mount points, disk usage, FS info | | **System** | `internal/system/` | System info (`/proc`), CPU collector, mount points, disk usage, FS info |
| **Monitor** | `internal/monitor/` | System health checks, storage watchdog, legacy Healthchecks pinger (deprecated) | | **Monitor** | `internal/monitor/` | System health checks, storage watchdog, legacy Healthchecks pinger (deprecated) |
| **Metrics** | `internal/metrics/` | SQLite time-series store, system + container metric collection | | **Metrics** | `internal/metrics/` | SQLite time-series store, system + container metric collection |
| **Scheduler** | `internal/scheduler/` | Central job scheduler (periodic + daily, skip-if-running, panic recovery) | | **Scheduler** | `internal/scheduler/` | Central job scheduler (periodic + daily, skip-if-running, panic recovery). `UpdateDaily` reschedules a daily job at runtime (no restart) via a per-job reschedule signal (v0.168.0). |
| **Backupwindow** | `internal/backupwindow/` | Pure time math for the customer-configurable backup window (v0.168.0): `ParseHHMM`/`FmtHHMM`, `LegTimes` (W / W+60m / W+105m, wrap-safe), `GateWindow` ([W+2h, W+6h)), `EffectiveWindow` (settings > yaml > "02:30"). Offsets are constants — derived, never stored. |
| **SelfUpdate** | `internal/selfupdate/` | Version checking (registry), update trigger, state persistence, startup verification | | **SelfUpdate** | `internal/selfupdate/` | Version checking (registry), update trigger, state persistence, startup verification |
| **Notify** | `internal/notify/` | Email notifications via hub relay, preference sync, per-event cooldowns | | **Notify** | `internal/notify/` | Email notifications via hub relay, preference sync, per-event cooldowns |
| **Report** | `internal/report/` | Hub report builder + HTTP pusher (system, stacks, backup, health) | | **Report** | `internal/report/` | Hub report builder + HTTP pusher (system, stacks, backup, health) |
@@ -195,6 +261,52 @@ backups, monitoring and notifications. All Proxmox/disk operations are delegated
## Features ## Features
### Disk health — "Lemezek állapota" card + degradation alert (v0.169.0; severity ladder v0.215.0)
The dashboard shows a per-physical-disk health card driven by the agent's SMART summary (serialized
into `/disks` from agent v0.94.0 — no new smartctl load; the controller only reads it). One pure
verdict function (`agentapi.DiskVerdictFor`) is the shared truth for the card chip and the check.
**Why it does not trust `smart_status.passed`:** attributes 187/197/198 all carry `thresh: 0` and a
normalized SMART value floors at 1, so a drive's own overall verdict is **structurally incapable** of
failing on unreadable sectors — the real failing drive read `PASSED` at 352 pending sectors. The ladder
reads the raw counters instead. Evidence:
`felhom.eu/documentation/audits/DIAG-smart-passed-trap-2026-08-14.md`.
**Four labels, and there will not be a fifth** — a predicted failure is „Hiba", the same word a
self-reported failure gets:
- **Rendben** — clean.
- **Figyelmeztetés** — a first sighting of unreadable sectors below the bar, reallocated sectors, NVMe
media errors, `percentage_used ≥ 90`, or `temperature_c ≥ 55`.
- **Hiba** — the drive reports FAILING, **or** (v0.215.0) unreadable sectors are **sustained** across
two consecutive checks, unreadable sectors sit alongside reallocation, the count reaches **64**,
`temperature_c ≥ 60`, NVMe's `critical_warning` is set, or `percentage_used ≥ 100`.
- **Nincs adat** — nil/UNKNOWN/old agent. Never alarms, and never erases a disk's stored history.
`DiskVerdictFor(smart, prior)` takes an `agentapi.DiskPrior`; a zero prior is the fail-safe, so a first
sighting can only reach Figyelmeztetés. The full 14-row truth table, evaluated top-down, is documented
in `internal/agentapi/diskverdict.go` together with the provenance of 64 / 55 / 60.
- The card fetches `/disks` through a **60 s TTL cache** (dashboard refresh-spam can't smartctl-storm
the host); an unreachable agent renders "Nincs adat" and the page still loads. The card replays the
prior that produced the stored verdict (`diskRecord.PriorSawUncorrectable`) so the chip and the email
cannot disagree.
- An **hourly `disk-health-check`** (v0.215.0; was 6-hourly — the real drive's benign excursion lasted
about one hour, and `/disks` measures ~0.82 s) emits `disk_health_degraded` against **persisted**
per-disk state in `disk-health-state.json` under `cfg.Paths.DataDir`. It fires on an escalation
against the **last alerted** verdict, and re-alerts a disk already at Hiba once it has **both**
doubled its unreadable-sector count **and** waited out a 24 h cooldown. A disk's first verdict
baselines silently; recovery and UNKNOWN never notify; a disappeared disk is forgotten. A restart no
longer re-baselines — that is why the state is persisted. Every cycle logs a positive observable
(`disk-health check complete: N disk(s) evaluated, M alert(s)`).
- **Severity is a wire contract:** the alert emits `warning` / `critical` — the hub coerces anything
outside `{info, warning, error, critical}` to `info` and never emails it. `DiskAlertKind.Severity()`
is the single mapping. The hub allowlist must include `disk_health_degraded`.
- Five message shapes (`notify.DiskAlert`), because the customer's action differs: Figyelmeztetés,
drive-reported FAILING, Hiba-from-sector-count, Hiba-from-heat, and the still-worsening re-alert.
- **No global banner** (deliberate): the card + email carry it.
### 1. App Management ### 1. App Management
The controller manages Docker Compose stacks through a complete lifecycle: catalog sync, first-time deployment, runtime operations, and deletion. The controller manages Docker Compose stacks through a complete lifecycle: catalog sync, first-time deployment, runtime operations, and deletion.
@@ -256,6 +368,65 @@ When a user visits a stopped or undeployed app's subdomain (e.g., `travel.demo-f
- **`catchall.html`** — standalone template (no layout, inline CSS) showing the app name, status ("leállítva" / "nincs telepítve" / "nem található"), and links to the controller dashboard or the app's detail page. - **`catchall.html`** — standalone template (no layout, inline CSS) showing the app name, status ("leállítva" / "nincs telepítve" / "nem található"), and links to the controller dashboard or the app's detail page.
- **Subdomain links** on the Alkalmazások page are only shown for deployed apps (non-deployed apps have no guaranteed subdomain yet). - **Subdomain links** on the Alkalmazások page are only shown for deployed apps (non-deployed apps have no guaranteed subdomain yet).
#### Indítópult (app launcher page) (v0.163.0)
`/launcher` (`launcherHandler` + `templates/launcher.html`) is the **first** sidebar item, above
Vezérlőpult; `/` still lands on the Vezérlőpult. It renders a grid of large tappable tiles — one per
**openable** deployed app. Openability has a single criterion, shared with the „Megnyitás" button: the
stack has a subdomain (env `SUBDOMAIN` > `.felhom.yml` subdomain > `protectedStackSubdomains`),
resolved through the extracted `Server.subdomainMap` helper (the dashboard and Alkalmazások pages use
the same helper). The controller's own stack is excluded by name.
Each tile is a colored rounded square: `funcmap.tileColor(slug, brand)` returns a validated
`.felhom.yml` **`brand_color`** (`#rgb`/`#rrggbb`, `Metadata.BrandColor`) or, when absent/invalid, a
deterministic FNV-1a-of-slug → HSL color (fixed S/L, hue varies per app). The white monochrome logo
renders on top of a **monogram** initial (`funcmap.initial`, multibyte-safe); if the logo fails to
load the monogram shows through (the launcher does NOT use the app-placeholder here). Operational apps
are `<a target="_blank" rel="noopener">` links; stopped/degraded apps render greyed + unclickable with
the Hungarian state badge. Empty state links to `/stacks`.
#### Indítópult megosztása — guest launcher via capability URL (v0.165.0)
The admin launcher's **"Indítópult megosztása"** button mints a **capability URL**
`https://<host>/s/<token>`, where `token` is a 160-bit `crypto/rand` value
(`newShareToken`, base64.RawURLEncoding, 27 chars) — that serves a **standalone, read-only guest
launcher** with **no account and no admin session**. The link grants **information only, zero
control**: app names + public URLs; every privilege stays behind each app's own auth and the
controller admin password. The tile visual is shared with the admin launcher via the `launch_tile`
template partial; the app slice comes from the extracted `Server.launcherApps()` helper.
- **Routing** (`internal/web/share.go`, `share_handlers.go`): `/s/<token>` joins the RequireAuth
pre-auth allowlist **after** the claim-gate block (an unclaimed box never serves the guest page —
the claim gate stays supreme) and is exempted from session CSRF (the guest password POST carries a
pre-auth HMAC CSRF, `validShareCSRF`, mirroring the claim POST). Token match is
`subtle.ConstantTimeCompare`; an empty stored token (= sharing OFF, there is no separate flag)
matches nothing, so a wrong/disabled token returns a **byte-identical mux-default 404** (`share404`).
Guest responses set `X-Robots-Tag: noindex, nofollow` / `Referrer-Policy: no-referrer` /
`Cache-Control: no-store`. The token is a secret: the ServeHTTP debug line and the 404 WARN redact
`/s/` paths to `/s/<redacted>`.
- **Optional per-share password** (`settings.LauncherSharePasswordHash`): a SEPARATE bcrypt credential
(never the admin `PasswordHash`), guarded by its OWN per-IP 5/1-min attempt map (`shareAttempts`,
never the admin `loginAttempts`). A correct password mints a signed gate cookie =
HMAC-SHA256(`token|passwordHash`) keyed with the persisted, box-scoped `web.session_secret` — so
rotating the token OR changing the password invalidates every outstanding cookie with no bookkeeping.
- **Guest state labels** ride the v0.164.0 ruling and never expose internal vocabulary: clickable ⇔
`isOperationalState && !routeUnpublished` (operational AND route actually published, so a tap never
dead-ends); `StateStopped` ⇒ "A tulajdonos leállította"; any other non-clickable state ⇒
"Átmenetileg nem elérhető". Empty ⇒ "Jelenleg nincs elérhető alkalmazás." (`buildGuestApps` is the
pure, tested mapping; templates `launcher_shared.html` + `launcher_share_password.html`).
- **Admin modal** (in `launcher.html`): current link + copy button, QR code
(`GET /launcher/share/qr.png`, ~256px PNG via `github.com/skip2/go-qrcode`, admin-authed, `no-store`),
set/clear share password, "Új link készítése" (rotate), "Megosztás kikapcsolása" (clears token AND
password). The management POSTs live under `/launcher/share/*` and ride the normal admin session +
session CSRF; rotate/disable use the inline `data-confirm` (felhomConfirm) affordance. A
feature-detected **"Megosztás…"** button (v0.165.1) opens the OS share sheet via `navigator.share`
(title + text + URL only — never the QR as a `files:` attachment); hidden unless the browser
supports it, with "Link másolása" as the universal fallback (the non-cancel rejection path falls
back to it too).
Design ruling: member accounts are superseded by this capability-URL model; per-member tile
visibility is parked under the SSO arc.
#### Dashboard "Megnyitás" Button #### Dashboard "Megnyitás" Button
Running apps on the Vezérlőpult now show a "Megnyitás ↗" button that opens the app's subdomain in a new tab. The `Subdomains` map is built in `dashboardHandler` from `app.yaml` env or metadata fallback. Running apps on the Vezérlőpult now show a "Megnyitás ↗" button that opens the app's subdomain in a new tab. The `Subdomains` map is built in `dashboardHandler` from `app.yaml` env or metadata fallback.
@@ -312,8 +483,18 @@ Each app can define rich metadata in `.felhom.yml`:
surfaces it on `/apps/{slug}` as a "Kezdeti belépési adatok" card (masked password + reveal/copy), surfaces it on `/apps/{slug}` as a "Kezdeti belépési adatok" card (masked password + reveal/copy),
labelled as the *initial* credential (stays valid only until the customer changes it in-app). Hidden labelled as the *initial* credential (stays valid only until the customer changes it in-app). Hidden
when the container is down / file missing / parse fails. Reuse for any future self-seeding app. when the container is down / file missing / parse fails. Reuse for any future self-seeding app.
- `backup` (v0.132.0): the **referential-coupling classification** block (Task 2). Two optional lists, - `data_paths` (v0.172.0, R-75): the **customer-facing folder annotation**`{path, root, role, label}`
`userdata:` (relative to `${USERDATA_PATH}`) and `hdd:` (relative to `${HDD_PATH}`), each of where `root ∈ {import, userdata, hdd}` and `role ∈ {import, library, export}`. It ANNOTATES paths that
must already exist as compose binds and can never declare one (so no new filesystem-write primitive
comes from catalog data). Validation is deliberately ASYMMETRIC: a malformed **path** is a whole-block
reject (reusing `appbackup.ValidateRelPath`, the same refusal set as `backup:`), an unknown **role**
fails OPEN with one WARN (the `Lifecycle` precedent — presentation, not data handling). Rendered on
`/apps/{slug}` as „Hova tegyem a fájlokat?" for DEPLOYED apps only, each row a FileBrowser deep link
plus a **class-driven** consequence line. Full contract:
`felhom.eu/documentation/controller/import-and-data-paths.md`.
- `backup` (v0.132.0; **three** lists since v0.172.0): the **referential-coupling classification** block
(Task 2). Optional lists `userdata:` (relative to `${USERDATA_PATH}`), `hdd:` (relative to
`${HDD_PATH}`) and `import:` (relative to `${IMPORT_PATH}`, R-75), each of
`{path, class}` where `class ∈ {mandatory, optional, excluded}` (COUPLED / DECOUPLED-precious / `{path, class}` where `class ∈ {mandatory, optional, excluded}` (COUPLED / DECOUPLED-precious /
DECOUPLED-bulk). `LoadMetadata` validates the block against the app's compose binds and **rejects the DECOUPLED-bulk). `LoadMetadata` validates the block against the app's compose binds and **rejects the
WHOLE block** (→ nil + one `[ERROR]`, app behaves as legacy) on any defect. Semantics WHOLE block** (→ nil + one `[ERROR]`, app behaves as legacy) on any defect. Semantics
@@ -359,6 +540,17 @@ Each app can define rich metadata in `.felhom.yml`:
+ `dumps_at`; a manifest without them is a pre-v0.148 pair of unknown skew, surfaced at restore + `dumps_at`; a manifest without them is a pre-v0.148 pair of unknown skew, surfaced at restore
time. The periodic refresh carries the prior stamp forward and never invents one. A dump-leg time. The periodic refresh carries the prior stamp forward and never invents one. A dump-leg
failure is a loud WARN that does NOT abort the push (data-first: a degraded backup beats none). failure is a loud WARN that does NOT abort the push (data-first: a degraded backup beats none).
- **A scratch restore NAMES ITS SCOPE (v0.198.0, R-204 item 3 — `restoreScratchOutcomeMsg`).**
`mode=unit` (the default) restores the recovery unit only — the app's definition, configuration
and DB dumps — because `RestoreOffboxScratch` passes `--include <unit path>`; the userdata that is
in the SAME snapshot is excluded by it. The outcome flash used to be one sentence for both modes
and named neither scope, so on the last step of a disaster recovery the customer was told
„visszaállítva" after the thing they were looking for had not been. The unit outcome now states
what came back, that the customer's own files did NOT, and the step that gets them; the full
outcome states that the files came with it (an absence is not a statement). The wizard's intent
card 1 carries the same scope BEFORE the choice. **The `mode=full` two-step size gate is
untouched**, and the default stays `unit` — all three wizard forms set `mode` explicitly, so
changing it would alter nothing the customer sees while silently changing a mode-less POST.
- **Offsite reconstitution (v0.148.0, R-43 — `offbox_reconstitute.go`):** the leg that was missing. - **Offsite reconstitution (v0.148.0, R-43 — `offbox_reconstitute.go`):** the leg that was missing.
`ReconstituteFromOffsite` (`/backup/offbox/reconstitute`, „Teljes visszaállítás (fájlok + `ReconstituteFromOffsite` (`/backup/offbox/reconstitute`, „Teljes visszaállítás (fájlok +
adatbázis)") makes the live app equal to the chosen snapshot: **safety dump → stop → files adatbázis)") makes the live app equal to the chosen snapshot: **safety dump → stop → files
@@ -420,6 +612,16 @@ The `/apps/{slug}` page renders hero section, screenshots, setup guide, and opti
#### Base-infrastructure bring-up (`stacks/infra.go` + `internal/infra/`, v0.41.0) #### Base-infrastructure bring-up (`stacks/infra.go` + `internal/infra/`, v0.41.0)
**Canonical import root (v0.172.0, R-75).** `${IMPORT_PATH}` = `<system namespace root>/userdata/import`
— ONE drop-zone per box, on the always-available system drive, never per data drive (each drop-zone app
has exactly one ingest bind, so a per-drive `import/` would put a dead lookalike on every other drive,
and `import/*` is `class: excluded` so files stranded there are unbacked too). Injected at BOTH
compose-env builders; **no per-drive fallback** — unresolvable leaves it unset so compose fails loudly.
The system drive is deliberately NOT a registered `StoragePath`, so the FileBrowser bind
(`/srv/beolvasas`, sidebar „Beolvasás"), the skeleton and the system-owned `beolvasas` SMB share each
reach it explicitly. The **userdata skeleton is catalog-derived** (`DeriveUserdataDirs` +
`UserdataSkeletonCarry`, sorted — the sort is load-bearing, see REUSE.md) and can only ever ADD.
The controller stands up its own base stack — **traefik** (reverse proxy), **cloudflared** (external tunnel), **filebrowser** — instead of relying on the bare-metal `scripts/docker-setup.sh` (which a Proxmox-provisioned guest never runs). `internal/infra` renders the compose + config files from `controller.yaml` via embedded `text/template`s (lifted from `docker-setup.sh`); image tags are **pinned constants there** (`TraefikImage`/`CloudflaredImage`/`FileBrowserImage`) and the web FileBrowser sync path delegates to the same renderers, so the pinned versions can never diverge. The controller stands up its own base stack — **traefik** (reverse proxy), **cloudflared** (external tunnel), **filebrowser** — instead of relying on the bare-metal `scripts/docker-setup.sh` (which a Proxmox-provisioned guest never runs). `internal/infra` renders the compose + config files from `controller.yaml` via embedded `text/template`s (lifted from `docker-setup.sh`); image tags are **pinned constants there** (`TraefikImage`/`CloudflaredImage`/`FileBrowserImage`) and the web FileBrowser sync path delegates to the same renderers, so the pinned versions can never diverge.
`Manager.EnsureBaseStack()` creates the `traefik-public` network, then deploys traefik → cloudflared → filebrowser under `${stacks_dir}/<name>`. It is: `Manager.EnsureBaseStack()` creates the `traefik-public` network, then deploys traefik → cloudflared → filebrowser under `${stacks_dir}/<name>`. It is:
@@ -516,6 +718,27 @@ Per-app export creates a self-contained `.fab` file (tar.gz, optionally encrypte
The backup system implements a **3-2-1 backup architecture**. Each tier is a **complete, The backup system implements a **3-2-1 backup architecture**. Each tier is a **complete,
self-sufficient backup** — any single tier can fully restore an app. self-sufficient backup** — any single tier can fully restore an app.
**The reserve — per-app backup admission (v0.192.0 decision B2, widened by v0.193.0 / R-181).**
`internal/backup/admission.go`. Since the `mp1``mp0` merge (R-165) local backups and Docker's
data-root share one filesystem, so an unbounded backup write is a stopped box rather than a slow one.
Before **any** of an app's three local write legs runs — DB dump, volume dump, recovery-unit capture —
`admitApp` takes **one verdict for that app for that run** and the other two legs reuse it. A refused
app writes nothing at all, is **not stopped**, keeps its previous unit byte-identical, and produces
**exactly one** operator alert (`recovery_unit_capture_failed`, operator-tier).
- **The verdict is lazy, not run-wide.** It is taken at the app's first write, because app A's dump
can put app B under the reserve; a verdict taken at run start would read a disk that no longer
exists by the time B writes.
- **It is never re-decided between an app's own legs**, and the memo is reset per run.
- **Two questions, both against two thresholds (97% used / 1 GiB free).** *Headroom*: is the
filesystem already below the reserve? *Size*: would this app's own write take it below? The size
estimate is the app's **previous** `.sql` + `.tar` already on disk. **No history → headroom-only**,
deliberately — otherwise the first backup is the one that can never happen — and the alert says so.
- **The thresholds sit beyond `fillwatch`'s critical band (95% / 2 GiB)**, so the customer is always
warned before a refusal is possible.
- **It refuses; it never deletes.** Nothing here is generational — one unit per app at one fixed path
— so "prune the oldest" could only destroy a different app's only local copy.
**Sidebar behaviour (v0.146.0).** Groups that own sub-pages — Tárhely, Biztonsági mentés, **Sidebar behaviour (v0.146.0).** Groups that own sub-pages — Tárhely, Biztonsági mentés,
Megosztás — render as **accordions**: the header is a real `<button class="nav-group-toggle">` Megosztás — render as **accordions**: the header is a real `<button class="nav-group-toggle">`
(keyboard- and AT-reachable for free) carrying a chevron, and **exactly one group is open at a (keyboard- and AT-reachable for free) carrying a chevron, and **exactly one group is open at a
@@ -532,12 +755,28 @@ height with no magic number to drift as item counts change.
| Route | Page | Sections | | Route | Page | Sections |
|-------|------|----------| |-------|------|----------|
| `/backups` | Áttekintés | storage overview, whole-guest Rendszermentés, status stat cards, single-copy warning | | `/backups` | Áttekintés | storage overview, whole-guest Rendszermentés, status stat cards, single-copy warning, **backup-target banner + offer (v0.186.0)** |
| `/backups/remote` | Távoli mentés | Felhom-offsite status card (3 states, display-only), tier-3 status block + quota, participation toggles (+ zero-toggle hint; the persisted zero-toggle run-warning is DISPLAY-replaced by a "kijelölés módosult" note once ≥1 app is toggled — `offboxWarningDisplay`, v0.126.0), manual-target form (`#offbox-section`) | | `/backups/remote` | Távoli mentés | Felhom-offsite status card (3 states, display-only), tier-3 status block + quota, participation toggles (+ zero-toggle hint; the persisted zero-toggle run-warning is DISPLAY-replaced by a "kijelölés módosult" note once ≥1 app is toggled — `offboxWarningDisplay`, v0.126.0), manual-target form (`#offbox-section`) |
| `/backups/apps` | Alkalmazások | schedule, Adatbázisok table, per-app 1./2./3. tier rows (tier-2 config entry; tier-3 actions deep-link to `/backups/remote#offbox-section`) | | `/backups/apps` | Alkalmazások | schedule, Adatbázisok table, per-app 1./2./3. tier rows (tier-2 config entry; tier-3 actions deep-link to `/backups/remote#offbox-section`) |
| `/backups/restore` | Visszaállítás | restore panel, offsite restore list (**one „Visszaállítás…" entry per app** since v0.154.0), existing verification copies, .fab download/import loop | | `/backups/restore` | Visszaállítás | restore panel, offsite restore list (**one „Visszaállítás…" entry per app** since v0.154.0), existing verification copies, .fab download/import loop |
| `/backups/restore/app?name=<app>` | Visszaállítás — <app> | **R-48 per-app offsite restore wizard** (v0.154.0). GET-only; three described intent cards (ellenőrzés / hiányzó fájlok / teljes visszaállítás), a visible phase strip, and a server-derived step. Adds NO mutation endpoint — every card posts to the pre-existing `/backup/offbox/{restore,place,reconstitute}` | | `/backups/restore/app?name=<app>` | Visszaállítás — <app> | **R-48 per-app offsite restore wizard** (v0.154.0). GET-only; three described intent cards (ellenőrzés / hiányzó fájlok / teljes visszaállítás), a visible phase strip, and a server-derived step. Adds NO mutation endpoint — every card posts to the pre-existing `/backup/offbox/{restore,place,reconstitute}` |
**Backup-target banner + offer (E-2 · v0.186.0, R-114 + R-112).** The `/backups` page renders the
whole-system backup-target state server-side, from the AGENT's view (never from our own intent flag).
Four outcomes, three of which the customer sees nothing for or one thing for:
| State | Renders |
|---|---|
| healthy — a real drive holds the target | **nothing** (no badge, no reassurance: a working box must look normal) |
| degraded, never configured (`local`/unset) | the system-disk copy **+ an offer control** that POSTs `/api/storage/backup-target/assign` |
| **configured, drive absent** (`TargetAbsent`) | the absent-drive copy, **no offer** — the remedy is to reconnect *that* drive |
| unknown (agent unreachable / pre-R-82) | **nothing** — absence of an answer is not degradation |
`degradedMessageFor` is the single decision point for customer copy; `backupTargetView` returns nil
for the two silent states. The absent copy is verbatim the hub's `backup_target_absent` email so the
banner and the mail agree. The offer never auto-submits, and `restart_required` from assign is shown
rather than papered over with a self-restart (the agent deliberately does not restart itself).
Shared data builders: `backupsCommonData` (chrome + full-status + flash) + `backupsOffboxData` Shared data builders: `backupsCommonData` (chrome + full-status + flash) + `backupsOffboxData`
(offbox target/toggles) in `handlers.go`; shared partials in `templates/backups_shared.html`. (offbox target/toggles) in `handlers.go`; shared partials in `templates/backups_shared.html`.
(The v0.124.0 split was MOVE-only, gated one-shot by `backups_split_move_check.py`; the gate was (The v0.124.0 split was MOVE-only, gated one-shot by `backups_split_move_check.py`; the gate was
@@ -591,6 +830,29 @@ retired in v0.126.0 when the moved blocks were legitimately rewritten onto the s
The nightly backup has two phases that run sequentially. All paths are **per-drive** — each physical drive gets its own restic repo and per-app DB dump directories. The nightly backup has two phases that run sequentially. All paths are **per-drive** — each physical drive gets its own restic repo and per-app DB dump directories.
> **Customer-configurable backup window (v0.168.0).** ONE setting on the backup page — **"Mentési
> időablak kezdete"** (start W, default "02:30") — drives every leg at FIXED, never-stored offsets so
> they can never be misordered: DB dump at **W**, tier-2 mirror at **W+60m**, off-box at **W+105m**
> (wrap-safe). The whole-guest (agent PBS/vzdump) cycle is gated to **[W+2h, W+6h)** with a safety valve
> (runs regardless once the last successful backup is older than cadence+24h, so a box only ever on
> outside its window never starves); **manual "Mentés most" is never gated**. A saved window fans out to
> the three daily legs via `scheduler.UpdateDaily` and takes effect **without a restart**. Precedence:
> settings > controller.yaml `db_dump_schedule` > "02:30". See `internal/backupwindow`.
> **Multi-tier whole-guest backup (v0.174.0, R-82 Slice B).** The agent can serve SEVERAL whole-guest
> backup tiers with independent cadences — "local daily + PBS weekly" (agent >= v0.97.0,
> `GET /backup/tiers`). The controller owns quiescing, so it reconciles them: it collects EVERY due
> tier up front and runs them inside **ONE quiesce window** — one stop, N sequential backups (vzdump
> holds a guest lock), one resume. Two cycles on the weekly night would mean two app outages for one
> night's work. The app stays quiesced until the **LAST** tier snapshots, so every tier is
> app-consistent; the consequence is that both-due-night downtime is *(first tier's full backup)* +
> *(last tier's snapshot)*, which is why tiers run fast-first (the agent advertises primary/local
> first). A manual **"Mentés most"** covers every tier, due-ness ignored. The window gate's safety
> valve evaluates the OLDEST due tier, so a stale DR tier cannot be starved by a fresher local one.
> Against a **pre-R-82 agent** (`/backup/tiers` 404s) the loop degrades to the single untargeted
> tier, logs it once, and still takes the backup — MinAgent is unchanged. See `internal/quiesce`
> (`tiers.go`) and `internal/agentapi/backup_tiers.go`.
> **Atomic dump writes (v0.118.0, CAMPAIGN-3 F7).** BOTH dump paths are crash-safe: the DB dump > **Atomic dump writes (v0.118.0, CAMPAIGN-3 F7).** BOTH dump paths are crash-safe: the DB dump
> (`dbdump.go` DumpOne) and the Docker-volume dump (`DumpAppVolumes`) write to a `.tmp` sibling, fsync, > (`dbdump.go` DumpOne) and the Docker-volume dump (`DumpAppVolumes`) write to a `.tmp` sibling, fsync,
> then `os.Rename` over the restore point ONLY on success. A mid-write failure (a NFS cut mid-tar, an > then `os.Rename` over the restore point ONLY on success. A mid-write failure (a NFS cut mid-tar, an
@@ -604,12 +866,24 @@ The nightly backup has two phases that run sequentially. All paths are **per-dri
> drive. Guarded: only for a DEPLOYED app whose CURRENT drive differs from the dir's drive; never the > drive. Guarded: only for a DEPLOYED app whose CURRENT drive differs from the dir's drive; never the
> current-drive dir (the live restore point) or an undeployed app's dir; strictly under `backups/primary/`. > current-drive dir (the live restore point) or an undeployed app's dir; strictly under `backups/primary/`.
> **NAS backup locality (v0.118.0, CAMPAIGN-3 Part 4 — decision A).** A NAS-resident app's tier-1 > **~~NAS backup locality (v0.118.0, CAMPAIGN-3 Part 4 — decision A)~~ — SUPERSEDED by R-108
> artifacts live **on the NAS itself** (`nas-media/backups/primary/<app>`), beside the data. During a > (v0.187.0, 2026-07-30).** Decision A said a NAS-resident app's tier-1 artifacts live **on the NAS
> NAS outage both the app data AND its freshest tier-1 dump are on the dead device — the **tier-2** > itself** (`nas-media/backups/primary/<app>`), beside the data, with the tier-2 cross-drive copy as the
> cross-drive copy to a local drive is the off-NAS leg that saves them (and only after it has run). > off-NAS mitigation. That locality is exactly what made a `backups/` tree reachable through
> This locality is deliberate (kept in the fork over retargeting tier-1 to a local drive); the tier-2 > FileBrowser's share-ROOT bind (`download: true`), and it is why architectural target **D5** — app
> copy is the mitigation. Stated here so the outage window is never a surprise. > secrets in the local recovery unit — could not be adopted.
>
> **An app's data namespace may no longer live on network storage at all** (operator ruling
> 2026-07-30), so the case decision A described can no longer arise: no app on a NAS ⇒ no
> `backups/primary/` on a NAS. `settings.RefuseAsAppNamespace` is the single predicate; every
> placement surface consults it (deploy POST, per-app migrate, decommission-with-migrate). The NAS
> keeps its **browse** capability unchanged — the share-root `:rslave` bind is load-bearing for
> automount wake (R-67) and was deliberately NOT narrowed; scoping it is undefinable anyway, since
> apps on a share store at `<share>/<app>` and creating a `userdata/` layer would write Felhom
> convention onto a customer's own NAS.
>
> The NAS-outage window decision A warned about is therefore also gone: an app's tier-1 artifacts are
> always on a local drive now, because the app itself always is.
**Drive layout (v0.26.0):** **Drive layout (v0.26.0):**
``` ```
@@ -656,36 +930,64 @@ Path computation is centralized in `backup/paths.go` via the `FelhomDataDir = "f
> `AppSecondaryRsyncPath`, `SecondaryInfraPath`) describe the pre-strip layout — restic/cross-drive was > `AppSecondaryRsyncPath`, `SecondaryInfraPath`) describe the pre-strip layout — restic/cross-drive was
> removed in slice 8C. This section is rewritten when Tier 2 (Phase 3) lands. > removed in slice 8C. This section is rewritten when Tier 2 (Phase 3) lands.
#### Per-app recovery unit (Phase 2, v0.53.x) — SECRET-FREE #### Per-app recovery unit (Phase 2, v0.53.x; secret model rewritten by **D5**, v0.188.0)
Each app's `backups/primary/<app>/` is a self-contained, recreatable **recovery unit**: Each app's `backups/primary/<app>/` is a self-contained, recreatable **recovery unit**:
``` ```
backups/primary/<app>/ backups/primary/<app>/
├── compose/ docker-compose.yml + .felhom.yml + a SECRET-STRIPPED app.yaml ├── compose/ docker-compose.yml + .felhom.yml + app.yaml (0600 — CARRIES the portable secrets)
├── db-dumps/ app-consistent DB dump(s) ├── db-dumps/ app-consistent DB dump(s)
├── volume-dumps/ named-volume tars ├── volume-dumps/ named-volume tars
└── manifest.json image pins, secret env-var NAMES, data_key names, checksums, secret_source └── manifest.json image pins, secret NAMES, data_key names, portable NAMES, checksums, secret_source
``` ```
- **Secret-free by design.** The unit stores **no secret value, no data-encrypting key, and not the - **The secret split (D5, schema 2, operator ruling 2026-07-30).** The unit was secret-free until
Docker image** — only the pinned image tag(s) (re-pulled on restore) and the *names* of the secret / v0.188.0, and that made "restore from the drive alone" false: the fast, local, customer-doable
`data_key` env vars. Rationale: app.yaml + the encryption key live on the guest rootfs → already in Tier-1/2 restore secretly depended on the slow, operator-driven whole-guest restore, because a
the PBS whole-guest snapshot, and the hub is deliberately zero-knowledge. Restore recovers the data-encrypting key or a DB password absent from the guest cannot be regenerated without leaving the
original secrets from the guest's own app.yaml (live, or via PBS); for a `data_key` app it restored data unreachable. **Tier-1/2 now needs the drive and nothing else.** What travels is decided
**fails closed** (refuse + warn) if the key can't be recovered — data-keys are NEVER generated. in ONE place, `stacks.PortableSecretEnvVars`:
**Resettable secrets (O4, v0.99.0):** an unrecoverable resettable secret (DB password etc.) gets a - **TRAVELS — every `type: secret` field** (45 of 53 across the catalog): the declared `data_key`s,
**generated replacement** from its catalog `generate` spec (`stacks.GenerateSecretForField` via the the 18 DB/root passwords, and the internal signing/encryption secrets. Each of these either
`backup.SetSecretGenerator` seam) instead of redeploying blank (which failed compose-up); the new decrypts data sitting on the SAME drive or authenticates to a container on an internal compose
value persists encrypted through the normal `RecreateStackDefinitionFromUnit``SaveAppConfig` path. Fields network with no external listener, so possessing it adds nothing to possessing the drive — which is
with no `generate` spec still proceed with a loud "may fail to start" WARN. Residual case: a restored exactly D2's argument for keeping the DATA plaintext. Written into the unit's app.yaml at **0600**,
volume tar carrying the OLD internal credential hash may still need a manual in-DB reset. plaintext, like the data beside it.
- **WITHHELD — every `type: password` field** (7 admin/UI logins) **plus the `nonPortableSecrets`
register** (`vaultwarden/ADMIN_TOKEN`, whose `/admin` panel is on the app's public web port).
These authenticate against published services, so their blast radius is NOT bounded by the drive.
They stay in the guest and are regenerated on restore (O4). **Excluding this class is what licenses
the plaintext ruling — the two are coupled and must not be relaxed independently.**
- The register is **code, not a catalog flag**, deliberately: a security boundary a catalog push can
silently move is not a boundary (cf. R-97a). Adding an app whose `type: secret` field gates an
internet-reachable login means adding a row there.
- **Fail-closed is unchanged.** A `data_key` missing from **both** the unit and the guest still refuses
the restore outright (never generated). D5 makes the key normally present; "normally" is not a reason
to soften the gate.
- **Precedence: the UNIT WINS** over the guest when both hold a value. Not "newest wins" — the unit's
secrets are captured in the same run as the dumps beside them, so the unit's value is the one that
MATCHES THE DATA BEING RESTORED, while the guest's is merely the most recent. A rotated data key does
not decrypt data encrypted with the old one, and a rotated DB password does not match the hash inside
the restored data directory. Pinned in both directions.
- **Resettable secrets (O4, v0.99.0)** — now the rare path, since the portable class comes from the
unit. An unrecoverable withheld secret gets a **generated replacement** from its catalog `generate`
spec (`stacks.GenerateSecretForField` via the `backup.SetSecretGenerator` seam) rather than redeploying
blank; the value persists encrypted through `RecreateStackDefinitionFromUnit``SaveAppConfig`.
⚠️ **R-127:** a regenerated **database** password is NOT harmless — `POSTGRES_PASSWORD` is ignored once
PGDATA is non-empty, so the restored data dir keeps the old role hash and the app cannot authenticate
against its own rows, while the dump replay (local trust socket) still reports success. The WARN says so.
- Helpers: `RecoveryUnitPath` / `RecoveryUnitComposePath` / `RecoveryUnitManifestPath` - Helpers: `RecoveryUnitPath` / `RecoveryUnitComposePath` / `RecoveryUnitManifestPath`
(`internal/appbackup/paths.go`). Capture: `Manager.CaptureRecoveryUnit` (`internal/backup/recovery_unit.go`), (`internal/appbackup/paths.go`). Capture: `Manager.CaptureRecoveryUnit` (`internal/backup/recovery_unit.go`),
run from the daily DB dump and the periodic `RefreshCache` (idempotent checksum-skip). The non-secret run from the daily DB dump and the periodic `RefreshCache` (idempotent checksum-skip); the split itself
env comes from `StackDataProvider.GetStackRecoveryInfo` (excludes secret-named + encrypted values, so is in `buildUnitAppYaml`. The env + portable values come from `StackDataProvider.GetStackRecoveryInfo`,
the capture never touches a secret). `data_key` fields are marked in `.felhom.yml` which keeps `NonSecretEnv` and the secret set disjoint by construction. `data_key` fields are marked in
(`DeployField.DataKey`). `.felhom.yml` (`DeployField.DataKey`).
- **A schema-1 (pre-D5) unit carries no secrets** and still restores from the guest — the restore
degrades rather than failing, and the next capture rewrites the unit (the app.yaml checksum changes).
- **Consequence for the other tiers:** the unit is copied by Tier 2 (another customer drive, plaintext,
same reasoning) and pushed offsite by restic (`offbox.go` — encrypted at rest under the customer-owned
repo password). Neither tier's code changed; the secrets simply travel with the unit they already carried.
- **Restore replays the DB dump (F17, v0.61.0; re-sequenced v0.153.0, R-47).** `RestoreFromRecoveryUnit` - **Restore replays the DB dump (F17, v0.61.0; re-sequenced v0.153.0, R-47).** `RestoreFromRecoveryUnit`
(and the `RestoreApp` fallback) stops the app → restores named-volume tars → recreates the compose (and the `RestoreApp` fallback) stops the app → restores named-volume tars → recreates the compose
definition and persists the recovered env (`RecreateStackDefinitionFromUnit`**starts nothing**) definition and persists the recovered env (`RecreateStackDefinitionFromUnit`**starts nothing**)
@@ -729,10 +1031,31 @@ customer edit after the last copy wins) and **nothing is ever deleted** — this
`CrossDriveBackup.DestinationPath` (never a fresh target selection). Single-flight with `CrossDriveBackup.DestinationPath` (never a fresh target selection). Single-flight with
backup/restore; refusals (no copy / never ran / copy dir gone / either drive disconnected / backup/restore; refusals (no copy / never ran / copy dir gone / either drive disconnected /
decommissioned) happen before the app is stopped, with customer-readable Hungarian reasons; decommissioned) happen before the app is stopped, with customer-readable Hungarian reasons;
stop → copy → start → health-wait; zero files copied is a success ("Nincs hiányzó fájl…"). Out of stop → copy → start → health-wait. Out of scope by design: overwrite/point-in-time restore (offbox +
scope by design: overwrite/point-in-time restore (offbox + operator paths), per-file selection, operator paths) and per-file selection. Apps that index their data dir (e.g. Nextcloud) may need a
`recovery-unit/`. Apps that index their data dir (e.g. Nextcloud) may need a rescan (occ rescan (occ files:scan) before restored files appear in their own UI.
files:scan) before restored files appear in their own UI.
> **COVERAGE — read this before assuming an app is protected by this button (C9-F1, v0.183.0).**
> This restore reads `hdd/` and `userdata/` **only**. It has never read `recovery-unit/`, which every
> Tier-2 run also writes and which holds the app's DB dumps and named-volume tarballs. Enumerated
> across all 53 catalog templates: **43 apps have no readable subtree at all** (their data is entirely
> in named volumes — BookStack, Docmost, Vaultwarden, Gitea, …), **9** have file legs but never their
> database or volumes, 1 is stateless. So the button is a guaranteed no-op for 81% of the catalog and
> only ever partial for the rest.
>
> Since v0.183.0 it is HONEST about that instead of silently reporting success:
> `Tier2RestoreCoverage` is consulted **before** anything starts, an app with no readable subtree is
> refused **without being stopped** and told which action does work („…Használd a Visszaállítás
> indítása gombot a Biztonsági mentés → Visszaállítás oldalon."), and a run that does proceed claims
> only what it **examined** („Minden vizsgált fájl megvan a helyén.") plus a disclosure that the
> database and internal volumes are not part of this restore.
>
> The action that DOES cover those apps is the keep-side recovery-unit restore
> (`POST /backup/restore``RestoreFromRecoveryUnit`), which replays volume tarballs and DB dumps.
> Routing customers there from the Tier-2 card is filed as **C9-F1b** — it puts a destructive
> operation behind a button reached via a non-destructive one, so the confirm copy must carry that
> difference. **C9-F4** is filed separately: nothing reads the Tier-2 copy's `recovery-unit/` mirror,
> so the second local copy that exists precisely for drive loss is unreachable by any customer action.
**Per-app Tier-2 config panel (v0.57.0)** — `GET/POST /stacks/{name}/backup` **Per-app Tier-2 config panel (v0.57.0)** — `GET/POST /stacks/{name}/backup`
(`internal/web/tier2_config_handler.go` + `templates/tier2_config.html`). The "2. mentés" row's (`internal/web/tier2_config_handler.go` + `templates/tier2_config.html`). The "2. mentés" row's
@@ -1602,8 +1925,9 @@ Each event carries typed detail structs (e.g., `BackupDetails`, `DiskDetails`, `
**Deployed-app-down alerting (fix-3, v0.120.0, CAMPAIGN-3).** A `deadapp-check` scheduler job (every **Deployed-app-down alerting (fix-3, v0.120.0, CAMPAIGN-3).** A `deadapp-check` scheduler job (every
30 s, after a 90 s boot grace) scans `stackMgr.GetStacks()`: a DEPLOYED app whose containers are 30 s, after a 90 s boot grace) scans `stackMgr.GetStacks()`: a DEPLOYED app whose containers are
`stopped`/`exited` (`stacks.IsDownState` — a Docker `created`/`dead` container, the F11 dead-at-boot `exited`/`degraded` (`stacks.IsDownState` minus the `stopped` exclusion added in v0.164.0 — see below;
case, resolves to `stopped`) gets a **state-based WARN dashboard banner** ("Telepített alkalmazás nem a Docker `created`/`dead` container, the F11 dead-at-boot case, resolves to `exited`) gets a
**state-based WARN dashboard banner** ("Telepített alkalmazás nem
fut: <app>", grouped above 3 so a reboot storm doesn't wall the dashboard) that self-clears when the fut: <app>", grouped above 3 so a reboot storm doesn't wall the dashboard) that self-clears when the
app runs again, AND an `app_start_failed` hub event fired **once per running→down transition** app runs again, AND an `app_start_failed` hub event fired **once per running→down transition**
(`Notifier.NotifyAppStartFailures` tracks per-app state; down→down cycles are silent — the hub owns (`Notifier.NotifyAppStartFailures` tracks per-app state; down→down cycles are silent — the hub owns
@@ -1626,17 +1950,124 @@ colour, counted with the stopped apps, URL flagged unpublished (Traefik withhold
routed member is the dead one). Policy reads are one `docker inspect` per down member of a mixed routed member is the dead one). Policy reads are one `docker inspect` per down member of a mixed
stack, cached per container+state. stack, cached per container+state.
**Boot desired-state reconciliation (R-52, v0.156.0, `internal/bootrecon`).** A `deployed: true` app **Deliberate stops are silent (v0.164.0).** Stopping an app from the UI (Leállítás → `StopStack`
that missed its boot start used to stay down until a human noticed — the same shutdown that produced `docker compose down` → zero containers → the deployed stack aggregates to `StateStopped`) is the
F4 left immich and calibre-web `Exited` while ten sibling containers came back, and they were still user's own action, not a fault, and must not raise the banner OR the `app_start_failed` email. The
down 18 h later (F5). At startup (5 s after the quiesce recovery, so the two never race) the scan's pure core was extracted to `classifyRunStates([]stacks.Stack)` and its down predicate is now
controller performs **one bounded sweep**: every deployed, non-protected, not-mid-deploy stack that `stacks.IsDownState(st.State) && st.State != stacks.StateStopped` — the SINGLE fix-3 derivation point,
still HAS containers and is down gets `StartStack`, at most **2 attempts 30 s apart**, then it stops so `StateStopped` is dropped from both the banner dead-list and the notifier Down-set at once (the
and the alarm owns the problem. Never a restart loop. **An app the customer stopped is never launcher tile still shows greyed „Leállítva"; the monitoring page and dashboard counters are factual
touched** — the UI's Stop is `compose down`, which removes the containers, so "has containers and display, not alarms, and are unchanged). This rests on two invariants: **I1** — a UI stop always ends
they are down" is what distinguishes an interrupted boot from a deliberate stop. The whole sweep at `StateStopped` (compose down removes the containers); **I2** — the P2 restart-policy census
fits inside the 90 s boot grace, so a successful recovery is silent and a failed one still alerts. (53 templates / 78 services, all `unless-stopped`) means a crashing app never comes to rest at
Outcome is logged per attempt at INFO; no new hub event (the existing alarm is the escalation). `stopped`, so faults still surface as `exited`/`degraded`/`restarting`/`unhealthy`. If either
invariant changes, revisit the suppression.
> **C9-F2 (v0.183.0) — the `restarting` half of that sentence was a wish, not a fact.** `restarting`
> was named above as a state through which faults "still surface", but it was in no down set at all:
> `IsDownState` excludes it, so a crash-looping app raised no banner, no `app_start_failed`, no email
> and no hub event — and `unless-stopped` means Docker retries forever, so the silence was permanent.
> Campaign 9 watched docmost loop for nine minutes while the F-OBS heartbeat printed
> „180 scans since boot, 4 deployed app(s) evaluated, **0 currently down**".
>
> The fix does **not** add `StateRestarting` to `IsDownState` — that alarms on every deploy and update
> fleet-wide. A SUSTAINED restarting run becomes down after `stacks.crashLoopAfter` (**5 min**), chosen
> above the deploy flow's 120 s health timeout, Mealie's 60 s `start_period` and R-97b's 180 s quiesce
> grace, so the suppression windows compose into one bounded delay rather than leaving a gap. Carried
> by `Stack.RestartingSince` (stamped in `refreshStatusLocked`, cleared on any other state, not
> persisted) and read via `Stack.CrashLooping(now)` — used by BOTH the alarm and the dashboard
> "how many of my apps work" counter, which previously counted `restarting` as running and so
> contradicted the alarm on the same screen. Pinned by `crashloop_classify_test.go`; the test that a
> brief restart stays silent is the one that fails against the naive fix.
`IsDownState` itself is deliberately UNCHANGED (other
callers rely on stopped counting as down). An out-of-band `docker compose stop` leaves the containers
present → `StateExited` → still alerts, which is correct (out-of-band tampering is reportable).
**Boot desired-state reconciliation (R-52, v0.156.0, `internal/bootrecon`; rebuilt on recorded intent
in R-166, v0.189.0).** A `deployed: true` app that missed its boot start used to stay down until a
human noticed — the same shutdown that produced F4 left immich and calibre-web `Exited` while ten
sibling containers came back, and they were still down 18 h later (F5). At startup (5 s after the
quiesce and app-stop recoveries, so the three never race) the controller performs **one bounded
sweep**: every deployed, non-protected, not-mid-deploy stack that is down gets `StartStack`, at most
**2 attempts 30 s apart**, then it stops and the alarm owns the problem. Never a restart loop. The
whole sweep fits inside the 90 s boot grace, so a successful recovery is silent and a failed one
still alerts. Outcome is logged per attempt at INFO; no new hub event (the existing alarm is the
escalation).
**What "down" means here changed in v0.189.0.** Until then the sweep required the stack to still HAVE
containers, because the UI's Stop is `compose down` (which removes them) and "zero containers" was
read as a deliberate stop. That inference was wrong in two silent ways: a **power cut mid-compose**
and an **interrupted deploy** also leave zero containers, and both were skipped as "the customer
stopped it" and left down indefinitely. Since R-166 the sweep reads the customer's **recorded
intent** (`desired_state` in `app.yaml`) instead:
| `desired_state` | containers | result |
|---|---|---|
| `stopped` | any | **never** started — the customer said so, and no observation overrides it |
| `running` | 0 | **recovered** — the power-cut / interrupted-deploy case, invisible before v0.189.0 |
| `running` | >0 and down | **recovered** (unchanged) |
| `running` | >0 and up | left alone |
| absent (legacy) | 0 | **not** started — byte-identical to the pre-v0.189.0 behaviour |
| absent (legacy) | >0 and down | **recovered** — byte-identical to the pre-v0.189.0 behaviour |
**Absent means UNKNOWN, never "running".** Every `app.yaml` written before v0.189.0 lacks the field,
so absent is what an upgraded box reads for every app that has not been started or stopped since;
reading it as "running" would start every deliberately-stopped app on the first boot after the
upgrade. Where intent is unknown the sweep falls back to the old inference rather than inventing an
answer, and a running-only startup **backfill** converges the unambiguous cases (deployed and
observed up) without waiting for a button press. `stopped` is never backfilled from any signal.
**The sweep no longer looks only once (R-157 mechanism A, v0.190.0).** It used to sample the fleet at
T+5 s and return; at that moment docker is still restoring containers after a hard reset, so an app
that had not yet settled into a down state was never a candidate — **measured failing on three of six
hard resets**. It is now a **settle-then-sweep window**: the fleet (name, state, container count) is
sampled every 5 s, called settled after 3 identical samples, and swept **once**, at the end, on a
settled fleet. The window ends on whichever comes first — settled, or a **50 s budget** — and the log
says which. Sampling is read-only and there is still exactly one sweep with its per-app attempt bound
intact: this widens a bounded window, it does not remove the bound. `settle (5 s) + budget (50 s) +
one 30 s retry = 85 s` stays inside the 90 s `deadAppBootGrace`, which is what keeps a successful
recovery silent; a window that genuinely overruns emits a `LATE RECOVERY` WARN naming the apps rather
than the grace being widened to hide it.
**The sweep asks before it starts (R-171, v0.190.0).** Three things legitimately hold an app down,
and starting it would be the wrong repair for all three: its **data drive is absent** (compose would
create the bind sources on the guest rootfs — the hazard the drive gate exists to prevent), a
**quiesce** is holding it for a whole-guest backup, or an **app-data operation** (volume dump, offsite
restore, `.fab` export) is holding it. All three are refused through one seam, reusing the signals
their owners already publish. **Fail-safe: a drive whose liveness cannot be determined is treated as
absent.** Held apps are reported separately from `StillDown` — they are not a fault the sweep failed
to fix, and reporting them as one is a false alarm. This closed a regression v0.189.0 introduced:
before it, a drive-gate-stopped app read as `running` + zero containers, so the sweep started it,
burned both attempts and handed it to the dead-app alarm.
**Both boot gates read intent (R-170, v0.190.0).** There are two: the R-52 sweep above, and the
drive-backed **boot recreate gate** (`shouldRecreateOnBoot`, `internal/web/intermediary.go`), which
re-creates a drive-backed app onto its re-propagated drive after a guest reboot. Until v0.190.0 the
second still ended in `&& hasContainers`, so the two disagreed about the same question. It now uses
the identical three-way table — `stopped` → never, `running` → recreate whatever the container count,
absent → exactly the old `hasContainers` behaviour. Its drive-presence term is untouched and still
load-bearing: an app whose drive is absent is never recreated there either.
**Desired state — who owns it (R-166, v0.189.0).** `app.yaml` gains `desired_state`, a tri-state
`""` / `running` / `stopped`. It is written by **the customer's own action and nothing else**: the
`/api/stacks/{name}/{action}` switch (`start`/`restart`/`update` → running, `stop` → stopped),
`DeployStack`, `UpdateOptionalConfig`'s redeploy branch, and the `.fab` import. `StartStack` and
`StopStack` are deliberately **not** writers — a census found 14 callers of which only 2 are the
customer, and recording intent in the primitive would make a nightly backup indistinguishable from
the customer pressing Stop, which is the confusion the feature exists to end. Intent is written
**before** the act, and an action whose intent cannot be recorded is **refused**.
**Interrupted app-data operations (R-166, v0.189.0, `backup.AppStopGuard`).** A volume dump, an
off-site reconstitution and a `.fab` export all stop an app, work on its data, and start it again.
A controller killed inside that window left the app down with nothing on disk recording why or that
it was owed a restart. A persisted marker (`<data_dir>/appstop-state.json` — its **own** file, never
quiesce's, so one file has one writer) is now written **before** the stop and cleared only after a
restart that succeeded; a failed restart deliberately keeps it. At startup `Recover()` restarts the
recorded apps, clears the marker, and its outcome is reported to the operator on the existing
`backup_failed` event — an interrupted operation means the backup did not complete. **The `defer` in
those functions is not the mechanism**: a SIGKILL runs no deferred function (Campaign 8 fault 10, on
live hardware), which is exactly what the marker covers.
#### Default Enabled Events #### Default Enabled Events
@@ -1905,6 +2336,17 @@ race where a new `felhom.<domain>` cert appears in CT logs minutes before any pa
never rewrites, a hub outage never clears). The report carries `claimed` (hub ingests set-only). never rewrites, a hub outage never clears). The report carries `claimed` (hub ingests set-only).
- **Escape hatch**: `felhom-controller --print-reset-code` prints a one-time local code (generation - **Escape hatch**: `felhom-controller --print-reset-code` prints a one-time local code (generation
above cached/baked/consumed); the same gate consumes it. Root-gated by `docker exec` reachability. above cached/baked/consumed); the same gate consumes it. Root-gated by `docker exec` reachability.
**It runs as a SEPARATE PROCESS, which is why `effectiveClaimCode` READS THROUGH to the persisted
state (v0.198.0, R-204 item 1).** Until then the hatch persisted a new code while the running
server kept the old one cached, so the minted code was refused until the controller was restarted —
and nothing said so. The read-through (`settings.ReloadClaimCode`) is on the claim path only, and
only while the box carries no password (`claimGateActive` returns on `authEnabled()` first).
**Deliberately not a watcher, a signal handler or a TTL:** a TTL would leave a window in which a
SUPERSEDED code still works, which is worse than the bug. Refreshes hash/generation/issuedAt only —
never `ClaimConsumedGeneration`, which this process alone writes and must stay monotonic.
`effectiveClaimCode` returns an error and **every caller fails closed** (the gate stays UP, the claim
is refused); an ABSENT settings file is not an error, since a pre-first-save box falls back to the
controller.yaml bake.
#### Session Auth (`internal/web/auth.go`) #### Session Auth (`internal/web/auth.go`)
@@ -2142,6 +2584,18 @@ During setup wizard drive scan, both current and historical backups are discover
Generates `recovery-info.txt` on the system data partition with customer ID, Hub URL, retrieval password, and recovery instructions in Hungarian. Updated on startup and after config changes. Also displayed on the Settings page in a "Vészhelyzeti információk" section. Generates `recovery-info.txt` on the system data partition with customer ID, Hub URL, retrieval password, and recovery instructions in Hungarian. Updated on startup and after config changes. Also displayed on the Settings page in a "Vészhelyzeti információk" section.
**No secret is rendered into a page's response body (v0.207.0 + v0.208.0, R-249/R-254).** Three endpoints implement one rule — the page carries a BOOLEAN, the value comes from an explicit authenticated POST with `Cache-Control: no-store`, and the reveal is LOGGED (reading a value off markup left no trace at all):
| Secret | Endpoint | Read from |
|---|---|---|
| retrieval passphrase | `POST /settings/retrieval-password/reveal` | settings |
| an app's generated first-login password | `POST /apps/<slug>/initial-credentials/reveal` | **live from the container** — never a cached copy |
| an already-deployed app's auto-generated secret field | `POST /stacks/<name>/auto-field/reveal` | the decrypted `app.yaml`; authorised by requiring a `type: secret` auto-field of that stack |
They are deliberately **per-secret**, not one generic "reveal any named secret" endpoint — that would turn three narrow exposures into one lever with a parameter. **The PRE-DEPLOY hidden input is untouched and deliberate:** a form must carry what it submits (see §318 below). `scripts/secret_in_markup_gate.py` (in `controller_gates.py`) enforces the rule over all templates; its measured blind spot — a secret arriving under a neutral page-data key — is in its docstring.
**The retrieval passphrase is NOT rendered into that page (v0.207.0, R-249).** `securityPageData` passes only `HasRetrievalPassword` (a boolean), and the value is fetched by an explicit act: **`POST /settings/retrieval-password/reveal`** → `{"ok":true,"data":{"password":"…"}}`, behind RequireAuth + CsrfProtect like every other POST, `Cache-Control: no-store`, and logged (`retrieval passphrase revealed via the security page from <ip>` — the value is never logged). Until v0.207.0 the page rendered the plaintext into a `display:none` span, so any fetch of the page returned it; the toggle was cosmetic. This follows the rule `escrow_handlers.go` already states for the recovery code: a secret is revealed by an XHR, never templated server-side into HTML.
### 11. Disaster Recovery ### 11. Disaster Recovery
When a system drive fails and is replaced, the recovery flow uses the setup wizard: When a system drive fails and is replaced, the recovery flow uses the setup wizard:
@@ -0,0 +1,495 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"strings"
"testing"
)
// R-166 §10 seam discipline — the recovery and the backfill are seams, and a seam that is never
// called is the defect class this project has shipped four times: a correct component, green unit
// tests that inject it directly, and no production caller.
//
// These walk main.go's AST. NOT strings.Contains — the sibling bootrecon test records the reason at
// first hand: a commented-out call still satisfies a substring match, so the text version passed the
// very red-proof it existed to fail. Comments are not code.
// mainBody returns func main()'s body from main.go, parsed.
func mainBody(t *testing.T) *ast.BlockStmt {
t.Helper()
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
for _, decl := range f.Decls {
if fn, ok := decl.(*ast.FuncDecl); ok && fn.Name.Name == "main" && fn.Body != nil {
return fn.Body
}
}
t.Fatal("func main() not found in main.go")
return nil
}
// callsInMain returns, in source order, the names of every call in func main() whose function
// expression is `x.Sel(...)` or `Sel(...)` — enough to identify the wiring calls by name.
func callsInMain(t *testing.T, body *ast.BlockStmt) []string {
t.Helper()
var names []string
ast.Inspect(body, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
switch fun := call.Fun.(type) {
case *ast.SelectorExpr:
names = append(names, fun.Sel.Name)
case *ast.Ident:
names = append(names, fun.Name)
}
return true
})
return names
}
func indexOfCall(names []string, want string) int {
for i, n := range names {
if n == want {
return i
}
}
return -1
}
// TestMainWiresAppStopRecovery is the Group-I seam test. Comment out the `appStopGuard.Recover()`
// line in main.go and this fails, where every behavioural test in internal/backup still passes.
func TestMainWiresAppStopRecovery(t *testing.T) {
names := callsInMain(t, mainBody(t))
if indexOfCall(names, "NewAppStopGuard") < 0 {
t.Fatal("func main() no longer builds the R-166 app-stop guard — nothing writes or reads the marker")
}
if indexOfCall(names, "SetStarter") < 0 {
t.Fatal("func main() no longer calls SetStarter on the app-stop guard — Recover would find the " +
"marker and be unable to start anything, leaving every interrupted app down")
}
if indexOfCall(names, "Recover") < 0 {
t.Fatal("func main() no longer calls Recover() on the app-stop guard — apps left stopped by an " +
"interrupted backup stay down forever (the R-166 defect, un-fixed)")
}
if indexOfCall(names, "SetAppStopGuard") < 0 {
t.Fatal("func main() no longer hands the recovered guard to the backup manager — the manager " +
"would build a SECOND guard over the same file, i.e. one file with two owners")
}
if indexOfCall(names, "SetStopGuard") < 0 {
t.Fatal("func main() no longer wires the exporter's stop guard — the .fab export path would be " +
"the one uncovered stop-and-restart site, which is how a reader concludes the class is handled")
}
}
// TestMainWiresDesiredStateBackfill pins the Part-1.5 call.
func TestMainWiresDesiredStateBackfill(t *testing.T) {
if indexOfCall(callsInMain(t, mainBody(t)), "BackfillDesiredState") < 0 {
t.Fatal("func main() no longer calls BackfillDesiredState — every existing app would stay on " +
"legacy inference until someone pressed a button on it")
}
}
// TestAppStopRecoveryPrecedesTheBootReconciler is §8.4's ORDERING requirement, and it is the reason
// the recovery returns its result instead of pushing it through a notifier seam.
//
// The recovery must COMPLETE — not merely be reached — before `go runBootReconcile(...)` is
// launched. If the boot reconciler ran first it would see an app the marker already explains, list
// it as an unexplained boot orphan, and one fault would be reported as two.
func TestAppStopRecoveryPrecedesTheBootReconciler(t *testing.T) {
names := callsInMain(t, mainBody(t))
recover := indexOfCall(names, "Recover")
bootrecon := indexOfCall(names, "runBootReconcile")
backfill := indexOfCall(names, "BackfillDesiredState")
if recover < 0 || bootrecon < 0 || backfill < 0 {
t.Fatalf("missing a call: Recover=%d runBootReconcile=%d BackfillDesiredState=%d", recover, bootrecon, backfill)
}
if recover >= bootrecon {
t.Fatal("the app-stop Recover no longer runs BEFORE the boot reconciler is launched — an app " +
"the marker explains would also be reported as an unexplained boot orphan (§8.4)")
}
if backfill >= bootrecon {
t.Fatal("the desired-state backfill no longer runs BEFORE the boot reconciler — the reconciler " +
"would decide from intent the backfill had not yet written")
}
if recover >= backfill {
t.Fatal("the backfill no longer runs AFTER the app-stop recovery — an app the recovery just " +
"restarted would still read as down and be left unrecorded")
}
}
// TestMainReportsTheInterruptedOperation pins §2.4: the recovery's outcome reaches the operator.
//
// The reporting call is deliberately far from the recovery (the notifier does not exist yet at
// recovery time), which is exactly the distance across which a wiring gets dropped.
func TestMainReportsTheInterruptedOperation(t *testing.T) {
body := mainBody(t)
names := callsInMain(t, body)
if indexOfCall(names, "NotifyBackupFailed") < 0 {
t.Fatal("func main() no longer reports an interrupted app-data operation to the operator — the " +
"controller died mid-backup and nobody is told (§2.4)")
}
// It must be guarded, not unconditional: a box with nothing to recover must not email an operator
// on every single boot.
//
// R-174 STRENGTHENED THIS. `!= nil` alone is no longer sufficient, because Recover now returns a
// non-nil result for a recovery that merely REFUSED starts (an absent data drive) — the drive
// gate working as designed. `NotifyBackupFailed` sends `backup_failed`, which is customer-enabled
// by default (settings.DefaultEnabledEvents), so a nil-only guard would email the customer
// "A biztonsági mentés sikertelen!" about an app nothing is wrong with. The guard must consult
// Alarming().
guardedByNil, guardedByAlarming := false, false
ast.Inspect(body, func(n ast.Node) bool {
ifst, ok := n.(*ast.IfStmt)
if !ok || ifst.Cond == nil {
return true
}
carries := false
for _, name := range callsInMain(t, ifst.Body) {
if name == "NotifyBackupFailed" {
carries = true
}
}
if !carries {
return true
}
// Walk the whole condition: it may be `a != nil && a.Alarming()`.
ast.Inspect(ifst.Cond, func(c ast.Node) bool {
switch e := c.(type) {
case *ast.BinaryExpr:
if x, ok := e.X.(*ast.Ident); ok && x.Name == "appStopRecovery" && e.Op == token.NEQ {
guardedByNil = true
}
case *ast.CallExpr:
if sel, ok := e.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Alarming" {
if x, ok := sel.X.(*ast.Ident); ok && x.Name == "appStopRecovery" {
guardedByAlarming = true
}
}
}
return true
})
return true
})
if !guardedByNil {
t.Fatal("the interrupted-operation alert is not guarded by `appStopRecovery != nil` — every " +
"healthy boot would page the operator about a backup that was never interrupted")
}
if !guardedByAlarming {
t.Fatal("the interrupted-operation alert is not guarded by appStopRecovery.Alarming() — a " +
"recovery that only REFUSED starts (drive absent) would be reported through " +
"NotifyBackupFailed, a customer-enabled event type, telling the customer their backup " +
"failed when the drive gate was simply doing its job (R-174)")
}
}
// --- R-171 seam: the boot drive gate must be WIRED in production -------------------------------
// TestMainWiresBootDriveGate is the Group-H seam test. An unwired drive gate is not a crash — it is
// SILENTLY the pre-v0.190.0 behaviour, which started apps onto absent drives (observed live,
// audits/DIAG-bootrecon-drive-absent-2026-08-02.md). Every behavioural test in internal/bootrecon
// still passes with the wiring gone, which is exactly the hole this walks the AST to close.
//
// AST, not strings.Contains: a commented-out call still contains the string — the distinction that
// made a previous version of this project's own seam test pass its red-proof (2026-07-21).
func TestMainWiresBootDriveGate(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
// (a) the settings handle the gate reads is assigned somewhere in main().
assigned := false
for _, name := range assignedIdentsIn(mainBody(t)) {
if name == "bootDriveSettings" {
assigned = true
}
}
if !assigned {
t.Fatal("func main() no longer assigns bootDriveSettings — the boot drive gate would read a " +
"nil settings handle and could not see a disconnected drive")
}
// (b) SetDriveGate is actually called where the reconciler is constructed.
called := false
ast.Inspect(f, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
if sel, ok := call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "SetDriveGate" {
called = true
}
return true
})
if !called {
t.Fatal("main.go no longer calls SetDriveGate on the boot reconciler — the sweep would start " +
"apps whose data drive is absent (R-171, a regression observed live on 2026-08-02)")
}
}
// --- R-174 seam: the app-stop guard's starter must be GATED in production -----------------------
// TestMainWiresGatedAppStopStarter pins Part 0's production wiring. `SetStarter(stackMgr)` — the raw
// manager, which is what shipped in v0.189.0 — compiles, passes every behavioural test in
// internal/backup (they inject their own gating starter), and silently starts apps onto absent
// drives at boot. The ONLY thing that distinguishes the fixed wiring from the broken one is the
// argument at the call site, so that is what this reads.
//
// AST, not strings.Contains: a commented-out call still contains the string.
func TestMainWiresGatedAppStopStarter(t *testing.T) {
body := mainBody(t)
var arg ast.Expr
found := false
ast.Inspect(body, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
sel, ok := call.Fun.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != "SetStarter" || len(call.Args) != 1 {
return true
}
// Only the app-stop guard's SetStarter, not some other type's.
if x, ok := sel.X.(*ast.Ident); !ok || x.Name != "appStopGuard" {
return true
}
arg, found = call.Args[0], true
return false
})
if !found {
t.Fatal("func main() no longer calls appStopGuard.SetStarter — Recover would find the marker " +
"and be unable to start anything")
}
// The argument must be a gatedAppStopStarter composite literal. A bare identifier (`stackMgr`)
// is precisely the v0.189.0 defect.
lit, ok := arg.(*ast.CompositeLit)
if !ok {
t.Fatalf("appStopGuard.SetStarter is wired with %T, not a gatedAppStopStarter literal — an "+
"un-gated starter restarts apps onto MISSING drives at boot (R-174, the R-171 defect one "+
"path over)", arg)
}
id, ok := lit.Type.(*ast.Ident)
if !ok || id.Name != "gatedAppStopStarter" {
t.Fatalf("appStopGuard.SetStarter is wired with a %v literal, want gatedAppStopStarter", lit.Type)
}
// And that gate must be a driveStartGate — the SAME predicate the boot sweep uses, so the two
// cannot disagree about whether an app's drive is available.
gated := false
for _, el := range lit.Elts {
kv, ok := el.(*ast.KeyValueExpr)
if !ok {
continue
}
k, ok := kv.Key.(*ast.Ident)
if !ok || k.Name != "gate" {
continue
}
if gl, ok := kv.Value.(*ast.CompositeLit); ok {
if gid, ok := gl.Type.(*ast.Ident); ok && gid.Name == "driveStartGate" {
gated = true
}
}
}
if !gated {
t.Fatal("the app-stop starter's gate is not a driveStartGate — the crash recovery and the " +
"boot sweep would answer \"may this app start?\" from two different implementations, " +
"which is the drift the extraction exists to prevent")
}
}
// TestBootDriveGateAndAppStopShareTheDrivePredicate pins the OTHER half of the same claim: the boot
// sweep must keep delegating to driveStartGate rather than growing its own copy of the drive checks.
//
// This is the "a comment asserting an invariant needs a test pinning it" rule. The claim — that the
// two gates cannot disagree — is true only while both call the same code.
func TestBootDriveGateAndAppStopShareTheDrivePredicate(t *testing.T) {
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "main.go", nil, 0)
if err != nil {
t.Fatalf("parse main.go: %v", err)
}
var mayStart *ast.FuncDecl
for _, decl := range f.Decls {
fn, ok := decl.(*ast.FuncDecl)
if !ok || fn.Name.Name != "MayStart" || fn.Recv == nil || len(fn.Recv.List) != 1 {
continue
}
if id, ok := fn.Recv.List[0].Type.(*ast.Ident); ok && id.Name == "bootDriveGate" {
mayStart = fn
}
}
if mayStart == nil {
t.Fatal("bootDriveGate.MayStart not found in main.go")
}
// It must call through to the shared predicate.
delegates := false
ast.Inspect(mayStart.Body, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok {
return true
}
sel, ok := call.Fun.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != "MayStart" {
return true
}
if x, ok := sel.X.(*ast.SelectorExpr); ok && x.Sel.Name == "drive" {
delegates = true
}
return true
})
if !delegates {
t.Fatal("bootDriveGate.MayStart no longer delegates to the shared driveStartGate — the boot " +
"sweep and the app-stop crash recovery would each carry their own drive logic, and the " +
"two can then disagree about whether an app may start (R-174)")
}
}
// --- R-158 / R-167 seams: both new alerts must be WIRED in production ---------------------------
// TestMainWiresTheUnitCaptureAlert pins Part 1's seam. `SetUnitNotify` is nil-safe by design, so an
// unwired seam is not a crash — it is SILENTLY the pre-v0.191.0 behaviour, in which a per-app Tier-1
// capture failure is a `[WARN]` line and reaches no hub channel at all. Every behavioural test in
// internal/backup injects its own callback and passes with the production wiring gone, which is
// exactly the hole this closes. THIS PROJECT'S COUNT OF "BUILT BUT NEVER WIRED" REACHES FIVE WITH
// R-158 — the defect being fixed here IS an instance of it.
func TestMainWiresTheUnitCaptureAlert(t *testing.T) {
names := callsInMain(t, mainBody(t))
if indexOfCall(names, "SetUnitNotify") < 0 {
t.Fatal("func main() no longer calls backupMgr.SetUnitNotify — a per-app recovery-unit " +
"capture failure would reach no hub channel, which is R-158 un-fixed (the seam built " +
"and left disconnected, for the fifth time in this project)")
}
if indexOfCall(names, "NotifyRecoveryUnitCaptureFailed") < 0 {
t.Fatal("main.go no longer calls NotifyRecoveryUnitCaptureFailed — the seam is wired to " +
"something that pushes no event, which looks identical to a working alert from inside " +
"internal/backup")
}
}
// TestMainWiresTheFillWatcher pins Part 2's seam. Three separate things can be dropped and each one
// silently reverts the customer to "nothing warns before a disk fills": the watcher can go
// unconstructed, its notify can go unwired (the Watcher is nil-safe), or it can never be scheduled.
func TestMainWiresTheFillWatcher(t *testing.T) {
body := mainBody(t)
names := callsInMain(t, body)
if indexOfCall(names, "New") < 0 || !assignsIdent(body, "fillWatcher") {
t.Fatal("func main() no longer constructs the fill watcher — nothing warns the customer " +
"before a filesystem fills (R-167, decision D-c's customer half)")
}
if indexOfCall(names, "SetNotify") < 0 {
t.Fatal("func main() no longer calls SetNotify on the fill watcher — the Watcher is nil-safe, " +
"so it would run the checks, update its state, log, and tell the CUSTOMER nothing")
}
// It must actually be scheduled: a watcher nobody calls is a watcher that never fires.
scheduled := false
ast.Inspect(body, func(n ast.Node) bool {
call, ok := n.(*ast.CallExpr)
if !ok || len(call.Args) == 0 {
return true
}
sel, ok := call.Fun.(*ast.SelectorExpr)
if !ok || (sel.Sel.Name != "Daily" && sel.Sel.Name != "Every") {
return true
}
lit, ok := call.Args[0].(*ast.BasicLit)
if ok && strings.Contains(lit.Value, "fill-watch") {
scheduled = true
}
return true
})
if !scheduled {
t.Fatal("the fill watcher is never registered on the scheduler — it would be constructed, " +
"wired, and never run, which is indistinguishable from a filesystem that never fills")
}
// It must ALSO run once at startup. Neither `Every` nor `Daily` fires on registration (both wait
// for their first tick), so a schedule-only wiring means a box that BOOTS with a filesystem
// already over the line stays silent for up to 24 hours — a real fault visible only after a
// deadline elapses, which is the R-100 shape. The hub's own checkers leave already-breached keys
// unseeded at init for exactly this reason.
if indexOfCall(names, "After") < 0 {
t.Fatal("nothing delays a startup fill check — see fillWatchStartupDelay")
}
startupRun := false
ast.Inspect(body, func(n ast.Node) bool {
g, ok := n.(*ast.GoStmt)
if !ok || g.Call == nil {
return true
}
lit, ok := g.Call.Fun.(*ast.FuncLit)
if !ok {
return true
}
var sawDelay, sawCheck bool
ast.Inspect(lit.Body, func(m ast.Node) bool {
if id, ok := m.(*ast.Ident); ok && id.Name == "fillWatchStartupDelay" {
sawDelay = true
}
if call, ok := m.(*ast.CallExpr); ok {
if sel, ok := call.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "Check" {
if x, ok := sel.X.(*ast.Ident); ok && x.Name == "fillWatcher" {
sawCheck = true
}
}
}
return true
})
if sawDelay && sawCheck {
startupRun = true
}
return true
})
if !startupRun {
t.Fatal("the fill watcher never runs at STARTUP — Daily/Every both wait for their first " +
"tick, so a box that boots with a full disk would not warn for up to 24 hours (the " +
"R-100 shape: a real fault visible only after a deadline elapses)")
}
}
// assignsIdent reports whether a block assigns to the named identifier.
func assignsIdent(body *ast.BlockStmt, want string) bool {
for _, n := range assignedIdentsIn(body) {
if n == want {
return true
}
}
return false
}
// assignedIdentsIn returns the names assigned to in a block (plain `=` and `:=`).
func assignedIdentsIn(body *ast.BlockStmt) []string {
var names []string
ast.Inspect(body, func(n ast.Node) bool {
as, ok := n.(*ast.AssignStmt)
if !ok {
return true
}
for _, lhs := range as.Lhs {
if id, ok := lhs.(*ast.Ident); ok {
names = append(names, id.Name)
}
}
return true
})
return names
}
@@ -0,0 +1,366 @@
package main
import (
"context"
"io"
"log"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/bootrecon"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
)
// R-157 mechanism A — the sweep that looked once.
//
// TIMING IS NOT TESTED BY SLEEPING (§10). The window's constants are package vars, so each test
// shrinks them to sub-millisecond values: the CONTRACT under test is "how many samples, and what
// ends the window", not "how long a second is". A test that waited real seconds would be slow,
// flaky, and would still not prove the contract.
// windowStacks is a StackProvider whose fleet CHANGES over successive GetStacks() calls — which is
// the whole point: the pre-v0.190.0 sweep sampled once and could not see a late settler.
type windowStacks struct {
// frames is the fleet as seen on each successive GetStacks() call; the last frame repeats.
frames [][]stacks.Stack
calls int
starts map[string]int
onStart func(*windowStacks, string)
refreshes int
refreshErr error
// cycle makes the fleet NEVER settle: frames repeat forever instead of the last one sticking.
// Required by the budget test — with frames that eventually stop changing, the window terminates
// by SETTLING even with the budget removed, so the red-proof would not reach the hang it exists
// to demonstrate.
cycle bool
}
func (w *windowStacks) GetStacks() []stacks.Stack {
i := w.calls
w.calls++
if i >= len(w.frames) {
if w.cycle {
i = i % len(w.frames)
} else {
i = len(w.frames) - 1
}
}
return w.frames[i]
}
func (w *windowStacks) RefreshStatus() error {
w.refreshes++
if w.refreshErr != nil {
return w.refreshErr
}
return nil
}
func (w *windowStacks) StartStack(name string) error {
if w.starts == nil {
w.starts = map[string]int{}
}
w.starts[name]++
if w.onStart != nil {
w.onStart(w, name)
}
return nil
}
// shrinkWindow makes the window fast and deterministic, and restores the shipped values after.
func shrinkWindow(t *testing.T, sample time.Duration, stableFor int, budget time.Duration) {
t.Helper()
os, ost, ob, osettle := bootReconcileSample, bootReconcileStableFor, bootReconcileBudget, bootReconcileSettle
t.Cleanup(func() {
bootReconcileSample, bootReconcileStableFor, bootReconcileBudget, bootReconcileSettle = os, ost, ob, osettle
})
bootReconcileSample, bootReconcileStableFor, bootReconcileBudget = sample, stableFor, budget
bootReconcileSettle = time.Millisecond
}
// captureSweep replaces the sweep with a recorder and returns the fleet it was handed.
func captureSweep(t *testing.T) *[][]stacks.Stack {
t.Helper()
orig := bootReconcileFn
t.Cleanup(func() { bootReconcileFn = orig })
var seen [][]stacks.Stack
bootReconcileFn = func(_ context.Context, mgr bootrecon.StackProvider, _ *log.Logger) bootrecon.Result {
seen = append(seen, mgr.GetStacks())
return bootrecon.Result{}
}
return &seen
}
func upStack(name string) stacks.Stack {
return stacks.Stack{
Name: name, Deployed: true, State: stacks.StateRunning,
Containers: []stacks.ContainerInfo{{Name: name, State: stacks.StateRunning}},
AppConfig: &stacks.AppConfig{Deployed: true, DesiredState: stacks.DesiredStateRunning},
}
}
// settlingLate is the R-157-A shape: at T+5s the app is still `starting` with its containers coming
// up, and it only comes to rest in a DOWN state later.
func settlingLate(name string) stacks.Stack {
return stacks.Stack{
Name: name, Deployed: true, State: stacks.StateStarting,
Containers: []stacks.ContainerInfo{{Name: name, State: stacks.StateStarting}},
AppConfig: &stacks.AppConfig{Deployed: true, DesiredState: stacks.DesiredStateRunning},
}
}
func settledDown(name string) stacks.Stack {
return stacks.Stack{
Name: name, Deployed: true, State: stacks.StateExited,
Containers: []stacks.ContainerInfo{{Name: name, State: stacks.StateExited}},
AppConfig: &stacks.AppConfig{Deployed: true, DesiredState: stacks.DesiredStateRunning},
}
}
// --- Group A / Scenario B — a late settler IS swept -----------------------------------------------
func TestBootWindow_LateSettlerIsSweptOnASettledFleet(t *testing.T) {
// The fleet is still moving for the first frames and settles only later. The sweep must run
// AFTER it settles and must be handed the SETTLED fleet — because the pre-v0.190.0 defect was a
// candidate set derived from a fleet that had not finished moving.
//
// RED-PROOF: restore the single-sweep shape (delete the sampling loop so runBootReconcile calls
// bootReconcileFn straight after the settle delay) and this test fails — the sweep is handed the
// `starting` frame, in which the app is not a down-state candidate at all.
// Demonstrated in REPORT.md §4.
shrinkWindow(t, time.Millisecond, 2, 500*time.Millisecond)
seen := captureSweep(t)
w := &windowStacks{frames: [][]stacks.Stack{
{settlingLate("immich")}, // T+5s: still coming up
{settlingLate("immich")},
{settledDown("immich")}, // settles into a down state only now
{settledDown("immich")},
{settledDown("immich")},
}}
runBootReconcile(context.Background(), w, log.New(io.Discard, "", 0))
if len(*seen) != 1 {
t.Fatalf("the sweep ran %d times, want exactly 1 — the window samples, it does not sweep per sample", len(*seen))
}
got := (*seen)[0]
if len(got) != 1 || got[0].State != stacks.StateExited {
t.Fatalf("the sweep was handed state=%v, want the SETTLED (exited) fleet — a candidate set "+
"derived from a still-moving fleet is exactly the R-157 mechanism-A defect", got)
}
}
func TestBootWindow_SweepRunsExactlyOnceEvenOnAQuietBoot(t *testing.T) {
shrinkWindow(t, time.Millisecond, 2, 500*time.Millisecond)
seen := captureSweep(t)
w := &windowStacks{frames: [][]stacks.Stack{{upStack("bookstack")}}}
runBootReconcile(context.Background(), w, log.New(io.Discard, "", 0))
if len(*seen) != 1 {
t.Fatalf("sweeps=%d, want exactly 1 on a quiet boot", len(*seen))
}
}
// --- Group B / Scenario C — the window TERMINATES -------------------------------------------------
func TestBootWindow_BudgetEndsAForeverChangingFleet(t *testing.T) {
// A fleet that never stops changing must not sample forever. The budget ends it, the sweep runs
// once anyway (a churning box is exactly the box that needs it), and the log SAYS the budget
// ended it — "settled and found nothing" and "ran out of time" are different facts.
//
// RED-PROOF: remove the `time.Since(started) < bootReconcileBudget` loop condition and this test
// hangs — the unbounded-loop shape §5 bans. Demonstrated in REPORT.md §4 (observed as a timeout).
shrinkWindow(t, time.Millisecond, 3, 30*time.Millisecond)
seen := captureSweep(t)
var buf strings.Builder
// Every frame differs, so `stable` can never reach stableFor.
frames := make([][]stacks.Stack, 0, 200)
for i := 0; i < 200; i++ {
s := upStack("immich")
s.Containers = make([]stacks.ContainerInfo, i%7) // container count changes every sample
frames = append(frames, []stacks.Stack{s})
}
w := &windowStacks{frames: frames, cycle: true}
done := make(chan struct{})
go func() {
runBootReconcile(context.Background(), w, log.New(&buf, "", 0))
close(done)
}()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("runBootReconcile did not terminate on a forever-changing fleet — this is the " +
"unbounded restart-loop shape the package's own boundary forbids")
}
if len(*seen) != 1 {
t.Fatalf("sweeps=%d, want exactly 1 after the budget expired", len(*seen))
}
if out := buf.String(); !strings.Contains(out, "budget") {
t.Fatalf("the log does not say the BUDGET ended the window, so a churning boot reads like a "+
"quiet one:\n%s", out)
}
}
func TestBootWindow_SettledPathSaysSettled(t *testing.T) {
shrinkWindow(t, time.Millisecond, 2, 500*time.Millisecond)
captureSweep(t)
var buf strings.Builder
w := &windowStacks{frames: [][]stacks.Stack{{upStack("docmost")}}}
runBootReconcile(context.Background(), w, log.New(&buf, "", 0))
out := buf.String()
if !strings.Contains(out, "settled") {
t.Fatalf("a settled window must say so — otherwise it is indistinguishable from a budget "+
"expiry:\n%s", out)
}
if strings.Contains(out, "budget") {
t.Fatalf("a settled window must NOT claim the budget ended it:\n%s", out)
}
}
func TestBootWindow_CancelledContextStopsImmediately(t *testing.T) {
shrinkWindow(t, time.Millisecond, 3, time.Second)
seen := captureSweep(t)
ctx, cancel := context.WithCancel(context.Background())
cancel()
runBootReconcile(ctx, &windowStacks{frames: [][]stacks.Stack{{upStack("x")}}}, log.New(io.Discard, "", 0))
if len(*seen) != 0 {
t.Fatalf("the sweep ran %d times on a cancelled context, want 0", len(*seen))
}
}
// --- Group C / Scenario D — a customer's Stop survives the WIDENED window -------------------------
func TestBootWindow_CustomerStoppedAppSurvivesEveryPass(t *testing.T) {
// THE REGRESSION THIS TASK COULD INTRODUCE. A longer window means more chances to resurrect an
// app the customer deliberately stopped. It must survive the whole window — this drives the REAL
// bootrecon sweep (not the captured stub), so the desired-state check is genuinely exercised.
//
// RED-PROOF: drop the DesiredStateStopped branch from isBootOrphan (make it fall through to the
// running case) and this test fails with a start count of 1. Demonstrated in REPORT.md §4.
shrinkWindow(t, time.Millisecond, 2, 200*time.Millisecond)
stopped := stacks.Stack{
Name: "nextcloud", Deployed: true, State: stacks.StateStopped, Containers: nil,
AppConfig: &stacks.AppConfig{Deployed: true, DesiredState: stacks.DesiredStateStopped},
}
// The fleet churns around it, so the window runs many passes before settling.
frames := [][]stacks.Stack{
{stopped, settlingLate("immich")},
{stopped, settlingLate("immich")},
{stopped, settledDown("immich")},
{stopped, upStack("immich")},
{stopped, upStack("immich")},
{stopped, upStack("immich")},
}
w := &windowStacks{frames: frames, onStart: func(w *windowStacks, _ string) {}}
runBootReconcile(context.Background(), w, log.New(io.Discard, "", 0))
if n := w.starts["nextcloud"]; n != 0 {
t.Fatalf("the customer-stopped app was started %d time(s) by the widened window — this is the "+
"regression a longer window makes possible and it is the worst outcome available here", n)
}
}
// --- §8.3 — a late recovery is REPORTED, never hidden ---------------------------------------------
func TestRecordLateRecovery_WarnsWhenTheGraceHasAlreadyExpired(t *testing.T) {
var buf strings.Builder
lg := log.New(&buf, "", 0)
// started far enough back that settle + elapsed exceeds the 90 s grace
recordLateRecovery(lg, time.Now().Add(-(deadAppBootGrace + 10*time.Second)), bootrecon.Result{Recovered: []string{"immich"}})
out := buf.String()
if !strings.Contains(out, "LATE RECOVERY") || !strings.Contains(out, "immich") {
t.Fatalf("a recovery past the dead-app grace must be reported by name — otherwise a stale "+
"alarm stands with no counter-evidence (§8.3):\n%s", out)
}
}
func TestRecordLateRecovery_SilentInsideTheGrace(t *testing.T) {
var buf strings.Builder
recordLateRecovery(log.New(&buf, "", 0), time.Now(), bootrecon.Result{Recovered: []string{"immich"}})
if buf.Len() != 0 {
t.Fatalf("a recovery INSIDE the grace must stay silent — that is what makes a successful "+
"recovery invisible to the customer:\n%s", buf.String())
}
}
func TestRecordLateRecovery_SilentWhenNothingRecovered(t *testing.T) {
var buf strings.Builder
recordLateRecovery(log.New(&buf, "", 0), time.Now().Add(-time.Hour), bootrecon.Result{})
if buf.Len() != 0 {
t.Fatalf("nothing was recovered, so there is nothing late to report:\n%s", buf.String())
}
}
// --- The window's constants must fit the grace they are justified against -------------------------
func TestBootWindow_CommonCaseFitsInsideTheDeadAppGrace(t *testing.T) {
// The comment on the window constants justifies them against deadAppBootGrace. A comment
// asserting an invariant needs a test pinning it, or it is a wish.
common := bootReconcileSettle + bootReconcileBudget + bootrecon.DefaultRetryDelay
if common > deadAppBootGrace {
t.Fatalf("settle(%s) + budget(%s) + one retry(%s) = %s exceeds the %s dead-app grace — the "+
"COMMON case must stay silent, or every slow boot alerts",
bootReconcileSettle, bootReconcileBudget, bootrecon.DefaultRetryDelay, common, deadAppBootGrace)
}
if bootReconcileSample <= 0 || bootReconcileStableFor < 2 {
t.Fatalf("sample=%s stableFor=%d — one sample cannot distinguish 'settled' from 'sampled "+
"between two docker events'", bootReconcileSample, bootReconcileStableFor)
}
}
// --- the sample must observe REALITY, not the Manager's cache ------------------------------------
func TestBootWindow_EverySampleRefreshesTheStatus(t *testing.T) {
// FOUND BY LIVE VALIDATION, not review. GetStacks() returns the Manager's in-memory map, which
// the scheduler refreshes on its own 10 s cadence. Sampling every 5 s WITHOUT refreshing means two
// consecutive samples can be identical because the cache did not update — so the window declares
// "settled" on stale data and sweeps on a picture of the box from up to 10 s ago. On 9201 a
// container removed ~5 s before the window closed was still in the sampled fleet, and the sweep
// logged "no boot-orphaned apps" for an app that had none.
//
// RED-PROOF: delete the `_ = mgr.RefreshStatus()` line from sampleBootFleet and this test fails
// with refreshes=0. Demonstrated in REPORT.md §4.
shrinkWindow(t, time.Millisecond, 3, 500*time.Millisecond)
captureSweep(t)
w := &windowStacks{frames: [][]stacks.Stack{{upStack("immich")}}}
runBootReconcile(context.Background(), w, log.New(io.Discard, "", 0))
if w.refreshes < 3 {
t.Fatalf("the window refreshed %d time(s) for %d samples — every sample must observe reality, "+
"or 'settled' can mean 'the cache did not update'", w.refreshes, w.calls)
}
// calls includes ONE extra GetStacks from the captured sweep itself, which does not sample.
if w.refreshes != w.calls-1 {
t.Fatalf("refreshes=%d but samples=%d — each sample must refresh exactly once before reading",
w.refreshes, w.calls-1)
}
}
func TestBootWindow_RefreshErrorDoesNotStopTheWindow(t *testing.T) {
// A boot window that cannot reach docker is exactly when a stale verdict is most dangerous, but
// giving up entirely would leave the sweep un-run. Degrade, do not abort.
shrinkWindow(t, time.Millisecond, 2, 200*time.Millisecond)
seen := captureSweep(t)
w := &windowStacks{frames: [][]stacks.Stack{{upStack("immich")}}, refreshErr: errRefresh{}}
runBootReconcile(context.Background(), w, log.New(io.Discard, "", 0))
if len(*seen) != 1 {
t.Fatalf("sweeps=%d, want 1 — a refresh error must not abort the window", len(*seen))
}
}
type errRefresh struct{}
func (errRefresh) Error() string { return "docker unreachable" }
@@ -0,0 +1,125 @@
package main
import (
"time"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/notify"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
"gitea.dooplex.hu/admin/felhom-controller/internal/web"
)
// v0.164.0: classifyRunStates is the single fix-3 derivation point. A deliberate user stop
// (StateStopped) must NOT alarm — it is excluded from both the banner dead-list and the notifier
// Down-set — while every genuine fault (StateExited / StateDegraded) keeps alerting byte-identically.
// Invariants behind the suppression are documented at classifyRunStates (I1: compose down ⇒ zero
// containers ⇒ StateStopped; I2: P2 census — all catalog services unless-stopped ⇒ faults never rest
// at stopped).
func stack(name string, st stacks.ContainerState, deployed, deploying bool) stacks.Stack {
return stacks.Stack{
Name: name,
Meta: stacks.Metadata{DisplayName: name},
State: st,
Deployed: deployed,
Deploying: deploying,
}
}
func downByName(states []notify.AppRunState) map[string]bool {
m := map[string]bool{}
for _, s := range states {
m[s.Name] = s.Down
}
return m
}
func deadNames(dead []web.DeadApp) map[string]bool {
m := map[string]bool{}
for _, d := range dead {
m[d.Name] = true
}
return m
}
// Group A (Scenario A) — suppression. Over a [running, stopped, exited, degraded] fixture, the dead
// list is EXACTLY {exited, degraded} and the Down flags are {false, false, true, true}: the stopped
// app is silent, the two faults still alarm.
//
// COMPANION red-proof: revert the filter to bare `stacks.IsDownState(st.State)` (drop the
// `&& st.State != stacks.StateStopped` guard) → stopped reports Down=true and enters the dead list →
// both the dead-set and the Down-flag assertions below fail. (Verified by hand-editing the seam.)
func TestClassifyRunStates_StoppedIsSuppressed(t *testing.T) {
sts := []stacks.Stack{
stack("radarr", stacks.StateRunning, true, false),
stack("cwa", stacks.StateStopped, true, false),
stack("immich", stacks.StateExited, true, false),
stack("nextcloud", stacks.StateDegraded, true, false),
}
dead, states := classifyRunStates(sts, nil, nil, time.Now())
gotDead := deadNames(dead)
if len(gotDead) != 2 || !gotDead["immich"] || !gotDead["nextcloud"] {
t.Fatalf("dead list must be exactly {immich(exited), nextcloud(degraded)}, got %+v", dead)
}
if gotDead["cwa"] {
t.Errorf("a deliberately stopped app must NOT be in the dead list (no banner)")
}
if gotDead["radarr"] {
t.Errorf("a running app must never be in the dead list")
}
down := downByName(states)
want := map[string]bool{"radarr": false, "cwa": false, "immich": true, "nextcloud": true}
if len(down) != len(want) {
t.Fatalf("every deployed app must have a run state, got %+v", down)
}
for name, w := range want {
if down[name] != w {
t.Errorf("Down[%s] = %v, want %v (stopped ⇒ false ⇒ no app_start_failed event)", name, down[name], w)
}
}
}
// Group B (Scenario B) — fault parity. With only exited + degraded present, BOTH surface in the dead
// list AND both report Down=true — byte-identical to v0.163.1 for every non-stopped down state. The
// suppression touches stopped and nothing else.
func TestClassifyRunStates_FaultParity(t *testing.T) {
sts := []stacks.Stack{
stack("immich", stacks.StateExited, true, false),
stack("nextcloud", stacks.StateDegraded, true, false),
}
dead, states := classifyRunStates(sts, nil, nil, time.Now())
gotDead := deadNames(dead)
if len(gotDead) != 2 || !gotDead["immich"] || !gotDead["nextcloud"] {
t.Fatalf("both faults must appear in the dead list, got %+v", dead)
}
down := downByName(states)
if !down["immich"] || !down["nextcloud"] {
t.Fatalf("both faults must report Down=true, got %+v", down)
}
// State strings must ride through to the banner unchanged (banner shows "(exited)"/"(degraded)").
byName := map[string]string{}
for _, d := range dead {
byName[d.Name] = d.State
}
if byName["immich"] != string(stacks.StateExited) || byName["nextcloud"] != string(stacks.StateDegraded) {
t.Errorf("dead-app State must carry the raw aggregate state, got %+v", byName)
}
}
// Deploying and undeployed stacks are skipped entirely (unchanged fix-3 behavior).
func TestClassifyRunStates_SkipsDeployingAndUndeployed(t *testing.T) {
sts := []stacks.Stack{
stack("mid", stacks.StateDeploying, true, true), // mid-deploy → skipped
stack("gone", stacks.StateExited, false, false), // not deployed → skipped
}
dead, states := classifyRunStates(sts, nil, nil, time.Now())
if len(dead) != 0 || len(states) != 0 {
t.Fatalf("deploying and undeployed stacks must be skipped, got dead=%+v states=%+v", dead, states)
}
}
@@ -0,0 +1,139 @@
package main
import (
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
)
// C9-F2 — a SUSTAINED `restarting` is a crash loop and must alarm; a BRIEF one must not.
//
// The defect: `IsDownState` excludes `restarting` as "self-recovering", but for the catalog's
// standard `restart: unless-stopped` Docker retries forever, so a crash loop sat in `restarting`
// indefinitely and was counted as working. Campaign 9 watched docmost loop for nine minutes
// (restartcount 18) while the F-OBS heartbeat printed "4 deployed app(s) evaluated, 0 currently down".
//
// The whole design tension is that B must keep passing while A does: an alarm that fires on every
// deploy is one the operator learns to ignore.
// restartingSince builds a deployed stack that has been restarting since `since`.
func restartingSince(name string, since time.Time) stacks.Stack {
s := stack(name, stacks.StateRestarting, true, false)
s.RestartingSince = since
return s
}
// SCENARIO A — a crash loop alarms. A stack restarting for longer than the threshold enters BOTH the
// banner dead-list and the notifier Down-set, so app_start_failed can fire.
//
// RED-PROOF (observed): drop `|| crashLooping` from the `down` expression in classifyRunStates →
//
// crashloop_classify_test.go:52: docmost is NOT in the Down-set — a crash loop is silent (this is C9-F2)
// crashloop_classify_test.go:55: docmost is NOT in the banner dead-list
func TestClassifyRunStates_SustainedRestartingAlarms(t *testing.T) {
now := time.Now()
sts := []stacks.Stack{
stack("paperless-ngx", stacks.StateRunning, true, false),
restartingSince("docmost", now.Add(-9*time.Minute)), // the Campaign 9 observation, exactly
}
dead, states := classifyRunStates(sts, nil, nil, now)
if !downByName(states)["docmost"] {
t.Errorf("docmost is NOT in the Down-set — a crash loop is silent (this is C9-F2)")
}
if !deadNames(dead)["docmost"] {
t.Errorf("docmost is NOT in the banner dead-list")
}
if downByName(states)["paperless-ngx"] {
t.Errorf("a healthy app was dragged down with it")
}
}
// SCENARIO B — a normal deploy or update does NOT alarm. `docker compose up -d` passes through
// restarting; alarming there would page the operator on every routine operation, fleet-wide.
//
// This is the test that must fail against the naive fix. RED-PROOF (observed): add StateRestarting
// to IsDownState instead of using the threshold →
//
// crashloop_classify_test.go:78: a BRIEFLY restarting app alarms — every deploy and update would page the operator
func TestClassifyRunStates_BriefRestartingIsSilent(t *testing.T) {
now := time.Now()
sts := []stacks.Stack{
restartingSince("mealie", now.Add(-30*time.Second)), // mid-deploy
restartingSince("ghost", now.Add(-2*time.Minute)), // slow image pull, still normal
}
dead, states := classifyRunStates(sts, nil, nil, now)
for _, name := range []string{"mealie", "ghost"} {
if downByName(states)[name] {
t.Errorf("a BRIEFLY restarting app alarms (%s) — every deploy and update would page the operator", name)
}
}
if len(dead) != 0 {
t.Errorf("banner dead-list should be empty during normal restarts, got %v", deadNames(dead))
}
}
// The boundary itself, asserted from both sides so the threshold cannot drift silently.
func TestCrashLooping_ThresholdBoundary(t *testing.T) {
now := time.Now()
for _, tc := range []struct {
name string
age time.Duration
want bool
}{
{"just under the threshold", 4*time.Minute + 59*time.Second, false},
{"exactly at the threshold", 5 * time.Minute, true},
{"well past it", 30 * time.Minute, true},
} {
s := restartingSince("app", now.Add(-tc.age))
if got := s.CrashLooping(now); got != tc.want {
t.Errorf("%s: CrashLooping(age=%s) = %v, want %v", tc.name, tc.age, got, tc.want)
}
}
// A stack that is not restarting is never a crash loop, however old the stamp.
s := stack("app", stacks.StateRunning, true, false)
s.RestartingSince = now.Add(-time.Hour)
if s.CrashLooping(now) {
t.Error("a RUNNING stack reported as crash-looping — the state test is missing")
}
// A zero stamp is "not yet observed restarting", never a crash loop — this is what makes the
// first scan after a controller restart silent instead of alarming on everything at once.
z := stack("app", stacks.StateRestarting, true, false)
if z.CrashLooping(now) {
t.Error("a zero RestartingSince reported as crash-looping — a controller restart would alarm fleet-wide")
}
}
// SCENARIO C — R-97b's quiesce suppression still wins inside its window. A stack the backup stopped
// and is restarting must stay silent while suppressed, even if its restarting run is old enough to
// qualify. The window EXPIRES, so a genuinely dead app still alarms afterwards — proven by the
// second half of this test.
//
// RED-PROOF (observed): drop `&& !quiesced[st.Name]` from the `down` expression →
//
// crashloop_classify_test.go:129: a quiesced stack alarms — every backup would page the customer
func TestClassifyRunStates_QuiesceSuppressionBeatsCrashLoop(t *testing.T) {
now := time.Now()
sts := []stacks.Stack{restartingSince("docmost", now.Add(-9*time.Minute))}
// Inside the R-97b window.
_, states := classifyRunStates(sts, map[string]bool{"docmost": true}, nil, now)
if downByName(states)["docmost"] {
t.Errorf("a quiesced stack alarms — every backup would page the customer")
}
// Window expired (the stack is no longer reported as suppressed): the same stack must now alarm.
dead, states := classifyRunStates(sts, nil, nil, now)
if !downByName(states)["docmost"] {
t.Errorf("suppression outlived its window — a genuinely dead app stayed silent (R-97b's own warning)")
}
if !deadNames(dead)["docmost"] {
t.Errorf("suppression outlived its window for the banner too")
}
}
@@ -0,0 +1,97 @@
package main
import (
"bytes"
"log"
"strings"
"testing"
)
// F-OBS (Campaign 8): on a default `info`-level box there was NO positive observable that
// `deadapp-check` had run. Its per-cycle scheduler line goes through Scheduler.dbg(), which is gated
// on logging.level==debug and therefore never PRODUCED on a default box — so it could not even reach
// the always-DEBUG ring — and a 30 s interval also puts the job on the scheduler's quiet path.
//
// "No alarms" was therefore indistinguishable from "the detector never ran", which is exactly the
// fallacy this project now has a standing rule against, and it undermines confidence in the
// F-CRIT-1 fix in the field.
//
// Scenario F — the observable must appear AT INFO LEVEL. These tests assert the emitted LINE, not
// merely that a function was called; asserting the call would reproduce the original mistake.
// RED-PROOF: delete the logger.Printf in noteDeadAppScan (or drop the whole call from the job
// closure) → every case below sees an empty buffer and this fails with
// "no observable emitted at scan 20 — silence is indistinguishable from not running".
func TestNoteDeadAppScan_EmitsAtInfoLevel(t *testing.T) {
var buf bytes.Buffer
lg := log.New(&buf, "", 0)
noteDeadAppScan(lg, deadAppHeartbeatEvery, 7, 2)
out := buf.String()
if out == "" {
t.Fatalf("no observable emitted at scan %d — silence is indistinguishable from not running", deadAppHeartbeatEvery)
}
if !strings.Contains(out, "[INFO]") {
t.Errorf("the observable is not at INFO level, so a default `logging.level: info` box would never see it:\n%s", out)
}
if !strings.Contains(out, "[deadapp]") {
t.Errorf("the observable does not identify the check that produced it:\n%s", out)
}
// it must carry WHAT IT SAW, not just "I ran" — an operator needs to distinguish
// "running and everything is up" from "running and 2 apps are down".
for _, want := range []string{"scans since boot", "evaluated", "currently down"} {
if !strings.Contains(out, want) {
t.Errorf("the observable omits %q — it proves the check ran but not what it found:\n%s", want, out)
}
}
}
// It must NOT be a line per run. At a 30 s cadence that is 2880 lines/day, which is precisely why
// the original author chose silence — so a fix that floods is not a fix.
//
// RED-PROOF: change the guard to `scans%1 != 0` (i.e. emit every run) → this fails with
// "emitted 60 lines across 60 scans — that is the flood that made silence attractive".
func TestNoteDeadAppScan_IsASummaryNotAFlood(t *testing.T) {
var buf bytes.Buffer
lg := log.New(&buf, "", 0)
const scans = 60
for i := 1; i <= scans; i++ {
noteDeadAppScan(lg, i, 3, 0)
}
got := strings.Count(buf.String(), "[deadapp] check alive")
want := scans / deadAppHeartbeatEvery
if got == scans {
t.Fatalf("emitted %d lines across %d scans — that is the flood that made silence attractive", got, scans)
}
if got != want {
t.Errorf("emitted %d heartbeat lines across %d scans, want %d (one per %d)", got, scans, want, deadAppHeartbeatEvery)
}
}
// The cadence must be frequent enough that a STALLED detector is obvious well inside the 180 s alarm
// grace this check feeds. 20 scans x 30 s = 10 min; if someone widens it to hours the observable
// stops being useful as a liveness signal, and this is the tripwire.
func TestDeadAppHeartbeatEvery_StaysUsefulAsALivenessSignal(t *testing.T) {
const scanInterval = 30 // seconds, matching sched.Every("deadapp-check", 30*time.Second, ...)
periodSec := deadAppHeartbeatEvery * scanInterval
if periodSec > 15*60 {
t.Errorf("heartbeat period is %ds (>15min) — too sparse to notice a stalled detector", periodSec)
}
if deadAppHeartbeatEvery < 2 {
t.Errorf("heartbeat every %d scans is a per-run flood", deadAppHeartbeatEvery)
}
}
// Off-cadence scans stay quiet, and a nil logger is tolerated (the job closure must never panic).
func TestNoteDeadAppScan_QuietOffCadenceAndNilSafe(t *testing.T) {
var buf bytes.Buffer
lg := log.New(&buf, "", 0)
noteDeadAppScan(lg, deadAppHeartbeatEvery-1, 1, 0)
if buf.Len() != 0 {
t.Errorf("emitted off-cadence:\n%s", buf.String())
}
noteDeadAppScan(nil, deadAppHeartbeatEvery, 1, 0) // must not panic
}
@@ -0,0 +1,124 @@
package main
import (
"time"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
)
// F-CRIT-1 cause 2 (Campaign 8): `classifyRunStates` whitelisted StateStopped on invariant I1
// ("StateStopped means the USER stopped it"). The quiesce loop broke I1 by stopping stacks via the
// same `docker compose down` path, so a stack quiesce stopped and then FAILED to restart was also
// StateStopped — and was whitelisted into total silence. Live evidence: a customer app dead
// indefinitely, no banner, no event, no email, while the dead-app scanner ran 11 times over it.
//
// The two cases are byte-identical on the Docker side. The ONLY thing that separates them is that
// the quiesce loop knows it tried to restart and could not — `failedRestart` is that knowledge.
// Scenario A (cause 2) — a stack quiesce failed to restart MUST alarm, despite being StateStopped.
//
// RED-PROOF: restore the unconditional whitelist (`down := IsDownState(st.State) &&
// st.State != stacks.StateStopped && !quiesced[st.Name]`) → immich reports Down=false and stays out
// of the dead list, and this fails with "a stack that FAILED to restart is silent".
func TestClassifyRunStates_FailedRestartAlarmsDespiteStateStopped(t *testing.T) {
sts := []stacks.Stack{
stack("bookstack", stacks.StateRunning, true, false),
stack("immich", stacks.StateStopped, true, false), // quiesce stopped it; restart FAILED
}
failed := map[string]bool{"immich": true}
dead, states := classifyRunStates(sts, nil, failed, time.Now())
if !downByName(states)["immich"] {
t.Error("a stack that FAILED to restart is silent (Down=false) — this is F-CRIT-1")
}
if !deadNames(dead)["immich"] {
t.Error("a stack that FAILED to restart is absent from the dashboard dead-list — this is F-CRIT-1")
}
if downByName(states)["bookstack"] {
t.Error("a healthy running stack was marked down")
}
}
// Scenario B — a DELIBERATE user stop must still be silent. This pins v0.164.0 and is what stops
// the fix above from becoming a regression.
//
// RED-PROOF: make the whitelist unconditional in the other direction (drop the `&& !failedRestart`
// term, i.e. treat every StateStopped as a failed restart) → cwa alarms and this fails with
// "a deliberate user stop alarmed".
func TestClassifyRunStates_UserStopStillSilent(t *testing.T) {
sts := []stacks.Stack{
stack("cwa", stacks.StateStopped, true, false), // the user stopped this from the UI
stack("immich", stacks.StateStopped, true, false),
}
// only immich failed to restart; cwa was never touched by a quiesce
failed := map[string]bool{"immich": true}
dead, states := classifyRunStates(sts, nil, failed, time.Now())
down := downByName(states)
if down["cwa"] || deadNames(dead)["cwa"] {
t.Error("a deliberate user stop alarmed — that is the v0.164.0 regression this must not reintroduce")
}
if !down["immich"] {
t.Error("the failed restart went silent")
}
}
// Scenario B, stronger form — with NO failed restarts at all, behaviour is byte-identical to
// v0.164.0: every StateStopped is silent.
func TestClassifyRunStates_NoFailedRestartsIsV0164Behaviour(t *testing.T) {
sts := []stacks.Stack{
stack("radarr", stacks.StateRunning, true, false),
stack("cwa", stacks.StateStopped, true, false),
stack("immich", stacks.StateExited, true, false),
stack("nextcloud", stacks.StateDegraded, true, false),
}
dead, states := classifyRunStates(sts, nil, nil, time.Now())
down := downByName(states)
if down["cwa"] {
t.Error("stopped alarmed with no failed restarts — v0.164.0 behaviour broken")
}
if !down["immich"] || !down["nextcloud"] {
t.Error("a genuine fault (exited/degraded) stopped alarming")
}
if got := len(deadNames(dead)); got != 2 {
t.Errorf("dead list has %d entries, want exactly {immich, nextcloud}", got)
}
}
// Scenario C — during the R-97b grace window the stack is suppressed even if its restart failed.
// The grace exists so a slow-starting app is not called dead; it EXPIRES, and the alarm follows.
//
// RED-PROOF: drop the `&& !quiesced[st.Name]` term → the app alarms mid-restart on every normal
// backup, which is the false-alarm R-97b was built to remove.
func TestClassifyRunStates_GraceWindowStillSuppresses(t *testing.T) {
sts := []stacks.Stack{stack("immich", stacks.StateStopped, true, false)}
quiesced := map[string]bool{"immich": true} // still inside quiesceAlarmGrace
failed := map[string]bool{"immich": true} // and we already know the restart failed
dead, states := classifyRunStates(sts, quiesced, failed, time.Now())
if downByName(states)["immich"] {
t.Error("alarmed while still inside the grace window — R-97b Scenario E broken")
}
if len(dead) != 0 {
t.Errorf("dead list not empty during grace: %v", deadNames(dead))
}
}
// An undeployed or mid-deploy stack is never classified, failed restart or not.
func TestClassifyRunStates_UndeployedIgnored(t *testing.T) {
sts := []stacks.Stack{
stack("ghost", stacks.StateStopped, false, false),
stack("deploying", stacks.StateStopped, true, true),
}
dead, states := classifyRunStates(sts, nil, map[string]bool{"ghost": true, "deploying": true}, time.Now())
if len(dead) != 0 || len(states) != 0 {
t.Errorf("undeployed/deploying stacks were classified: dead=%v states=%v", deadNames(dead), states)
}
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,30 @@
package main
import (
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/quiesce"
)
// R-97a — REACHABILITY, not behaviour.
//
// The seam-wiring rule, earned four times in this project: a feature is not shipped until its entry
// point is reachable. `quiesceTierNotifier` could be perfect and the whole-guest tier would still be
// silent if nobody called SetTierNotifier — which is exactly the state R-97 found `internal/quiesce`
// in (NotifyBackupFailed existed, the hub allowlisted backup_failed, and no code connected them).
//
// This asserts the adapter SATISFIES the interface the loop requires. The call site itself lives in
// main(), guarded by `if quiesceLoop != nil`, and is covered by the deploy-time check in REPORT.md.
func TestQuiesceTierNotifierIsWired(t *testing.T) {
var _ quiesce.TierNotifier = quiesceTierNotifier{}
// And it must not panic on a nil notifier — main() constructs it with a real one, but a future
// refactor that reorders startup must fail loudly here rather than at 03:00 on a customer box.
defer func() {
if r := recover(); r != nil {
t.Fatalf("the adapter panicked with a nil notifier: %v", r)
}
}()
var n quiesceTierNotifier
_ = n
}
@@ -0,0 +1,22 @@
package main
import "gitea.dooplex.hu/admin/felhom-controller/internal/quiesce"
// COMPILE-TIME WITNESSES for OPTIONAL interfaces satisfied by a RUNTIME type assertion.
//
// Moved here from a _test.go file (R-88 Part 2) on purpose: a witness in a test fires on `go test`
// and `go vet`, but NOT on `go build` alone. The failure it guards against — a signature change that
// silently breaks an interface nobody checks at compile time — is exactly the kind that gets pushed
// by a build-only step.
//
// THE INCIDENT THIS PREVENTS, which already happened once: when `TieredBackend.DueFor` gained a
// return value during R-88 Part 2, `quiesceBackend` stopped satisfying the interface and the whole
// repo still BUILT AND VETTED CLEAN, because `resolveDueTiers` only ever asserts it at runtime
// (`l.backend.(TieredBackend)`). A failed assertion silently falls back to the untargeted
// single-tier path — so every box would have quietly lost R-82's multi-tier backups, with no error
// anywhere. It was caught by accident, not by the toolchain.
//
// THIS DOES NOT MAKE THE INTERFACE REQUIRED. Optionality is deliberate: it is what lets a new
// controller meet an old agent, and what `resolveDueTiers` degrades through on purpose. The witness
// pins the IMPLEMENTATION, not the CONTRACT.
var _ quiesce.TieredBackend = quiesceBackend{}
+1
View File
@@ -5,6 +5,7 @@ go 1.24.0
require ( require (
github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6 github.com/emersion/go-sasl v0.0.0-20241020182733-b788ff22d5a6
github.com/emersion/go-smtp v0.24.0 github.com/emersion/go-smtp v0.24.0
github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e
golang.org/x/crypto v0.31.0 golang.org/x/crypto v0.31.0
gopkg.in/yaml.v3 v3.0.1 gopkg.in/yaml.v3 v3.0.1
modernc.org/sqlite v1.45.0 modernc.org/sqlite v1.45.0
+4
View File
@@ -16,6 +16,8 @@ github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOF
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls= github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo= github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e h1:MRM5ITcdelLK2j1vwZ3Je0FKVCfqOLp5zO6trqMLYs0=
github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e/go.mod h1:XV66xRDqSt+GTGFMVlhk3ULuV0y9ZmzeVGR4mloJI3M=
golang.org/x/crypto v0.31.0 h1:ihbySMvVjLAeSH1IbfcRTkD/iNscyz8rGzjF/E5hV6U= golang.org/x/crypto v0.31.0 h1:ihbySMvVjLAeSH1IbfcRTkD/iNscyz8rGzjF/E5hV6U=
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk= golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 h1:mgKeJMpvi0yx/sU5GsxQ7p6s2wtOnGAHZWCHUM4KGzY= golang.org/x/exp v0.0.0-20251023183803-a4bb9ffd2546 h1:mgKeJMpvi0yx/sU5GsxQ7p6s2wtOnGAHZWCHUM4KGzY=
@@ -27,6 +29,8 @@ golang.org/x/sync v0.17.0/go.mod h1:9KTHXmSnoGruLpwFjVSX0lNNA75CykiMECbovNTZqGI=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.37.0 h1:fdNQudmxPjkdUTPnLn5mdQv7Zwvbvpaxqs831goi9kQ= golang.org/x/sys v0.37.0 h1:fdNQudmxPjkdUTPnLn5mdQv7Zwvbvpaxqs831goi9kQ=
golang.org/x/sys v0.37.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks= golang.org/x/sys v0.37.0/go.mod h1:OgkHotnGiDImocRcuBABYBEXf8A9a87e/uXjp9XT3ks=
golang.org/x/term v0.27.0 h1:WP60Sv1nlK1T6SupCHbXzSaN0b9wUmsPoRS9b61A23Q=
golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
golang.org/x/tools v0.38.0 h1:Hx2Xv8hISq8Lm16jvBZ2VQf+RLmbd7wVUsALibYI/IQ= golang.org/x/tools v0.38.0 h1:Hx2Xv8hISq8Lm16jvBZ2VQf+RLmbd7wVUsALibYI/IQ=
golang.org/x/tools v0.38.0/go.mod h1:yEsQ/d/YK8cjh0L6rZlY8tgtlKiBNTL14pGDJPJpYQs= golang.org/x/tools v0.38.0/go.mod h1:yEsQ/d/YK8cjh0L6rZlY8tgtlKiBNTL14pGDJPJpYQs=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
@@ -0,0 +1,130 @@
package agentapi
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/http"
"net/url"
)
// R-82 Slice B — the per-tier backup surface (agent >= v0.97.0).
//
// Every method here is ADDITIVE. The untargeted BackupDue/StartBackup/BackupStatus keep their exact
// pre-R-82 meaning and are still the single-tier path used against an older agent.
// ErrTiersUnsupported reports that this agent does not serve GET /backup/tiers — it predates R-82.
// It is the DESIGNED capability probe (the route 404s), not a fault. The caller MUST degrade to the
// untargeted single-tier path and still take a backup; concluding "nothing to do" from it would
// silently stop backups during a fleet rollout.
var ErrTiersUnsupported = errors.New("agentapi: agent does not serve /backup/tiers (pre-R-82)")
// BackupTierInfo is one advertised tier.
type BackupTierInfo struct {
Target string `json:"target"`
CadenceSeconds int64 `json:"cadence_seconds"`
Primary bool `json:"primary"`
}
// TiersResponse mirrors the agent's GET /backup/tiers payload.
type TiersResponse struct {
VMID int `json:"vmid"`
Tiers []BackupTierInfo `json:"tiers"`
}
// BackupTiers lists the agent's backup tiers, primary first.
// Returns ErrTiersUnsupported (wrapped) on a pre-R-82 agent — key on it with errors.Is.
func (c *Client) BackupTiers(ctx context.Context) (TiersResponse, error) {
var out TiersResponse
body, err := c.get(ctx, "/backup/tiers")
if err != nil {
var se *StatusError
if errors.As(err, &se) && se.Code == http.StatusNotFound {
return out, ErrTiersUnsupported
}
return out, err
}
if err := json.Unmarshal(body, &out); err != nil {
return out, fmt.Errorf("agentapi: decode /backup/tiers: %w", err)
}
return out, nil
}
// targetQuery renders the ?target= suffix. An EMPTY target yields an empty string, so the caller
// hits the untargeted route byte-for-byte — that is what keeps the pre-R-82 contract intact when
// this client talks to an older agent.
func targetQuery(target string) string {
if target == "" {
return ""
}
return "?target=" + url.QueryEscape(target)
}
// BackupDueFor reports whether THIS TIER is due. A fresh backup on another tier must not satisfy it
// — that filtering happens agent-side (latestSuccessfulBackupForTarget); this just asks per tier.
func (c *Client) BackupDueFor(ctx context.Context, target string) (DueResponse, error) {
var out DueResponse
body, err := c.get(ctx, "/backup/due"+targetQuery(target))
if err != nil {
return out, err
}
if err := json.Unmarshal(body, &out); err != nil {
return out, fmt.Errorf("agentapi: decode /backup/due (target %q): %w", target, err)
}
return out, nil
}
// StartBackupFor enqueues a backup of this guest ON THE GIVEN TIER.
func (c *Client) StartBackupFor(ctx context.Context, target string) (BackupResponse, error) {
var out BackupResponse
body, err := c.post(ctx, "/backup"+targetQuery(target), struct{}{})
if err != nil {
return out, err
}
if err := json.Unmarshal(body, &out); err != nil {
return out, fmt.Errorf("agentapi: decode POST /backup (target %q): %w", target, err)
}
return out, nil
}
// BackupStatusFor reports THIS TIER's current/last job phase. Jobs are keyed per tier agent-side,
// so polling the wrong target would report a different tier's progress.
func (c *Client) BackupStatusFor(ctx context.Context, target string) (StatusResponse, error) {
var out StatusResponse
body, err := c.get(ctx, "/backup/status"+targetQuery(target))
if err != nil {
return out, err
}
if err := json.Unmarshal(body, &out); err != nil {
return out, fmt.Errorf("agentapi: decode /backup/status (target %q): %w", target, err)
}
return out, nil
}
// SetBackupTargetResponse mirrors POST /backup/target (agent >= v0.113.0).
type SetBackupTargetResponse struct {
Target string `json:"target"`
Where string `json:"where"`
// RestartRequired is always true on success: the agent builds its tiers once at daemon start, so
// the move needs a restart. The agent deliberately does NOT restart itself — restarting with a
// backup in flight cancels the wait and records a spurious tier failure for a backup that actually
// succeeded. The RESTART IS THE OPERATOR'S, behind an immediate in-flight check.
RestartRequired bool `json:"restart_required"`
}
// SetBackupTarget moves the primary whole-guest backup tier onto the drive at raw host mount `where`.
// Creates the storage and grants the agent access as one ordered operation.
func (c *Client) SetBackupTarget(ctx context.Context, where string) (SetBackupTargetResponse, error) {
var out SetBackupTargetResponse
// vmid is deliberately omitted: the agent derives the guest from the token and scopedFromBody
// treats an absent vmid as "use the token's" — the same shape as AssignDisk/GuestAttach.
body, err := c.post(ctx, "/backup/target", map[string]string{"where": where})
if err != nil {
return out, err
}
if err := json.Unmarshal(body, &out); err != nil {
return out, fmt.Errorf("agentapi: decode /backup/target: %w", err)
}
return out, nil
}
+62 -4
View File
@@ -174,6 +174,14 @@ type DueResponse struct {
Due bool `json:"due"` Due bool `json:"due"`
Reason string `json:"reason"` Reason string `json:"reason"`
AgeSecs *int64 `json:"age_seconds"` AgeSecs *int64 `json:"age_seconds"`
// AgeState (R-88 Part 2, agent >= v0.105.0) says WHY AgeSecs is nil: "absent" (a positive
// determination that no backup has ever landed) or "unknown" (the agent could not tell —
// unreadable storage, unparseable timestamp). "known" accompanies a real age.
//
// EMPTY MEANS LEGACY — an agent older than v0.105.0 simply omits the field. It does NOT mean
// "unknown", and the distinction is load-bearing: see quiesce.ageStateFromWire. Never
// discriminate on Reason instead; those strings are operator copy and will drift.
AgeState string `json:"age_state"`
} }
// BackupResponse mirrors the agent's POST /backup payload. // BackupResponse mirrors the agent's POST /backup payload.
@@ -315,6 +323,12 @@ type DiskInfo struct {
// opposed to merely present on the host (F9) — the signal whose absence let the HDD look available // opposed to merely present on the host (F9) — the signal whose absence let the HDD look available
// when it wasn't attached. LEGACY (per-drive mp model); the intermediary model uses BoundUnderParent. // when it wasn't attached. LEGACY (per-drive mp model); the intermediary model uses BoundUnderParent.
GuestAttached bool `json:"guest_attached"` GuestAttached bool `json:"guest_attached"`
// BackupTarget (E-2, agent >= v0.112.0) reports that this drive backs the PRIMARY whole-guest
// backup tier. The agent is the only component that can answer: our own
// settings.StoragePath.BackupTarget is customer INTENT, and on a box migrated by hand (E-1) that
// intent was never recorded while the drive really IS the target. Absent on an older agent →
// false, which degrades to the pre-E-2 behaviour (a generic disconnect alarm, never a wrong one).
BackupTarget bool `json:"backup_target,omitempty"`
// GuestPath is the drive's STABLE in-guest path in the intermediary-mount model // GuestPath is the drive's STABLE in-guest path in the intermediary-mount model
// (/mnt/felhom-drives/<name>) — what the controller registers + repoints HDD_PATH to. Distinct from // (/mnt/felhom-drives/<name>) — what the controller registers + repoints HDD_PATH to. Distinct from
// MountPath (the raw /mnt/<name> host PVE mount the agent ops on). "" for non-user-data drives. // MountPath (the raw /mnt/<name> host PVE mount the agent ops on). "" for non-user-data drives.
@@ -322,6 +336,10 @@ type DiskInfo struct {
// BoundUnderParent reports whether the drive's felhom-data is currently bound under the shared parent // BoundUnderParent reports whether the drive's felhom-data is currently bound under the shared parent
// (live + usable in the guest). The controller's drive-absent gate keys on this + State. // (live + usable in the guest). The controller's drive-absent gate keys on this + State.
BoundUnderParent bool `json:"bound_under_parent"` BoundUnderParent bool `json:"bound_under_parent"`
// Smart is the per-disk SMART health (agent v0.94.0+), nil when the device exposes no SMART or the
// agent predates the field — the disk-health card + 6h degradation check feature-detect on this and
// render "Nincs adat" (never alarm) when nil. See DiskVerdictFor.
Smart *SmartSummary `json:"smart,omitempty"`
} }
// FSUUID returns the raw filesystem UUID from a "uuid:<…>" DurableID, or "" if this disk's identity // FSUUID returns the raw filesystem UUID from a "uuid:<…>" DurableID, or "" if this disk's identity
@@ -406,6 +424,11 @@ type DiskCandidate struct {
Mountable bool `json:"mountable"` Mountable bool `json:"mountable"`
MountSource string `json:"mount_source,omitempty"` MountSource string `json:"mount_source,omitempty"`
DurableID string `json:"durable_id,omitempty"` DurableID string `json:"durable_id,omitempty"`
// AlreadyMounted marks a candidate the CONTROLLER contributed from its own mount table (R-280),
// not one the agent scanned. The agent NEVER sets it. Its action is REGISTER the existing
// mountpoint — sending it down the device-attach path would try to mount an in-guest path as if
// it were a raw device. See web/attach_sources.go for why the agent's scan cannot supply these.
AlreadyMounted bool `json:"already_mounted,omitempty"`
} }
// CandidatesResult mirrors GET /disks/candidates: disks free to enroll, split into initialize (all // CandidatesResult mirrors GET /disks/candidates: disks free to enroll, split into initialize (all
@@ -1003,14 +1026,32 @@ type ThinPoolFill struct {
MetadataUsedFraction *float64 `json:"metadata_used_fraction"` MetadataUsedFraction *float64 `json:"metadata_used_fraction"`
} }
// SmartSummary mirrors the agent's per-disk SMART health (only the fields the UI renders). Pointers // SmartSummary mirrors the agent's per-disk SMART health. Pointers are null when the device type
// are null when the device type does not expose that attribute. // does not expose that attribute (a null is "unknown / not-applicable", distinct from a real zero).
// The SATA set (reallocated/pending/offline-uncorrectable) and the NVMe set
// (critical_warning/media_errors/percentage_used) are both carried; a device populates only its own.
type SmartSummary struct { type SmartSummary struct {
Health string `json:"health"` // PASSED | FAILING | UNKNOWN Health string `json:"health"` // PASSED | FAILING | UNKNOWN
ModelName *string `json:"model_name,omitempty"` // smartctl device model (agent v0.95.0+); nil on older agents
TemperatureC *int `json:"temperature_c"` TemperatureC *int `json:"temperature_c"`
PowerOnHours *int `json:"power_on_hours"`
// SATA attributes.
ReallocatedSectors *int `json:"reallocated_sectors"`
PendingSectors *int `json:"pending_sectors"`
OfflineUncorrectable *int `json:"offline_uncorrectable"`
// NVMe attributes.
CriticalWarning *int `json:"critical_warning"`
MediaErrors *int `json:"media_errors"`
PercentageUsed *int `json:"percentage_used"` // NVMe wear (%); null for SATA/USB PercentageUsed *int `json:"percentage_used"` // NVMe wear (%); null for SATA/USB
} }
// SMART health vocabulary (mirrors the agent's).
const (
SmartPassed = "PASSED"
SmartFailing = "FAILING"
SmartUnknown = "UNKNOWN"
)
// StorageTarget mirrors the agent's GET /host/metrics storage_targets entry (the per-storage // StorageTarget mirrors the agent's GET /host/metrics storage_targets entry (the per-storage
// capacity + health the monitoring view renders). It is a SUBSET of the agent's wire shape — only // capacity + health the monitoring view renders). It is a SUBSET of the agent's wire shape — only
// the fields the UI reads; unknown JSON keys are ignored. // the fields the UI reads; unknown JSON keys are ignored.
@@ -1059,13 +1100,28 @@ func (c *Client) HostMetrics(ctx context.Context) (HostMetricsResponse, error) {
// StatusError is a non-2xx agent HTTP status surfaced as a TYPED error (same text the old // StatusError is a non-2xx agent HTTP status surfaced as a TYPED error (same text the old
// fmt.Errorf produced). errors.As-able — the capability probe (features.go) keys on Code 404 to // fmt.Errorf produced). errors.As-able — the capability probe (features.go) keys on Code 404 to
// distinguish "this agent predates the route" from every other failure. Never match the string. // distinguish "this agent predates the route" from every other failure. Never match the string.
// StatusError is a non-2xx response from the agent, carrying the STATUS CODE so callers can react
// to specific ones rather than string-matching an error message.
//
// F-A1: this exists on the POST path because HTTP 409 from `POST /backup` is not a failure — it is
// the agent's R-85 single-flight gate correctly refusing while a restore-test holds it. Treating
// that refusal as a tier failure armed the breaker and emailed the operator about a backup that was
// never actually broken. The controller now needs to tell 409 apart from a real error, and a typed
// code is the only honest way to do that.
type StatusError struct { type StatusError struct {
// Method is the HTTP method. Empty means GET, so the message stays byte-identical for the
// pre-existing GET call sites.
Method string
Path string Path string
Code int Code int
} }
func (e *StatusError) Error() string { func (e *StatusError) Error() string {
return fmt.Sprintf("agentapi: GET %s: HTTP %d", e.Path, e.Code) m := e.Method
if m == "" {
m = http.MethodGet
}
return fmt.Sprintf("agentapi: %s %s: HTTP %d", m, e.Path, e.Code)
} }
// get issues an authenticated GET and unwraps the {ok,data,error} envelope. // get issues an authenticated GET and unwraps the {ok,data,error} envelope.
@@ -1122,7 +1178,9 @@ func (c *Client) post(ctx context.Context, path string, body any) (json.RawMessa
logx.Debugf(c.logger, "[agentapi] POST %s -> %d (%dms)", path, resp.StatusCode, time.Since(start).Milliseconds()) logx.Debugf(c.logger, "[agentapi] POST %s -> %d (%dms)", path, resp.StatusCode, time.Since(start).Milliseconds())
raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusAccepted { if resp.StatusCode != http.StatusOK && resp.StatusCode != http.StatusAccepted {
return nil, fmt.Errorf("agentapi: POST %s: HTTP %d", path, resp.StatusCode) // Typed, not fmt.Errorf: callers must be able to distinguish 409 (the agent's single-flight
// gate refusing — contention, not failure) from a genuine 5xx. See StatusError.
return nil, &StatusError{Method: http.MethodPost, Path: path, Code: resp.StatusCode}
} }
var env apiResponse var env apiResponse
if err := json.Unmarshal(raw, &env); err != nil { if err := json.Unmarshal(raw, &env); err != nil {
+201
View File
@@ -0,0 +1,201 @@
package agentapi
// DiskVerdict is the customer-facing disk-health verdict derived from a SmartSummary (v0.169.0).
// It is the SHARED source of truth for both the "Lemezek állapota" dashboard card and the periodic
// degradation check — one pure function so the chip and the alert can never disagree.
type DiskVerdict int
const (
// DiskVerdictUnknown — no SMART data (nil / UNKNOWN / old agent). Renders "Nincs adat"; NEVER
// alarms and NEVER participates in degradation transitions (excluded both directions).
DiskVerdictUnknown DiskVerdict = iota
DiskVerdictOK // "Rendben" — clean
DiskVerdictWarn // "Figyelmeztetés" — a wear/relocation counter is non-zero, below the Hiba bar
DiskVerdictFail // "Hiba" — FAILING, or failing-but-not-self-reported (v0.215.0)
)
// Thresholds. A number without a reason becomes permanent by default, so each carries its provenance.
// The evidence is committed at felhom.eu/documentation/audits/DIAG-smart-passed-trap-2026-08-14.md
// and its two fixtures (ST3000VX010 S/N Z6A07P2G, /dev/sdg on DooPlex, 11-13 Aug 2026).
const (
// percentageUsedWarn / percentageUsedFail — NVMe wear (%). 100 means the vendor's rated endurance
// is spent; that is a declaration, not a trend, so it is Hiba.
percentageUsedWarn = 90
percentageUsedFail = 100
// uncorrectableFailCount — unreadable sectors too numerous to be a blip.
//
// PROVENANCE: on the one real failing drive observed, the benign excursion peaked at 16 and
// cleared COMPLETELY within an hour (11 Aug 12:28 -> 13:28); the terminal run passed 64 at
// 13 Aug 11:28 and never came back below it. 64 sits above the one observed transient and below
// the observed terminal run. This is a judgement from ONE drive: it is a static BACKSTOP behind
// the sustain rule, not the primary signal, and Phase 3 is expected to replace it with
// growth-rate detection once the box keeps history.
uncorrectableFailCount = 64
// temperatureWarnC / TemperatureFailC — adopted UNCHANGED from the operator's existing Prometheus
// bands on DooPlex, so the two systems cannot disagree about the same drive.
temperatureWarnC = 55
// TemperatureFailC is exported because the alert-copy layer must pick the "overheated" message
// shape from the SAME number the verdict fired on. A second literal elsewhere would be free to
// drift, and the drift would show up as a customer told the wrong reason.
TemperatureFailC = 60
)
// DiskPrior is what the previous check observed for THIS SAME disk. It is the only history the
// verdict consults, and it is passed in rather than read so the function stays pure — the caller
// (internal/web) owns loading it from the persisted per-disk state.
//
// Plain value type: no methods, no I/O. A zero DiskPrior means "nothing known", which is the correct
// fail-safe — a first-ever observation can only reach Figyelmeztetés from counters, never Hiba.
type DiskPrior struct {
// SawUncorrectable reports whether unreadable sectors (pending OR offline-uncorrectable) were
// present at the previous check. It is what turns a one-off excursion into a sustained fault.
SawUncorrectable bool
}
// DiskVerdictFor maps a SmartSummary plus the previous observation to a verdict. Rules are evaluated
// TOP-DOWN and the FIRST match wins (v0.215.0):
//
// 1. nil / "" / UNKNOWN -> Nincs adat
// 2. Health == FAILING -> Hiba (drive self-reports)
// 3. temperature_c >= 60 -> Hiba
// 4. critical_warning > 0 (NVMe's own flag: a declaration) -> Hiba
// 5. percentage_used >= 100 -> Hiba
// 6. unreadable > 0 AND prior.SawUncorrectable -> Hiba (SUSTAINED)
// 7. unreadable > 0 AND reallocated > 0 -> Hiba (accumulating + remapping)
// 8. unreadable >= 64 -> Hiba (too large to be a blip)
// 9. unreadable > 0 -> Figyelmeztetés (first sighting)
// 10. reallocated > 0 -> Figyelmeztetés
// 11. media_errors > 0 -> Figyelmeztetés
// 12. percentage_used >= 90 -> Figyelmeztetés
// 13. temperature_c >= 55 -> Figyelmeztetés
// 14. otherwise -> Rendben
//
// WHY rows 2-8 exist at all: smart_status.passed CANNOT fail on unreadable sectors. Attributes 187,
// 197 and 198 all carry thresh 0, and a normalized SMART value floors at 1, so it can never drop to
// or below the threshold. The real drive stayed PASSED at 352 pending sectors with 1001 reported
// uncorrectable reads. A verdict built on the drive's own self-assessment is blind to this whole
// class of failure, which is why rows 3-8 read the raw counters instead.
//
// WHY row 6 sits ABOVE row 8: sustain is the PRIMARY rule and the count is the backstop. On the real
// drive sustain fires a full day earlier (12 Aug) than the count threshold (13 Aug). Row 8 exists for
// a box that was powered off or restarted across the sustain window and so has no prior.
//
// Pure: no clock, no I/O, no logging. Everything it needs arrives as an argument.
func DiskVerdictFor(s *SmartSummary, prior DiskPrior) DiskVerdict {
// 1 — no data. Never alarms.
if s == nil || s.Health == "" || s.Health == SmartUnknown {
return DiskVerdictUnknown
}
// 2 — the drive admits failure.
if s.Health == SmartFailing {
return DiskVerdictFail
}
// Health == PASSED (or any non-empty non-FAILING value we treat as passing): inspect the counters,
// because the overall verdict is structurally unable to report this class of fault.
switch {
case atLeast(s.TemperatureC, TemperatureFailC): // 3
return DiskVerdictFail
case positive(s.CriticalWarning): // 4
return DiskVerdictFail
case atLeast(s.PercentageUsed, percentageUsedFail): // 5
return DiskVerdictFail
}
unreadable := UncorrectableSectors(s)
switch {
case unreadable > 0 && prior.SawUncorrectable: // 6 — sustained across two consecutive checks
return DiskVerdictFail
case unreadable > 0 && positive(s.ReallocatedSectors): // 7 — accumulating and remapping together
return DiskVerdictFail
case unreadable >= uncorrectableFailCount: // 8 — too large to be a blip
return DiskVerdictFail
case unreadable > 0: // 9 — first sighting, below the bar
return DiskVerdictWarn
case positive(s.ReallocatedSectors): // 10
return DiskVerdictWarn
case positive(s.MediaErrors): // 11
return DiskVerdictWarn
case atLeast(s.PercentageUsed, percentageUsedWarn): // 12
return DiskVerdictWarn
case atLeast(s.TemperatureC, temperatureWarnC): // 13
return DiskVerdictWarn
}
return DiskVerdictOK // 14
}
// UncorrectableSectors is the disk's unreadable-sector count: max(pending, offline_uncorrectable).
// The two attributes track the same physical defect and on the real drive moved in lockstep, so the
// larger is the honest figure. 0 when neither is reported (an old agent or a device without them).
// Exported because the alert copy quotes this number and the persisted state remembers it.
func UncorrectableSectors(s *SmartSummary) int {
if s == nil {
return 0
}
n := 0
if s.PendingSectors != nil && *s.PendingSectors > n {
n = *s.PendingSectors
}
if s.OfflineUncorrectable != nil && *s.OfflineUncorrectable > n {
n = *s.OfflineUncorrectable
}
return n
}
// Label is the exact Hungarian customer copy for the verdict (shared by the card chip and the email).
//
// There are FOUR labels and there will not be a fifth: a predicted failure is "Hiba", the same word a
// self-reported failure gets. A fourth word sharing a root with "Figyelmeztetés" would make the MORE
// severe state read as the milder one (settled operator decision, v0.215.0).
func (v DiskVerdict) Label() string {
switch v {
case DiskVerdictOK:
return "Rendben"
case DiskVerdictWarn:
return "Figyelmeztetés"
case DiskVerdictFail:
return "Hiba"
default:
return "Nincs adat"
}
}
// DegradedAttributes returns the human-readable Hungarian names of the attribute(s) behind a
// degraded verdict, for the alert body.
//
// v0.215.0: this now also names the attributes behind a Hiba REACHED FROM COUNTERS (truth-table rows
// 3 and 6-8), not only a Figyelmeztetés — the alert message needs to say what is wrong, and those
// rows do have a triggering counter. It returns nil ONLY for row 2 (the drive self-reports FAILING,
// a whole-disk verdict with no single triggering counter) and, naturally, for Nincs adat / Rendben.
func DegradedAttributes(s *SmartSummary) []string {
if s == nil || s.Health == "" || s.Health == SmartUnknown || s.Health == SmartFailing {
return nil
}
var out []string
if positive(s.ReallocatedSectors) {
out = append(out, "áthelyezett szektorok")
}
if positive(s.PendingSectors) {
out = append(out, "függőben lévő szektorok")
}
if positive(s.OfflineUncorrectable) {
out = append(out, "javíthatatlan szektorok")
}
if positive(s.CriticalWarning) {
out = append(out, "kritikus figyelmeztetés")
}
if positive(s.MediaErrors) {
out = append(out, "adathordozó-hibák")
}
if atLeast(s.PercentageUsed, percentageUsedWarn) {
out = append(out, "elhasználódás")
}
// Newly able to trigger a verdict on its own (rows 3 and 13), so it must be nameable.
if atLeast(s.TemperatureC, temperatureWarnC) {
out = append(out, "hőmérséklet")
}
return out
}
func positive(p *int) bool { return p != nil && *p > 0 }
func atLeast(p *int, n int) bool { return p != nil && *p >= n }
@@ -0,0 +1,202 @@
package agentapi
import "testing"
// The v0.215.0 severity ladder, verdict half. The event half (emission, damping, cooldown,
// persistence) lives in internal/web — this file pins ONLY what the pure function decides.
//
// Every value used here is taken from the committed evidence:
// felhom.eu/documentation/audits/fixtures/smart-ST3000VX010-failing-2026-08-14.json
// (ST3000VX010-2E3166, S/N Z6A07P2G, /dev/sdg on DooPlex).
// realDrive is the failing drive AS CAPTURED on 2026-08-14: PASSED, 352 pending, 352 offline
// uncorrectable, 0 reallocated, 40 °C. The whole point of the fixture is that Health is PASSED.
func realDrive() *SmartSummary {
return &SmartSummary{
Health: SmartPassed,
PendingSectors: ip(352),
OfflineUncorrectable: ip(352),
ReallocatedSectors: ip(0),
TemperatureC: ip(40),
}
}
// Group A (verdict half) — Scenario A. The real drive on its SECOND observation reaches Hiba, and
// the chip label is exactly "Hiba".
//
// Red-proof: delete truth-table row 6 (the `prior.SawUncorrectable` case) from DiskVerdictFor →
// the drive still reaches Fail via row 8 (352 >= 64), so this test alone does NOT prove row 6.
// TestLadder_SustainIsWhatFires below is the one that isolates it.
func TestLadder_RealDrive_ReachesHiba(t *testing.T) {
got := DiskVerdictFor(realDrive(), DiskPrior{SawUncorrectable: true})
if got != DiskVerdictFail {
t.Fatalf("real failing drive verdict = %d (%s), want Fail/Hiba", got, got.Label())
}
if got.Label() != "Hiba" {
t.Errorf("label = %q, want %q", got.Label(), "Hiba")
}
// The trap this whole change exists for: the drive's own verdict says everything is fine.
if realDrive().Health != SmartPassed {
t.Fatal("fixture drift: the real drive's Health must be PASSED — that IS the defect")
}
}
// Groups B + C (verdict half) — Scenarios B and C. The SAME SmartSummary yields Figyelmeztetés on a
// first sighting and Hiba once it is sustained. This is the pair that isolates row 6: the counters
// are identical and only `prior` differs, so nothing else in the table can be producing the change.
//
// The values are the 11 August excursion (8 sectors), which cleared completely within an hour — a
// count deliberately far below the 64 backstop so row 8 cannot mask row 6.
//
// Red-proof: remove the `prior.SawUncorrectable` clause from row 6 → the sustained case stays Warn.
func TestLadder_SustainIsWhatFires(t *testing.T) {
excursion := func() *SmartSummary {
return &SmartSummary{Health: SmartPassed, PendingSectors: ip(8), OfflineUncorrectable: ip(8), ReallocatedSectors: ip(0)}
}
if got := DiskVerdictFor(excursion(), DiskPrior{}); got != DiskVerdictWarn {
t.Errorf("first sighting of 8 sectors = %d (%s), want Warn/Figyelmeztetés — a single "+
"excursion that clears by itself is normal and must NOT reach Hiba", got, got.Label())
}
if got := DiskVerdictFor(excursion(), DiskPrior{SawUncorrectable: true}); got != DiskVerdictFail {
t.Errorf("SAME 8 sectors, now sustained = %d (%s), want Fail/Hiba", got, got.Label())
}
if got := DiskVerdictFor(excursion(), DiskPrior{}).Label(); got != "Figyelmeztetés" {
t.Errorf("first-sighting label = %q, want Figyelmeztetés", got)
}
}
// Row 8, the backstop — for a box that was powered off or restarted across the sustain window and so
// has NO prior. 63 stays Warn, 64 reaches Hiba. The boundary is inclusive, which is what
// `uncorrectableFailCount` claims and what the real drive did at 13 Aug 11:28 (exactly 64).
//
// Red-proof: change `>=` to `>` in row 8 → the "exactly 64" case reads Warn.
func TestLadder_CountBackstopBoundary(t *testing.T) {
cases := []struct {
pending int
want DiskVerdict
}{
{63, DiskVerdictWarn},
{64, DiskVerdictFail},
{352, DiskVerdictFail},
}
for _, c := range cases {
s := &SmartSummary{Health: SmartPassed, PendingSectors: ip(c.pending)}
if got := DiskVerdictFor(s, DiskPrior{}); got != c.want {
t.Errorf("%d pending sectors, no prior = %d (%s), want %d", c.pending, got, got.Label(), c.want)
}
}
// Row 7 — unreadable AND remapping together is Hiba even at a low count with no prior.
s := &SmartSummary{Health: SmartPassed, PendingSectors: ip(8), ReallocatedSectors: ip(1)}
if got := DiskVerdictFor(s, DiskPrior{}); got != DiskVerdictFail {
t.Errorf("row 7 (unreadable + reallocated) = %d, want Fail", got)
}
}
// Group I — Scenario I, heat. 61 → Hiba, 56 → Figyelmeztetés, 54 → Rendben, with all counters clean.
//
// Red-proof: remove rows 3 and 13 → all three read Rendben.
func TestLadder_Temperature(t *testing.T) {
cases := []struct {
temp int
want DiskVerdict
}{
{54, DiskVerdictOK},
{55, DiskVerdictWarn}, // inclusive boundary
{56, DiskVerdictWarn},
{59, DiskVerdictWarn},
{60, DiskVerdictFail}, // inclusive boundary
{61, DiskVerdictFail},
}
for _, c := range cases {
s := &SmartSummary{Health: SmartPassed, TemperatureC: ip(c.temp), PendingSectors: ip(0), ReallocatedSectors: ip(0)}
if got := DiskVerdictFor(s, DiskPrior{}); got != c.want {
t.Errorf("%d °C = %d (%s), want %d", c.temp, got, got.Label(), c.want)
}
}
}
// Group J (verdict half) — Scenario J. No data never alarms, and a prior must not manufacture one:
// a nil/UNKNOWN SmartSummary reads Nincs adat EVEN WITH SawUncorrectable set. Row 1 is first in the
// table for exactly this reason.
//
// Red-proof: move row 1 below row 6 → the UNKNOWN-with-prior case reads Hiba, i.e. a disk whose
// SMART briefly became unreadable would be reported as failing.
func TestLadder_UnknownNeverAlarms(t *testing.T) {
for _, s := range []*SmartSummary{nil, {Health: ""}, {Health: SmartUnknown}} {
if got := DiskVerdictFor(s, DiskPrior{SawUncorrectable: true}); got != DiskVerdictUnknown {
t.Errorf("no-data disk with a prior = %d (%s), want Unknown/Nincs adat", got, got.Label())
}
}
if got := DiskVerdictFor(&SmartSummary{Health: SmartUnknown}, DiskPrior{}).Label(); got != "Nincs adat" {
t.Errorf("label = %q, want Nincs adat", got)
}
}
// The zero DiskPrior must be the SAFE default: a caller that forgets to load history can only
// under-report (Figyelmeztetés), never over-report (Hiba) on a first sighting. This pins the
// fail-safe direction the persisted-state loader relies on when its file is missing or corrupt.
func TestLadder_ZeroPriorIsFailSafe(t *testing.T) {
s := &SmartSummary{Health: SmartPassed, PendingSectors: ip(8)}
if got := DiskVerdictFor(s, DiskPrior{}); got != DiskVerdictWarn {
t.Fatalf("zero prior must degrade to Warn, not Fail; got %d (%s)", got, got.Label())
}
}
// UncorrectableSectors is max(pending, offline) — the number the alert copy quotes and the persisted
// state remembers. A wrong answer here puts a wrong count in a customer's email.
func TestUncorrectableSectors(t *testing.T) {
cases := []struct {
name string
in *SmartSummary
want int
}{
{"nil summary", nil, 0},
{"neither reported (old agent)", &SmartSummary{Health: SmartPassed}, 0},
{"both zero", &SmartSummary{PendingSectors: ip(0), OfflineUncorrectable: ip(0)}, 0},
{"pending only", &SmartSummary{PendingSectors: ip(8)}, 8},
{"offline only", &SmartSummary{OfflineUncorrectable: ip(24)}, 24},
{"pending larger", &SmartSummary{PendingSectors: ip(40), OfflineUncorrectable: ip(24)}, 40},
{"offline larger", &SmartSummary{PendingSectors: ip(24), OfflineUncorrectable: ip(40)}, 40},
{"the real drive", realDrive(), 352},
}
for _, c := range cases {
if got := UncorrectableSectors(c.in); got != c.want {
t.Errorf("%s: UncorrectableSectors = %d, want %d", c.name, got, c.want)
}
}
}
// DegradedAttributes must NAME the counters behind a Hiba reached from counters (v0.215.0) — the
// alert body is built from this and an empty list produces a message that says nothing is wrong.
// It still returns nil for row 2 (drive-reported FAILING), which has no single triggering counter.
//
// Red-proof: restore the pre-v0.215.0 body (nil for anything at Fail) → the real-drive case returns
// an empty list.
func TestDegradedAttributes_NamesFailCounters(t *testing.T) {
got := DegradedAttributes(realDrive())
if len(got) == 0 {
t.Fatal("a Hiba reached from counters must name its attributes, got none")
}
found := map[string]bool{}
for _, a := range got {
found[a] = true
}
for _, want := range []string{"függőben lévő szektorok", "javíthatatlan szektorok"} {
if !found[want] {
t.Errorf("missing attribute %q in %v", want, got)
}
}
// Row 2 — the drive self-reports FAILING: no single triggering counter, so nil.
if a := DegradedAttributes(&SmartSummary{Health: SmartFailing, PendingSectors: ip(5)}); a != nil {
t.Errorf("FAILING (row 2) must return nil attributes, got %v", a)
}
// Nincs adat must never produce attribute names either.
if a := DegradedAttributes(&SmartSummary{Health: SmartUnknown}); a != nil {
t.Errorf("UNKNOWN must return nil attributes, got %v", a)
}
// Temperature is newly able to trigger on its own, so it must be nameable.
hot := DegradedAttributes(&SmartSummary{Health: SmartPassed, TemperatureC: ip(61)})
if len(hot) != 1 || hot[0] != "hőmérséklet" {
t.Errorf("hot disk attributes = %v, want [hőmérséklet]", hot)
}
}
@@ -0,0 +1,71 @@
package agentapi
import "testing"
func ip(v int) *int { return &v }
// Verdict table (Part 2, extended v0.215.0). Red-proof: change the PercentageUsed boundary from
// `>= 90` to `> 90` in DiskVerdictFor → the "NVMe percentage_used exactly 90 → Figyelmeztetés" case
// fails.
//
// v0.215.0 moved ONE pre-existing case deliberately: critical_warning>0 was Figyelmeztetés and is
// now Hiba (truth-table row 4). It is NVMe's own critical flag — a declaration by the device, not a
// counter that might drift back — so it belongs with the self-reported failures, not below them.
func TestDiskVerdictFor(t *testing.T) {
noPrior := DiskPrior{}
cases := []struct {
name string
in *SmartSummary
prior DiskPrior
want DiskVerdict
}{
{"nil → unknown", nil, noPrior, DiskVerdictUnknown},
{"empty health → unknown", &SmartSummary{Health: ""}, noPrior, DiskVerdictUnknown},
{"UNKNOWN → unknown", &SmartSummary{Health: SmartUnknown}, noPrior, DiskVerdictUnknown},
{"FAILING → fail", &SmartSummary{Health: SmartFailing}, noPrior, DiskVerdictFail},
{"FAILING beats counters", &SmartSummary{Health: SmartFailing, ReallocatedSectors: ip(0)}, noPrior, DiskVerdictFail},
{"PASSED clean → ok", &SmartSummary{Health: SmartPassed, ReallocatedSectors: ip(0), PendingSectors: ip(0), TemperatureC: ip(30)}, noPrior, DiskVerdictOK},
{"PASSED nil counters → ok", &SmartSummary{Health: SmartPassed}, noPrior, DiskVerdictOK},
{"reallocated>0 alone → warn", &SmartSummary{Health: SmartPassed, ReallocatedSectors: ip(1)}, noPrior, DiskVerdictWarn},
{"pending>0 first sighting → warn", &SmartSummary{Health: SmartPassed, PendingSectors: ip(5)}, noPrior, DiskVerdictWarn},
{"offline_unc>0 first sighting → warn", &SmartSummary{Health: SmartPassed, OfflineUncorrectable: ip(2)}, noPrior, DiskVerdictWarn},
{"critical_warning>0 → fail (row 4)", &SmartSummary{Health: SmartPassed, CriticalWarning: ip(1)}, noPrior, DiskVerdictFail},
{"media_errors>0 → warn", &SmartSummary{Health: SmartPassed, MediaErrors: ip(3)}, noPrior, DiskVerdictWarn},
{"percentage_used 89 → ok", &SmartSummary{Health: SmartPassed, PercentageUsed: ip(89)}, noPrior, DiskVerdictOK},
{"percentage_used exactly 90 → warn", &SmartSummary{Health: SmartPassed, PercentageUsed: ip(90)}, noPrior, DiskVerdictWarn},
{"percentage_used 95 → warn", &SmartSummary{Health: SmartPassed, PercentageUsed: ip(95)}, noPrior, DiskVerdictWarn},
{"percentage_used exactly 100 → fail (row 5)", &SmartSummary{Health: SmartPassed, PercentageUsed: ip(100)}, noPrior, DiskVerdictFail},
}
for _, c := range cases {
if got := DiskVerdictFor(c.in, c.prior); got != c.want {
t.Errorf("%s: DiskVerdictFor = %d, want %d", c.name, got, c.want)
}
}
}
func TestDiskVerdict_Label(t *testing.T) {
want := map[DiskVerdict]string{
DiskVerdictUnknown: "Nincs adat",
DiskVerdictOK: "Rendben",
DiskVerdictWarn: "Figyelmeztetés",
DiskVerdictFail: "Hiba",
}
for v, w := range want {
if got := v.Label(); got != w {
t.Errorf("verdict %d Label = %q, want %q", v, got, w)
}
}
}
// A warn lists every triggering attribute at once (Scenario "multiple attributes degrade" → ONE event).
func TestDegradedAttributes_ListsAll(t *testing.T) {
s := &SmartSummary{Health: SmartPassed, PendingSectors: ip(5), ReallocatedSectors: ip(2), PercentageUsed: ip(91)}
got := DegradedAttributes(s)
if len(got) != 3 {
t.Fatalf("want 3 attributes, got %d: %v", len(got), got)
}
// clean disk → none
if a := DegradedAttributes(&SmartSummary{Health: SmartPassed}); len(a) != 0 {
t.Errorf("clean disk should list no attributes, got %v", a)
}
}
+163
View File
@@ -3,6 +3,7 @@ package agentapi
import ( import (
"context" "context"
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"net/http" "net/http"
) )
@@ -117,3 +118,165 @@ func (c *Client) EscrowCeremonyClaim(ctx context.Context) (string, int, error) {
} }
return out.RecoveryCode, status, nil return out.RecoveryCode, status, nil
} }
// RecoverOffsiteRepoPassword asks the agent to open this host's hub-held sealed bundle with the
// customer's recovery code and return ONLY the offsite restic repository password, plus its sha256
// (R-199, agent >= v0.125.0).
//
// R CROSSES HERE, AND NOWHERE ELSE IN THIS DIRECTION. It travels in the request body over the pinned
// local-API channel (the operator's 2026-08-04 acceptance) and is not retained by this client. The
// shared POST helper logs path/status/duration and never bodies — do not add a body log, on either
// the request or the response side: the request carries R and the response carries the password.
func (c *Client) RecoverOffsiteRepoPassword(ctx context.Context, recoveryCode string) (password, sha256hex string, err error) {
env, status, perr := c.postWithStatus(ctx, "/escrow/recover-offsite-password",
map[string]string{"recovery_code": recoveryCode})
if perr != nil {
return "", "", perr
}
// R-224: this route's refusal keeps its STATUS as a value. `refusalError` flattens status into a
// sentence, and a sentence is not something a caller can branch on — which is exactly how a failed
// fetch and a wrong recovery code came to produce one customer-facing message.
if status < 200 || status > 299 || !env.OK {
return "", "", &RecoveryRefusal{Status: status, Reason: truncateErr(env.Error, 300)}
}
var out struct {
ResticRepoPassword string `json:"restic_repo_password"`
ResticPwSHA256 string `json:"restic_pw_sha256"`
}
if uerr := json.Unmarshal(env.Data, &out); uerr != nil {
return "", "", fmt.Errorf("agentapi: decode /escrow/recover-offsite-password: %w", uerr)
}
if out.ResticRepoPassword == "" || out.ResticPwSHA256 == "" {
return "", "", fmt.Errorf("agentapi: the agent returned an empty recovery result")
}
return out.ResticRepoPassword, out.ResticPwSHA256, nil
}
// ── R-224 — CLASSIFYING A FAILED UNLOCK ─────────────────────────────────────────────────────────
//
// CAMPAIGN-11 measured what happens without this. On 2026-08-05, with a CORRECT current recovery
// code: the hub firewalled off returned the customer "this code does not open your package" in
// 0.0556 s, and this agent stopped returned the same in 0.0299 s — against ~1.0 s for a genuine
// unseal. Neither attempted one. The failure path had exactly two branches, both of them statements
// about the customer's code, and `rerr` was never inspected.
//
// The rule this type exists to enforce: **the customer is blamed only after a real attempt refused
// their code.** Everything else — including anything we cannot classify — says something else.
// RecoveryRefusal is the agent's refusal of an unlock, carrying the STATUS as a value so callers
// classify on it rather than on the sentence. The message keeps `refusalError`'s shape so operator
// logs read as they did.
type RecoveryRefusal struct {
Status int
Reason string
}
func (e *RecoveryRefusal) Error() string {
reason := e.Reason
if reason == "" {
reason = "(no reason in agent response)"
}
return fmt.Sprintf("agentapi: POST /escrow/recover-offsite-password: HTTP %d: %s", e.Status, reason)
}
// RecoveryFailure is what went wrong, as far as it can be known.
type RecoveryFailure int
const (
// RecoveryUnknown — the cause could not be determined. **The safe default**, and deliberately the
// zero value: a new status, a transport shape nobody anticipated, or an agent too old to
// distinguish fetch from refusal all land here, and none of them may blame the customer.
RecoveryUnknown RecoveryFailure = iota
// RecoveryHubUnreachable — the agent answered, and it could not FETCH the sealed package: the hub
// refused, was unreachable, or recovery is not configured on this agent. **The code was not used.**
RecoveryHubUnreachable
// RecoveryAskedAndRefused — the bundle was fetched and the code did not open it. The ONLY class
// from which the customer may be told to check their typing.
RecoveryAskedAndRefused
// RecoveryNoBundle — the hub holds no sealed package for this host at all.
RecoveryNoBundle
// RecoveryBundleTooOld — the bundle opened but predates the repository-password field.
RecoveryBundleTooOld
// RecoveryAgentUnreachable — the machine's own in-house service never answered, so there is no
// agent verdict at all. **The code was not used.** Distinct from RecoveryHubUnreachable because
// it is a different fault, with different words and a different remedy.
RecoveryAgentUnreachable
// RecoveryCodeOpensRetained — the code was used, it WORKED, and it opened a RETAINED earlier
// package rather than the one currently held (R-311, agent >= v0.129.0).
//
// **The customer is not at fault here and must not be told they might be.** This class exists
// because until 2026-08-12 this situation and a mistype were indistinguishable: both fail closed
// against the current package, and nothing ever tried the retained ones. The screen said as much
// out loud — a true sentence about our own incuriosity that a customer reads as a statement about
// their code.
RecoveryCodeOpensRetained
)
// ClassifyRecoveryFailure maps an unlock error to its class, from the VALUE and never the text.
//
// ⚠ `trustRefusal` is the agent-version gate and it is not optional. An agent older than v0.126.0
// answers **400 for BOTH** a fetch failure and a wrong code, so a 400 from one cannot be read as
// "the code was refused" — it means "one of two things, and we cannot tell which". Pass false there
// and the 400 degrades to RecoveryUnknown, which is neutral. That degradation is the point: it is
// safe, it is silent, and it heals itself when the agent updates.
// ⚠ `trustRetained` is the R-311 twin of `trustRefusal` and is separate on purpose: the two gates
// name different agent versions (v0.126.0 and v0.129.0) and a box can sit between them. Passing
// `trustRefusal` for both would let a v0.126128 agent's unexpected 422 be read as a verdict it
// cannot produce.
func ClassifyRecoveryFailure(err error, trustRefusal, trustRetained bool) RecoveryFailure {
if err == nil {
return RecoveryUnknown
}
var ref *RecoveryRefusal
if !errors.As(err, &ref) {
// Not a refusal at all — the request never produced an agent verdict (dial failure, TLS,
// timeout, or the channel could not be built). The machine could not even ASK its own service,
// which is a different sentence from "the hub was unreachable" and a different thing to fix.
return RecoveryAgentUnreachable
}
switch ref.Status {
case http.StatusBadGateway, http.StatusServiceUnavailable, http.StatusGatewayTimeout:
// 502 is agent >= v0.126.0's "the sealed bundle could not be fetched". 503 is its
// "recovery is not configured on this agent (no hub client)". Neither used the code.
return RecoveryHubUnreachable
case http.StatusNotFound:
return RecoveryNoBundle
case http.StatusConflict:
return RecoveryBundleTooOld
case http.StatusUnprocessableEntity:
// R-311. Gated on the SAME trust flag as 400, and for the mirror-image reason: an agent that
// predates the retained lookup cannot emit 422 at all, so a 422 from anywhere else is a shape
// we did not design and must not be read as a statement about the customer's code.
if trustRetained {
return RecoveryCodeOpensRetained
}
return RecoveryUnknown
case http.StatusBadRequest:
if trustRefusal {
return RecoveryAskedAndRefused
}
return RecoveryUnknown
default:
return RecoveryUnknown
}
}
// String names the class for the operator log. The customer never sees these words.
func (f RecoveryFailure) String() string {
switch f {
case RecoveryHubUnreachable:
return "hub-unreachable"
case RecoveryAgentUnreachable:
return "agent-unreachable"
case RecoveryAskedAndRefused:
return "asked-and-refused"
case RecoveryNoBundle:
return "no-bundle"
case RecoveryBundleTooOld:
return "bundle-too-old"
case RecoveryCodeOpensRetained:
return "code-opens-retained"
default:
return "unknown"
}
}
+104
View File
@@ -29,6 +29,60 @@ const FeatureNetstorageVerify Feature = "netstorage_verify"
// (GET/POST /guest/memory) shipped together, so GET /guest/memory IS the capability signal. // (GET/POST /guest/memory) shipped together, so GET /guest/memory IS the capability signal.
const FeatureGuestMemoryResize Feature = "guest_memory_resize" const FeatureGuestMemoryResize Feature = "guest_memory_resize"
// FeatureBackupAgeState is R-88 Part 2 (agent v0.105.0): GET /backup/due carries `age_state`,
// distinguishing "never backed up" (absent) from "could not tell" (unknown). There is no route
// probe for it — the signal is a FIELD on an existing route, so the version floor is the gate and
// an empty field means legacy.
const FeatureBackupAgeState Feature = "backup_age_state"
// FeatureOffsiteKeyRecovery is the customer-facing off-site key recovery (agent v0.125.0, R-199
// links 78): POST /escrow/recover-offsite-password fetches this host's sealed bundle, unseals it
// with R and returns the single repository-password field.
//
// ⚠ THIS GATE FAILS CLOSED, and it is the ONLY feature in this table that does. Read §7.1 of the
// R-216 fix before "correcting" it back to the package default.
//
// The package default is fail-OPEN: SupportUnknown proceeds, because for every other coupled feature
// a wrong "unsupported" would block something harmless while a down agent already speaks through the
// normal error paths. **That default is what produced R-216.** Measured live on 2026-08-05
// (CAMPAIGN-11 Phase 1): an agent 0.120.0 answered the recovery route with 404, the unlock attempt
// went ahead anyway, and the customer was told — in Hungarian, on the one screen whose whole purpose
// is to be believed about backups — that their perfectly correct recovery code was not accepted and
// they should check their typing. A correct code, refused in 0.134 s, blamed on the customer.
//
// So here: anything other than SupportYes means the screen says THE MACHINE cannot ask yet. The
// unlock is never attempted when it cannot complete, because the failure of an attempt that could
// never have worked is attributed to the code.
const FeatureOffsiteKeyRecovery Feature = "offsite_key_recovery"
// FeatureRecoveryFailureClass is agent v0.126.0's SPLIT of a failed unlock into distinguishable
// statuses (R-224): 502 the sealed bundle could not be FETCHED · 400 it was fetched and the code was
// refused · 404 no bundle · 409 the bundle predates the repository-password field.
//
// ⚠ WHAT THIS GATE ACTUALLY GUARDS is the meaning of **400**, and nothing else. An agent older than
// v0.126.0 answers 400 for BOTH a fetch failure and a wrong code — one status, one sentence, two
// situations — so on such an agent a 400 cannot be read as "the code was refused". It means "one of
// two things and we cannot tell which", which is `RecoveryUnknown`, which is neutral.
//
// So this gate does not block anything and has no fail-closed behaviour to get wrong: the unlock is
// attempted either way (FeatureOffsiteKeyRecovery already decides THAT). It only decides whether the
// customer may be told to check their typing. Unknown → they may not. **That is the safe direction,
// and it heals itself the moment the agent updates.**
const FeatureRecoveryFailureClass Feature = "recovery_failure_class"
// FeatureRetainedRecoveryClass is agent v0.129.0's FIFTH status on a failed unlock (R-311): 422, the
// code is correct and opens a RETAINED earlier package rather than the current one.
//
// ⚠ WHAT THIS GATE GUARDS is whether the screen may say WHICH of the two causes it is. Before
// v0.129.0 nothing ever tried the retained packages, so a correct-but-earlier code and a mistype were
// genuinely indistinguishable and the screen said so. That sentence was HONEST then and becomes a
// falsehood the moment the agent can tell them apart — so the gate decides which of two true
// sentences to print, never whether to attempt the unlock.
//
// Unknown → the older, hedged sentence. That is the safe direction: it claims less, it was correct
// for two months, and it heals itself when the agent updates.
const FeatureRetainedRecoveryClass Feature = "retained_recovery_class"
// SupportState is a probe verdict. The zero value is SupportUnknown (fail-open: unknown never // SupportState is a probe verdict. The zero value is SupportUnknown (fail-open: unknown never
// refuses — the existing agent-error paths speak honestly when the agent is down). // refuses — the existing agent-error paths speak honestly when the agent is down).
type SupportState int type SupportState int
@@ -86,12 +140,36 @@ var featureProbes = map[Feature]func(ctx context.Context, p SupportProber) error
_, err := gm.GuestMemory(ctx) _, err := gm.GuestMemory(ctx)
return err return err
}, },
// The recovery route is a POST that performs work and consumes a recovery code — it cannot be
// probed. Like the memory prober's negative case this returns a sentinel that classifies to
// SupportUnknown, so the decision falls to the VERSION path above.
//
// The row must exist even though it cannot probe: SupportsWithSource looks up featureProbes
// FIRST and returns "unregistered"/SupportUnknown on a table gap, before the version path runs.
// A featureMinAgent row without a featureProbes row is therefore never consulted at all.
FeatureOffsiteKeyRecovery: func(ctx context.Context, p SupportProber) error {
return errNoRecoveryProbe
},
// Same POST route, same reason it cannot be probed — the decision falls to the VERSION path.
FeatureRecoveryFailureClass: func(ctx context.Context, p SupportProber) error {
return errNoRecoveryProbe
},
// R-311, same route and same reason. The row must exist or SupportsWithSource returns
// "unregistered"/SupportUnknown on the table gap and the version row is never consulted.
FeatureRetainedRecoveryClass: func(ctx context.Context, p SupportProber) error {
return errNoRecoveryProbe
},
} }
// errNoMemoryProbe classifies to SupportUnknown (not a *StatusError 404), so a prober that cannot be // errNoMemoryProbe classifies to SupportUnknown (not a *StatusError 404), so a prober that cannot be
// asked never reads as "unsupported". // asked never reads as "unsupported".
var errNoMemoryProbe = errors.New("agentapi: prober does not support the guest-memory probe") var errNoMemoryProbe = errors.New("agentapi: prober does not support the guest-memory probe")
// errNoRecoveryProbe classifies to SupportUnknown: the off-site key recovery route is a POST that
// consumes a recovery code and so cannot be probed, leaving the VERSION path to decide. Its caller
// fails CLOSED on Unknown — see FeatureOffsiteKeyRecovery.
var errNoRecoveryProbe = errors.New("agentapi: the offsite key recovery route cannot be probed")
// featureMinAgent maps each coupled feature to the MINIMUM agent version that carries its coupled // featureMinAgent maps each coupled feature to the MINIMUM agent version that carries its coupled
// semantics (the CHANGELOG `MinAgent:` header value). Used by Supports when the agent's version is // semantics (the CHANGELOG `MinAgent:` header value). Used by Supports when the agent's version is
// KNOWN (the v0.82.0 X-Felhom-Agent-Version channel) — a direct comparison, no probe traffic. A // KNOWN (the v0.82.0 X-Felhom-Agent-Version channel) — a direct comparison, no probe traffic. A
@@ -99,8 +177,28 @@ var errNoMemoryProbe = errors.New("agentapi: prober does not support the guest-m
var featureMinAgent = map[Feature]string{ var featureMinAgent = map[Feature]string{
FeatureNetstorageVerify: "0.81.0", FeatureNetstorageVerify: "0.81.0",
FeatureGuestMemoryResize: "0.90.0", FeatureGuestMemoryResize: "0.90.0",
// R-88 Part 2: /backup/due carries age_state, distinguishing "never backed up" from "cannot tell".
FeatureBackupAgeState: "0.105.0",
// R-199 links 78: POST /escrow/recover-offsite-password. R-216 — this row is the whole reason a
// correct recovery code can no longer be reported as wrong on an agent that cannot answer.
FeatureOffsiteKeyRecovery: "0.125.0",
// R-224 — the four-way status split of a failed unlock.
FeatureRecoveryFailureClass: "0.126.0",
// R-311 — the FIFTH status: 422, "your code is correct, it opens an EARLIER package". Before
// v0.129.0 the agent never looked at retained packages, so this situation was indistinguishable
// from a mistype and arrived as 400. An older agent therefore cannot produce a 422 at all, and the
// screen must keep saying it cannot tell the two apart — which was true, and is what this gate
// preserves for boxes that have not updated yet.
FeatureRetainedRecoveryClass: "0.129.0",
} }
// MinAgentFor returns the declared minimum agent version for a feature ("" when the feature has no
// row). Read-only accessor over featureMinAgent so a refusal can NAME the version it needs instead of
// hard-coding the number a second time at the call site.
func MinAgentFor(f Feature) string { return featureMinAgent[f] }
// AgentVersionReporter is optionally implemented by a SupportProber (*Client is one): it reports // AgentVersionReporter is optionally implemented by a SupportProber (*Client is one): it reports
// the last strictly-validated agent version seen on its traffic ("" = unknown → probe fallback). // the last strictly-validated agent version seen on its traffic ("" = unknown → probe fallback).
type AgentVersionReporter interface { type AgentVersionReporter interface {
@@ -222,3 +320,9 @@ func classifySupportErr(err error) SupportState {
} }
return SupportUnknown return SupportUnknown
} }
// AgentVersionReporter witness. Asserted at SupportsWithSource (`p.(AgentVersionReporter)`); a failed
// assertion falls back from the version gate to the live probe. That degrade is benign — both paths
// decide correctly — but *Client is the production prober and losing the version path would silently
// turn every MinAgent floor into a probe round-trip, which is a behaviour change nobody would see.
var _ AgentVersionReporter = (*Client)(nil)
@@ -27,6 +27,7 @@ func (p *snapshotsStubProvider) GetStackComposePath(name string) (string, bool)
func (p *snapshotsStubProvider) ListDeployedStacks() []backup.StackSummary { return nil } func (p *snapshotsStubProvider) ListDeployedStacks() []backup.StackSummary { return nil }
func (p *snapshotsStubProvider) GetStackHDDMounts(string) []string { return nil } func (p *snapshotsStubProvider) GetStackHDDMounts(string) []string { return nil }
func (p *snapshotsStubProvider) GetStackHDDPath(string) string { return p.hdd } func (p *snapshotsStubProvider) GetStackHDDPath(string) string { return p.hdd }
func (p *snapshotsStubProvider) GetImportRoot() string { return "" } // R-75: no import binds in this fixture
func (p *snapshotsStubProvider) GetDockerVolumes(string) []string { return nil } func (p *snapshotsStubProvider) GetDockerVolumes(string) []string { return nil }
func (p *snapshotsStubProvider) StopStack(string) error { return nil } func (p *snapshotsStubProvider) StopStack(string) error { return nil }
func (p *snapshotsStubProvider) StartStack(string) error { return nil } func (p *snapshotsStubProvider) StartStack(string) error { return nil }
@@ -0,0 +1,149 @@
package api
import (
"go/ast"
"go/parser"
"go/token"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
)
// R-166 Part 1.3 — THE CUSTOMER-INTENT POINT.
//
// `stackMgr` is a concrete *stacks.Manager, so actionStack cannot be driven with a fake without
// Docker. The two properties that actually carry the correctness are therefore pinned the only way
// they can be: the mapping is a pure function with its own table test, and the ORDER (§8.2) is
// asserted structurally over actionStack's AST. Both fail if someone reverses the write and the act,
// which is the mistake that would undo a customer's Stop at the next boot.
func TestDesiredStateForAction_MapsEveryAction(t *testing.T) {
cases := []struct {
action string
want string
ok bool
}{
{"start", stacks.DesiredStateRunning, true},
// restart and update both END in `compose up -d`, so a customer who presses either is asking
// for the app to be up afterwards.
{"restart", stacks.DesiredStateRunning, true},
{"update", stacks.DesiredStateRunning, true},
{"stop", stacks.DesiredStateStopped, true},
// Anything unrecognised records NOTHING rather than guessing — a future action must not
// silently acquire an intent it was never meant to carry.
{"", "", false},
{"delete", "", false},
{"pause", "", false},
}
for _, tc := range cases {
got, ok := desiredStateForAction(tc.action)
if got != tc.want || ok != tc.ok {
t.Fatalf("desiredStateForAction(%q) = (%q, %v), want (%q, %v)", tc.action, got, ok, tc.want, tc.ok)
}
}
}
func TestDesiredStateForAction_NeverRecordsStoppedForANonStop(t *testing.T) {
// The asymmetry that matters: writing "stopped" for anything other than a Stop would permanently
// disable auto-recovery for an app nobody stopped.
for _, a := range []string{"start", "restart", "update", "deploy", "delete", ""} {
if got, _ := desiredStateForAction(a); got == stacks.DesiredStateStopped {
t.Fatalf("action %q maps to desired_state=stopped", a)
}
}
}
// TestActionStack_RecordsIntentBeforeActing is §8.2, asserted structurally.
//
// If the SetDesiredState call moved BELOW the action switch, a stop could remove every container
// while app.yaml still recorded `running` — and the boot reconciler would then start an app the
// customer had just deliberately stopped. That is the single worst outcome available in Part 1, and
// no behavioural test in this package can reach it without a Docker daemon.
func TestActionStack_RecordsIntentBeforeActing(t *testing.T) {
body := funcBody(t, "actionStack")
setPos, switchPos := -1, -1
ast.Inspect(body, func(n ast.Node) bool {
switch node := n.(type) {
case *ast.CallExpr:
if sel, ok := node.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "SetDesiredState" && setPos < 0 {
setPos = int(node.Pos())
}
case *ast.SwitchStmt:
// The action switch is the one whose tag is the `action` identifier.
if id, ok := node.Tag.(*ast.Ident); ok && id.Name == "action" && switchPos < 0 {
switchPos = int(node.Pos())
}
}
return true
})
if setPos < 0 {
t.Fatal("actionStack no longer calls SetDesiredState — the customer's start/stop decision is " +
"recorded nowhere, which is the R-166 defect un-fixed")
}
if switchPos < 0 {
t.Fatal("actionStack no longer has a `switch action` — this test needs updating")
}
if setPos >= switchPos {
t.Fatal("actionStack records the desired state AFTER performing the action (§8.2 violated): a " +
"stop whose intent write fails or lands late leaves zero containers with `running` " +
"recorded, and the boot reconciler would restart an app the customer just stopped")
}
}
// TestActionStack_RefusesTheActionWhenIntentCannotBeRecorded pins the other half of §8.2: a failed
// write REFUSES the act. Proceeding anyway would perform a stop that nothing records — exactly the
// ambiguity this release removes.
func TestActionStack_RefusesTheActionWhenIntentCannotBeRecorded(t *testing.T) {
body := funcBody(t, "actionStack")
refuses := false
ast.Inspect(body, func(n ast.Node) bool {
ifst, ok := n.(*ast.IfStmt)
if !ok || ifst.Init == nil {
return true
}
// Look for `if derr := ...SetDesiredState(...); derr != nil { ... return }`
assign, ok := ifst.Init.(*ast.AssignStmt)
if !ok || len(assign.Rhs) != 1 {
return true
}
call, ok := assign.Rhs[0].(*ast.CallExpr)
if !ok {
return true
}
sel, ok := call.Fun.(*ast.SelectorExpr)
if !ok || sel.Sel.Name != "SetDesiredState" {
return true
}
for _, stmt := range ifst.Body.List {
if _, isReturn := stmt.(*ast.ReturnStmt); isReturn {
refuses = true
}
}
return true
})
if !refuses {
t.Fatal("actionStack does not RETURN when SetDesiredState fails — it would go on to stop or " +
"start an app whose intent could not be recorded (§8.2)")
}
}
// funcBody parses router.go and returns the named method's body.
func funcBody(t *testing.T, name string) *ast.BlockStmt {
t.Helper()
fset := token.NewFileSet()
f, err := parser.ParseFile(fset, "router.go", nil, 0)
if err != nil {
t.Fatalf("parse router.go: %v", err)
}
for _, decl := range f.Decls {
if fn, ok := decl.(*ast.FuncDecl); ok && fn.Name.Name == name && fn.Body != nil {
return fn.Body
}
}
t.Fatalf("func %s not found in router.go", name)
return nil
}
+53
View File
@@ -448,6 +448,20 @@ func (r *Router) deployStack(w http.ResponseWriter, req *http.Request, name stri
return return
} }
// R-108: an app's data namespace may NOT live on network storage — its backups would land at
// `<share>/backups/primary/<stack>/`, inside the share-ROOT bind FileBrowser serves with
// download:true (and that bind cannot be narrowed — see settings.RefuseAsAppNamespace).
//
// THIS is the boundary, not the deploy dropdown. The dropdown is a UI list; this endpoint accepts
// whatever HDD_PATH a caller supplies and `DeployStack` validates only that it EXISTS on the
// filesystem (os.Stat, internal/stacks/deploy.go). A filter on the list alone would have left the
// surface wide open — the R-108 row's "no IsNetwork() filter on the dropdown" understates it.
if refuse, why := r.sett.RefuseAsAppNamespace(body.Values["HDD_PATH"]); refuse {
r.logger.Printf("[WARN] [api] Deploy refused for %s: HDD_PATH is not usable as an app namespace (R-108)", name)
writeJSON(w, http.StatusConflict, apiResponse{OK: false, Error: why})
return
}
deployReq := stacks.DeployRequest{ deployReq := stacks.DeployRequest{
StackName: name, StackName: name,
Values: body.Values, Values: body.Values,
@@ -520,6 +534,24 @@ func (r *Router) startGatedByMissingDrive(name string) (bool, string) {
return false, "" return false, ""
} }
// desiredStateForAction maps a stack action to the customer intent it expresses, or (_, false) for
// an action that expresses none. Pure, so the §8.1/§1.3 mapping is testable without a Manager.
//
// `restart` and `update` both mean running: a customer who updates or restarts an app is asking for
// it to be up afterwards, and both end in `compose up -d`. Anything not listed here — an unknown
// action string — records nothing rather than guessing, so a future action cannot silently acquire
// an intent it was never meant to carry.
func desiredStateForAction(action string) (string, bool) {
switch action {
case "start", "restart", "update":
return stacks.DesiredStateRunning, true
case "stop":
return stacks.DesiredStateStopped, true
default:
return "", false
}
}
func (r *Router) actionStack(w http.ResponseWriter, action, name string) { func (r *Router) actionStack(w http.ResponseWriter, action, name string) {
r.logger.Printf("[INFO] [api] %s requested for stack: %s", action, name) r.logger.Printf("[INFO] [api] %s requested for stack: %s", action, name)
r.dbg("actionStack: action=%s name=%s", action, name) r.dbg("actionStack: action=%s name=%s", action, name)
@@ -565,6 +597,26 @@ func (r *Router) actionStack(w http.ResponseWriter, action, name string) {
} }
} }
// R-166: THE CUSTOMER-INTENT POINT. This switch is where a human's decision about whether their
// app should be running enters the system, and until v0.189.0 that decision was recorded nowhere
// — so the box had to infer it from container counts, and inferred wrong for a power cut and for
// an interrupted backup alike.
//
// Written BEFORE the act (§8.2) and a failed write REFUSES the act: performing a stop whose
// intent could not be recorded would recreate exactly the ambiguity this closes. Both gates that
// can legitimately refuse an action (protected-stack, drive-absent, memory) have already run
// above, so nothing is recorded for an action that was never going to happen.
if desired, ok := desiredStateForAction(action); ok {
if derr := r.stackMgr.SetDesiredState(name, desired); derr != nil {
r.logger.Printf("[ERROR] [api] %s for %s refused: could not record desired state: %v", action, name, derr)
writeJSON(w, http.StatusInternalServerError, apiResponse{
OK: false,
Error: "A művelet nem hajtható végre: az alkalmazás beállításai nem menthetők.",
})
return
}
}
var err error var err error
switch action { switch action {
case "start": case "start":
@@ -958,6 +1010,7 @@ func (r *Router) triggerBackup(w http.ResponseWriter, _ *http.Request) {
} }
r.logger.Println("[INFO] [api] Manual app-data backup (DB dump) triggered") r.logger.Println("[INFO] [api] Manual app-data backup (DB dump) triggered")
r.backupMgr.MarkManualRun() // R-182: operator-triggered — its digest must not be collapsed into the nightly one
go r.backupMgr.RunDBDumps(context.Background()) go r.backupMgr.RunDBDumps(context.Background())
writeJSON(w, http.StatusOK, apiResponse{OK: true, Message: "Mentés elindítva"}) writeJSON(w, http.StatusOK, apiResponse{OK: true, Message: "Mentés elindítva"})
+24 -10
View File
@@ -20,14 +20,18 @@ type StackDataProvider interface {
ListDeployedStacks() []StackSummary ListDeployedStacks() []StackSummary
GetStackHDDMounts(name string) []string GetStackHDDMounts(name string) []string
GetStackHDDPath(name string) string // raw HDD_PATH from app.yaml (empty if no HDD) GetStackHDDPath(name string) string // raw HDD_PATH from app.yaml (empty if no HDD)
// GetImportRoot returns the CANONICAL drop-zone root (R-75): <system namespace root>/userdata/import.
// It is app-INDEPENDENT and lives on the SYSTEM drive, so ${IMPORT_PATH} binds cannot be resolved
// from GetStackHDDPath. Empty when unresolvable — structuralGuard refuses such binds loudly.
GetImportRoot() string
GetDockerVolumes(name string) []string // full Docker volume names (project-prefixed) GetDockerVolumes(name string) []string // full Docker volume names (project-prefixed)
StopStack(name string) error StopStack(name string) error
StartStack(name string) error StartStack(name string) error
RefreshAndIsRunning(name string) bool RefreshAndIsRunning(name string) bool
// GetStackRecoveryInfo returns the data needed to capture a SECRET-FREE recovery unit // GetStackRecoveryInfo returns the data needed to capture a recovery unit: the stack dir,
// (Phase 2): the stack dir, pinned image tags, the non-secret env, and the NAMES of the // pinned image tags, the non-secret env, the NAMES of the secret/data-key env vars, and (D5)
// secret/data-key env vars (values are NEVER returned — they are recovered at restore time // the decrypted VALUES of the portable class. A WITHHELD secret's value is never returned —
// from the guest's own app.yaml, live or via the PBS whole-guest snapshot). ok=false if the // it is recovered at restore time from the guest's app.yaml, or regenerated. ok=false if the
// stack is unknown. // stack is unknown.
GetStackRecoveryInfo(name string) (RecoveryInfo, bool) GetStackRecoveryInfo(name string) (RecoveryInfo, bool)
@@ -58,17 +62,27 @@ type StackDataProvider interface {
GetStackClassifiedBinds(name string) ([]ClassifiedBind, bool) GetStackClassifiedBinds(name string) ([]ClassifiedBind, bool)
} }
// RecoveryInfo carries everything needed to write a secret-free recovery unit for a stack. // RecoveryInfo carries everything needed to write a recovery unit for a stack.
// It deliberately holds NO secret values — only the names of secret/data-key env vars, so the //
// manifest can record what must be recovered from elsewhere (guest app.yaml / PBS) without the // D5: it now carries the VALUES of the PORTABLE secret class (stacks.PortableSecretEnvVars — every
// unit ever storing a secret or a data-encrypting key. // `type: secret` field bar the nonPortableSecrets register), because a Tier-1/2 restore that depends
// on the guest for a data-encrypting key or a DB password is not independent of the guest at all: the
// data sits safely on the drive and cannot be read back. The EXCLUDED class (`type: password` admin
// logins) is still name-only and never leaves the guest.
type RecoveryInfo struct { type RecoveryInfo struct {
StackDir string // dir holding docker-compose.yml + .felhom.yml + app.yaml StackDir string // dir holding docker-compose.yml + .felhom.yml + app.yaml
DisplayName string // app display name DisplayName string // app display name
ImagePins []string // pinned image tags from compose `image:` lines (re-pulled on restore) ImagePins []string // pinned image tags from compose `image:` lines (re-pulled on restore)
NonSecretEnv map[string]string // env with all secret/password/data-key values removed (plaintext only) NonSecretEnv map[string]string // env with ALL secret/password values removed (plaintext only)
SecretEnvVars []string // NAMES of stripped secret/password fields (recovered from guest/PBS) SecretEnvVars []string // NAMES of every secret/password field
DataKeyEnvVars []string // NAMES of data-encrypting-key fields (fail-closed gate on restore) DataKeyEnvVars []string // NAMES of data-encrypting-key fields (fail-closed gate on restore)
// PortableSecretEnvVars are the NAMES of the secrets that travel in the unit (D5), and
// PortableSecrets their DECRYPTED values. A name present here but absent from PortableSecrets was
// unset/empty in the guest's app.yaml — the restore's fail-closed gate decides what that means.
// Never logged, never in the manifest's value space: the values reach disk only inside the unit's
// 0600 app.yaml.
PortableSecretEnvVars []string
PortableSecrets map[string]string
} }
// ParseComposeImages extracts the pinned image references (`image: repo:tag`) from a // ParseComposeImages extracts the pinned image references (`image: repo:tag`) from a
+35 -13
View File
@@ -59,6 +59,8 @@ const (
reasonEscape = "path escapes the drive root" reasonEscape = "path escapes the drive root"
reasonBareRoot = "bare drive-root bind would capture the backups tree" reasonBareRoot = "bare drive-root bind would capture the backups tree"
reasonReserved = "path inside the reserved backups zone" reasonReserved = "path inside the reserved backups zone"
// reasonNoImportRoot: a ${IMPORT_PATH} bind with no resolvable system namespace root (R-75).
reasonNoImportRoot = "canonical import root unresolvable (system_data_path unconfigured)"
) )
// ComputeCaptureSet resolves an app's classified binds into the tier-filtered absolute capture set. // ComputeCaptureSet resolves an app's classified binds into the tier-filtered absolute capture set.
@@ -71,16 +73,17 @@ const (
// so the engines' no-block branch stays byte-identical to today (the SQ5 cost-regression guard). // so the engines' no-block branch stays byte-identical to today (the SQ5 cost-regression guard).
// //
// Resolution: RootHDD → path.Join(hddPath, relPath); RootUserdata → path.Join(hddPath, "userdata", // Resolution: RootHDD → path.Join(hddPath, relPath); RootUserdata → path.Join(hddPath, "userdata",
// relPath). Guards run AFTER the tier filter, so Skipped means exactly "would have been captured by // relPath); RootImport → path.Join(importRoot, relPath) — the SYSTEM drive, never hddPath (R-75).
// this tier, refused for structural safety". // Guards run AFTER the tier filter, so Skipped means exactly "would have been captured by this tier,
func ComputeCaptureSet(binds []ClassifiedBind, hasClassification bool, tier CaptureTier, hddPath string) CaptureSet { // refused for structural safety".
func ComputeCaptureSet(binds []ClassifiedBind, hasClassification bool, tier CaptureTier, hddPath, importRoot string) CaptureSet {
if !hasClassification { if !hasClassification {
return CaptureSet{HasClassification: false} return CaptureSet{HasClassification: false}
} }
cs := CaptureSet{HasClassification: true} cs := CaptureSet{HasClassification: true}
// Stages 13: tier filter → structural guards → equal-Abs collapse (shared with ComputeFabBuckets). // Stages 13: tier filter → structural guards → equal-Abs collapse (shared with ComputeFabBuckets).
uniq, skipped := resolveGuardCollapse(binds, hddPath, func(c BindClass) bool { return tierKeeps(tier, c) }) uniq, skipped := resolveGuardCollapse(binds, hddPath, importRoot, func(c BindClass) bool { return tierKeeps(tier, c) })
cs.Skipped = skipped cs.Skipped = skipped
// Stage 4: containment dedup — drop any path whose ancestor is already present (keep the ancestor). // Stage 4: containment dedup — drop any path whose ancestor is already present (keep the ancestor).
@@ -107,18 +110,18 @@ func ComputeCaptureSet(binds []ClassifiedBind, hasClassification bool, tier Capt
// collapse (mandatory > optional > excluded; ties by smaller Root/RelPath). It does NOT apply // collapse (mandatory > optional > excluded; ties by smaller Root/RelPath). It does NOT apply
// containment dedup — the caller decides (ComputeCaptureSet does; ComputeFabBuckets must not, so a // containment dedup — the caller decides (ComputeCaptureSet does; ComputeFabBuckets must not, so a
// mandatory child inside an excluded parent stays independently addressable). // mandatory child inside an excluded parent stays independently addressable).
func resolveGuardCollapse(binds []ClassifiedBind, hddPath string, keep func(BindClass) bool) (uniq []CapturePath, skipped []SkippedPath) { func resolveGuardCollapse(binds []ClassifiedBind, hddPath, importRoot string, keep func(BindClass) bool) (uniq []CapturePath, skipped []SkippedPath) {
var resolved []CapturePath var resolved []CapturePath
for _, b := range binds { for _, b := range binds {
if !keep(b.Class) { if !keep(b.Class) {
continue continue
} }
if reason, bad := structuralGuard(b.Root, b.RelPath); bad { if reason, bad := structuralGuard(b.Root, b.RelPath, importRoot); bad {
skipped = append(skipped, SkippedPath{Root: b.Root, RelPath: b.RelPath, Class: b.Class, Reason: reason}) skipped = append(skipped, SkippedPath{Root: b.Root, RelPath: b.RelPath, Class: b.Class, Reason: reason})
continue continue
} }
resolved = append(resolved, CapturePath{ resolved = append(resolved, CapturePath{
Abs: resolveAbs(hddPath, b.Root, b.RelPath), Root: b.Root, RelPath: b.RelPath, Class: b.Class, Abs: resolveAbs(hddPath, importRoot, b.Root, b.RelPath), Root: b.Root, RelPath: b.RelPath, Class: b.Class,
}) })
} }
byAbs := make(map[string]CapturePath, len(resolved)) byAbs := make(map[string]CapturePath, len(resolved))
@@ -153,12 +156,12 @@ type FabBuckets struct {
// selection UI + plan. Same resolution + structural guards + equal-Abs collapse as ComputeCaptureSet // selection UI + plan. Same resolution + structural guards + equal-Abs collapse as ComputeCaptureSet
// (via resolveGuardCollapse), bucketed by class, no cross-bucket containment dedup. Each bucket is // (via resolveGuardCollapse), bucketed by class, no cross-bucket containment dedup. Each bucket is
// Abs-sorted (deterministic). // Abs-sorted (deterministic).
func ComputeFabBuckets(binds []ClassifiedBind, hasClassification bool, hddPath string) FabBuckets { func ComputeFabBuckets(binds []ClassifiedBind, hasClassification bool, hddPath, importRoot string) FabBuckets {
if !hasClassification { if !hasClassification {
return FabBuckets{HasClassification: false} return FabBuckets{HasClassification: false}
} }
fb := FabBuckets{HasClassification: true} fb := FabBuckets{HasClassification: true}
uniq, skipped := resolveGuardCollapse(binds, hddPath, func(BindClass) bool { return true }) uniq, skipped := resolveGuardCollapse(binds, hddPath, importRoot, func(BindClass) bool { return true })
fb.Skipped = skipped fb.Skipped = skipped
for _, cp := range uniq { for _, cp := range uniq {
switch cp.Class { switch cp.Class {
@@ -201,10 +204,19 @@ func tierKeeps(tier CaptureTier, class BindClass) bool {
// there (ParseComposeClassifiableBinds path.Cleans; ValidateBackupSpec vets only SPEC entries), so an // there (ParseComposeClassifiableBinds path.Cleans; ValidateBackupSpec vets only SPEC entries), so an
// unlisted writable "${HDD_PATH}/../x" bind reaches here classed mandatory — this guard is // unlisted writable "${HDD_PATH}/../x" bind reaches here classed mandatory — this guard is
// load-bearing security, not defence-in-depth. // load-bearing security, not defence-in-depth.
func structuralGuard(root BindRoot, relPath string) (reason string, bad bool) { func structuralGuard(root BindRoot, relPath, importRoot string) (reason string, bad bool) {
if relPathEscapes(relPath) { if relPathEscapes(relPath) {
return reasonEscape, true return reasonEscape, true
} }
// RootImport (R-75) resolves against the SYSTEM drive, not hddPath. If that root is unresolvable
// (system_data_path unconfigured) the bind cannot be placed at all — refuse it LOUDLY into Skipped
// rather than let resolveAbs join onto "" and produce a relative, wrong-drive path. The other two
// roots cannot hit this: hddPath is checked by their own callers.
if root == RootImport && importRoot == "" {
return reasonNoImportRoot, true
}
// A bare ${IMPORT_PATH} bind is allowed: it resolves to <sysNS>/userdata/import, which nests no
// backups/ tree (backups live at <sysNS>/backups, a sibling of userdata).
if root == RootHDD { if root == RootHDD {
if relPath == "" { if relPath == "" {
return reasonBareRoot, true // bare ${HDD_PATH} would nest <hddPath>/backups into the capture return reasonBareRoot, true // bare ${HDD_PATH} would nest <hddPath>/backups into the capture
@@ -233,12 +245,22 @@ func relPathEscapes(relPath string) bool {
} }
// resolveAbs maps a guarded (root, relPath) to its in-container absolute path via slash algebra. // resolveAbs maps a guarded (root, relPath) to its in-container absolute path via slash algebra.
func resolveAbs(hddPath string, root BindRoot, relPath string) string { //
if root == RootUserdata { // RootImport is the one root that does NOT resolve against hddPath: the canonical drop-zone lives on
// the SYSTEM drive (R-75), so importRoot is supplied separately by the caller. Resolving it against
// hddPath would silently name a directory on the WRONG DRIVE — a .fab opt-in would then capture (or
// on restore, write) somewhere that merely looks plausible. An empty importRoot is the unresolvable
// case and is refused upstream by structuralGuard, never silently joined.
func resolveAbs(hddPath, importRoot string, root BindRoot, relPath string) string {
switch root {
case RootUserdata:
return path.Join(hddPath, "userdata", relPath) return path.Join(hddPath, "userdata", relPath)
} case RootImport:
return path.Join(importRoot, relPath)
default:
return path.Join(hddPath, relPath) return path.Join(hddPath, relPath)
} }
}
// strongerCapture picks the winner of an equal-Abs collision: mandatory beats optional; on equal // strongerCapture picks the winner of an equal-Abs collision: mandatory beats optional; on equal
// class strength, the lexicographically-smaller (Root, RelPath) wins (determinism). // class strength, the lexicographically-smaller (Root, RelPath) wins (determinism).
@@ -38,12 +38,12 @@ func TestComputeCaptureSet_PerTierSplit(t *testing.T) {
{ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media/photos", ReadOnly: true}, Class: ClassOptional, Origin: OriginExplicit}, {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media/photos", ReadOnly: true}, Class: ClassOptional, Origin: OriginExplicit},
} }
off := ComputeCaptureSet(binds, true, TierOffsite, drv) off := ComputeCaptureSet(binds, true, TierOffsite, drv, "")
if got, want := absList(off), []string{hdd("appdata/immich")}; !reflect.DeepEqual(got, want) { if got, want := absList(off), []string{hdd("appdata/immich")}; !reflect.DeepEqual(got, want) {
t.Errorf("offsite Paths = %v, want %v (mandatory only — the :ro optional must NOT ship offsite)", got, want) t.Errorf("offsite Paths = %v, want %v (mandatory only — the :ro optional must NOT ship offsite)", got, want)
} }
sec := ComputeCaptureSet(binds, true, TierSecondary, drv) sec := ComputeCaptureSet(binds, true, TierSecondary, drv, "")
want := []string{hdd("appdata/immich"), udat("media/photos")} want := []string{hdd("appdata/immich"), udat("media/photos")}
if got := absList(sec); !reflect.DeepEqual(got, want) { if got := absList(sec); !reflect.DeepEqual(got, want) {
t.Errorf("secondary Paths = %v, want %v (sorted)", got, want) t.Errorf("secondary Paths = %v, want %v (sorted)", got, want)
@@ -72,7 +72,7 @@ func TestComputeCaptureSet_LegacyInert(t *testing.T) {
{ComposeBind: ComposeBind{Root: RootHDD, RelPath: "appdata/sonarr"}, Origin: OriginLegacy}, {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "appdata/sonarr"}, Origin: OriginLegacy},
} }
for _, tier := range []CaptureTier{TierOffsite, TierSecondary} { for _, tier := range []CaptureTier{TierOffsite, TierSecondary} {
cs := ComputeCaptureSet(binds, false, tier, drv) cs := ComputeCaptureSet(binds, false, tier, drv, "")
if cs.HasClassification { if cs.HasClassification {
t.Errorf("%s: HasClassification=true for a legacy app", tier) t.Errorf("%s: HasClassification=true for a legacy app", tier)
} }
@@ -94,7 +94,7 @@ func TestComputeCaptureSet_ExcludedInvisible(t *testing.T) {
{ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "import/paperless"}, Class: ClassExcluded, Origin: OriginExplicit}, {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "import/paperless"}, Class: ClassExcluded, Origin: OriginExplicit},
} }
for _, tier := range []CaptureTier{TierOffsite, TierSecondary} { for _, tier := range []CaptureTier{TierOffsite, TierSecondary} {
cs := ComputeCaptureSet(binds, true, tier, drv) cs := ComputeCaptureSet(binds, true, tier, drv, "")
if got, want := absList(cs), []string{hdd("appdata/paperless/media")}; !reflect.DeepEqual(got, want) { if got, want := absList(cs), []string{hdd("appdata/paperless/media")}; !reflect.DeepEqual(got, want) {
t.Errorf("%s Paths = %v, want %v (excluded filtered)", tier, got, want) t.Errorf("%s Paths = %v, want %v (excluded filtered)", tier, got, want)
} }
@@ -113,7 +113,7 @@ func TestComputeCaptureSet_StructuralGuards(t *testing.T) {
{ComposeBind: ComposeBind{Root: RootHDD, RelPath: "backups/primary/x"}, Class: ClassMandatory, Origin: OriginDefaultWritable}, // d3 reserved zone {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "backups/primary/x"}, Class: ClassMandatory, Origin: OriginDefaultWritable}, // d3 reserved zone
{ComposeBind: ComposeBind{Root: RootUserdata, RelPath: ""}, Class: ClassMandatory, Origin: OriginDefaultWritable}, // d4 bare userdata — ALLOWED {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: ""}, Class: ClassMandatory, Origin: OriginDefaultWritable}, // d4 bare userdata — ALLOWED
} }
cs := ComputeCaptureSet(binds, true, TierOffsite, drv) cs := ComputeCaptureSet(binds, true, TierOffsite, drv, "")
// Paths: ONLY d4's userdata root — no escaped root, no backups/ anywhere. // Paths: ONLY d4's userdata root — no escaped root, no backups/ anywhere.
if got, want := absList(cs), []string{udat("")}; !reflect.DeepEqual(got, want) { if got, want := absList(cs), []string{udat("")}; !reflect.DeepEqual(got, want) {
@@ -156,7 +156,7 @@ func TestComputeCaptureSet_LegitDotDotName(t *testing.T) {
binds := []ClassifiedBind{ binds := []ClassifiedBind{
{ComposeBind: ComposeBind{Root: RootHDD, RelPath: "appdata/a..b"}, Class: ClassMandatory, Origin: OriginExplicit}, {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "appdata/a..b"}, Class: ClassMandatory, Origin: OriginExplicit},
} }
cs := ComputeCaptureSet(binds, true, TierOffsite, drv) cs := ComputeCaptureSet(binds, true, TierOffsite, drv, "")
if got, want := absList(cs), []string{hdd("appdata/a..b")}; !reflect.DeepEqual(got, want) { if got, want := absList(cs), []string{hdd("appdata/a..b")}; !reflect.DeepEqual(got, want) {
t.Errorf("Paths = %v, want %v (a..b is a legit name, not traversal)", got, want) t.Errorf("Paths = %v, want %v (a..b is a legit name, not traversal)", got, want)
} }
@@ -174,7 +174,7 @@ func TestComputeCaptureSet_ContainmentAndCollision(t *testing.T) {
{ComposeBind: ComposeBind{Root: RootHDD, RelPath: "userdata/media"}, Class: ClassOptional, Origin: OriginExplicit}, // Abs collides with next {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "userdata/media"}, Class: ClassOptional, Origin: OriginExplicit}, // Abs collides with next
{ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media"}, Class: ClassMandatory, Origin: OriginExplicit}, // same Abs, mandatory {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media"}, Class: ClassMandatory, Origin: OriginExplicit}, // same Abs, mandatory
} }
cs := ComputeCaptureSet(binds, true, TierSecondary, drv) cs := ComputeCaptureSet(binds, true, TierSecondary, drv, "")
want := []string{hdd("appdata/paperless"), udat("media")} want := []string{hdd("appdata/paperless"), udat("media")}
if got := absList(cs); !reflect.DeepEqual(got, want) { if got := absList(cs); !reflect.DeepEqual(got, want) {
@@ -186,7 +186,7 @@ func TestComputeCaptureSet_ContainmentAndCollision(t *testing.T) {
} }
// determinism: recompute and compare full struct // determinism: recompute and compare full struct
cs2 := ComputeCaptureSet(binds, true, TierSecondary, drv) cs2 := ComputeCaptureSet(binds, true, TierSecondary, drv, "")
if !reflect.DeepEqual(cs, cs2) { if !reflect.DeepEqual(cs, cs2) {
t.Error("ComputeCaptureSet is non-deterministic across runs") t.Error("ComputeCaptureSet is non-deterministic across runs")
} }
+54 -16
View File
@@ -34,12 +34,23 @@ type BindRoot string
const ( const (
RootUserdata BindRoot = "userdata" // relative to ${USERDATA_PATH} RootUserdata BindRoot = "userdata" // relative to ${USERDATA_PATH}
RootHDD BindRoot = "hdd" // relative to ${HDD_PATH} RootHDD BindRoot = "hdd" // relative to ${HDD_PATH}
// RootImport is relative to ${IMPORT_PATH} — the CANONICAL drop-zone root (R-75). Unlike the
// other two it does NOT resolve against the app's own drive: it lives on the system drive's
// namespace, so every app's ingest folder is in one place. Resolvers therefore need the import
// root passed in separately; they cannot derive it from hddPath.
RootImport BindRoot = "import"
) )
// BackupSpec is the .felhom.yml `backup:` block. Paths are forward-slash, relative, path.Clean'd. // BackupSpec is the .felhom.yml `backup:` block. Paths are forward-slash, relative, path.Clean'd.
type BackupSpec struct { type BackupSpec struct {
Userdata []BindSpec `yaml:"userdata,omitempty" json:"userdata,omitempty"` Userdata []BindSpec `yaml:"userdata,omitempty" json:"userdata,omitempty"`
HDD []BindSpec `yaml:"hdd,omitempty" json:"hdd,omitempty"` HDD []BindSpec `yaml:"hdd,omitempty" json:"hdd,omitempty"`
// Import classifies ${IMPORT_PATH}-relative binds (R-75). An app whose ingest bind moved from
// ${USERDATA_PATH}/import/<app> to ${IMPORT_PATH}/<app> MUST move its backup entry here in the
// same change: ValidateBackupSpec rejects an entry matching no compose bind, and the rejection is
// WHOLE-BLOCK, so a stale `userdata: import/<app>` would discard the app's OTHER classifications
// (e.g. an hdd appdata path classed mandatory) and silently degrade it to legacy.
Import []BindSpec `yaml:"import,omitempty" json:"import,omitempty"`
} }
// BindSpec is one classified entry in a BackupSpec. // BindSpec is one classified entry in a BackupSpec.
@@ -86,6 +97,42 @@ func validClass(c BindClass) bool {
} }
} }
// ValidateRelPath is THE path-safety refusal set for every ${VAR}-relative catalog path — the
// `backup:` block and `data_paths:` both run through it, so there is exactly ONE definition of what
// a safe relative path is. Refuses: empty, backslash, absolute, non-path.Clean'd, and any leading
// ".." escape. It deliberately does NOT check "matches a compose bind" — that rule needs the bind
// list and differs per caller (whole-block reject for backup:, per-entry for data_paths:).
func ValidateRelPath(root BindRoot, p string) error {
where := fmt.Sprintf("%s[%q]", root, p)
if p == "" {
return fmt.Errorf("%s: empty path", where)
}
if strings.ContainsRune(p, '\\') {
return fmt.Errorf("%s: backslash in path (paths are forward-slash relative)", where)
}
if path.IsAbs(p) {
return fmt.Errorf("%s: absolute path (must be relative to the %s root)", where, root)
}
if p != path.Clean(p) {
return fmt.Errorf("%s: non-clean path (want %q)", where, path.Clean(p))
}
// path.Clean has run — ".." can only survive as a leading "../" segment.
if p == ".." || strings.HasPrefix(p, "../") {
return fmt.Errorf("%s: path escapes the root (..)", where)
}
return nil
}
// ValidRoot reports whether r is one of the three known bind roots.
func ValidRoot(r BindRoot) bool {
switch r {
case RootUserdata, RootHDD, RootImport:
return true
default:
return false
}
}
// ValidateBackupSpec checks a parsed backup block against the app's actual compose binds and returns // ValidateBackupSpec checks a parsed backup block against the app's actual compose binds and returns
// the FIRST defect (whole-block semantics — the caller rejects the ENTIRE block on any error, so the // the FIRST defect (whole-block semantics — the caller rejects the ENTIRE block on any error, so the
// app degrades to legacy rather than partially classifying). A nil spec is vacuously valid (legacy). // app degrades to legacy rather than partially classifying). A nil spec is vacuously valid (legacy).
@@ -113,21 +160,8 @@ func ValidateBackupSpec(spec *BackupSpec, binds []ComposeBind) error {
if !validClass(e.Class) { if !validClass(e.Class) {
return fmt.Errorf("%s: invalid class %q (want mandatory|optional|excluded)", where, e.Class) return fmt.Errorf("%s: invalid class %q (want mandatory|optional|excluded)", where, e.Class)
} }
if e.Path == "" { if err := ValidateRelPath(root, e.Path); err != nil {
return fmt.Errorf("%s: empty path", where) return err
}
if strings.ContainsRune(e.Path, '\\') {
return fmt.Errorf("%s: backslash in path (paths are forward-slash relative)", where)
}
if path.IsAbs(e.Path) {
return fmt.Errorf("%s: absolute path (must be relative to the %s root)", where, root)
}
if e.Path != path.Clean(e.Path) {
return fmt.Errorf("%s: non-clean path (want %q)", where, path.Clean(e.Path))
}
// path.Clean has run — ".." can only survive as a leading "../" segment.
if e.Path == ".." || strings.HasPrefix(e.Path, "../") {
return fmt.Errorf("%s: path escapes the root (..)", where)
} }
key := string(root) + "\x00" + e.Path key := string(root) + "\x00" + e.Path
if seen[key] { if seen[key] {
@@ -143,7 +177,10 @@ func ValidateBackupSpec(spec *BackupSpec, binds []ComposeBind) error {
if err := check(RootUserdata, spec.Userdata); err != nil { if err := check(RootUserdata, spec.Userdata); err != nil {
return err return err
} }
return check(RootHDD, spec.HDD) if err := check(RootHDD, spec.HDD); err != nil {
return err
}
return check(RootImport, spec.Import)
} }
// ClassifyBinds resolves every compose bind to a class + origin, applying the two-level default. The // ClassifyBinds resolves every compose bind to a class + origin, applying the two-level default. The
@@ -181,6 +218,7 @@ func ClassifyBinds(spec *BackupSpec, binds []ComposeBind) (classified []Classifi
} }
add(RootUserdata, spec.Userdata) add(RootUserdata, spec.Userdata)
add(RootHDD, spec.HDD) add(RootHDD, spec.HDD)
add(RootImport, spec.Import)
for _, b := range binds { for _, b := range binds {
cb := ClassifiedBind{ComposeBind: b} cb := ClassifiedBind{ComposeBind: b}
+1
View File
@@ -765,6 +765,7 @@ func getMariaDBPassword(ctx context.Context, containerID string) string {
// - else longest known prefix → handles <stack>_postgres / <stack>-1 / compose-suffixed names. // - else longest known prefix → handles <stack>_postgres / <stack>-1 / compose-suffixed names.
// - else → candidate (fall back to today's suffix-strip; preserves behaviour when // - else → candidate (fall back to today's suffix-strip; preserves behaviour when
// the stack list is empty/unavailable, so nothing regresses). // the stack list is empty/unavailable, so nothing regresses).
//
// A nil/empty `known` map = the legacy fast path (pure suffix-strip). // A nil/empty `known` map = the legacy fast path (pure suffix-strip).
func deriveStackName(containerName string, known map[string]bool) string { func deriveStackName(containerName string, known map[string]bool) string {
candidate := suffixStripStackName(containerName) candidate := suffixStripStackName(containerName)
@@ -21,7 +21,7 @@ func TestComputeFabBuckets_Classified(t *testing.T) {
{ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media/movies"}, Class: ClassExcluded}, {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media/movies"}, Class: ClassExcluded},
{ComposeBind: ComposeBind{Root: RootHDD, RelPath: "appdata/app"}, Class: ClassMandatory}, {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "appdata/app"}, Class: ClassMandatory},
} }
fb := ComputeFabBuckets(binds, true, drv) fb := ComputeFabBuckets(binds, true, drv, "")
if !fb.HasClassification { if !fb.HasClassification {
t.Fatal("HasClassification must be true") t.Fatal("HasClassification must be true")
} }
@@ -39,7 +39,7 @@ func TestComputeFabBuckets_Classified(t *testing.T) {
// legacy (no block) → empty buckets (the full-root capture stays out of the classified plan). // legacy (no block) → empty buckets (the full-root capture stays out of the classified plan).
func TestComputeFabBuckets_LegacyEmpty(t *testing.T) { func TestComputeFabBuckets_LegacyEmpty(t *testing.T) {
binds := []ClassifiedBind{{ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media/tv"}, Origin: OriginLegacy}} binds := []ClassifiedBind{{ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media/tv"}, Origin: OriginLegacy}}
fb := ComputeFabBuckets(binds, false, drv) fb := ComputeFabBuckets(binds, false, drv, "")
if fb.HasClassification || fb.Mandatory != nil || fb.Optional != nil || fb.Excluded != nil { if fb.HasClassification || fb.Mandatory != nil || fb.Optional != nil || fb.Excluded != nil {
t.Errorf("legacy app must yield empty buckets, got %+v", fb) t.Errorf("legacy app must yield empty buckets, got %+v", fb)
} }
@@ -52,7 +52,7 @@ func TestComputeFabBuckets_GuardsAllClasses(t *testing.T) {
{ComposeBind: ComposeBind{Root: RootHDD, RelPath: "../evil"}, Class: ClassExcluded}, {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "../evil"}, Class: ClassExcluded},
{ComposeBind: ComposeBind{Root: RootHDD, RelPath: "appdata/ok"}, Class: ClassMandatory}, {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "appdata/ok"}, Class: ClassMandatory},
} }
fb := ComputeFabBuckets(binds, true, drv) fb := ComputeFabBuckets(binds, true, drv, "")
for _, b := range [][]CapturePath{fb.Mandatory, fb.Optional, fb.Excluded} { for _, b := range [][]CapturePath{fb.Mandatory, fb.Optional, fb.Excluded} {
for _, p := range b { for _, p := range b {
if p.RelPath == "../evil" { if p.RelPath == "../evil" {
@@ -71,7 +71,7 @@ func TestComputeFabBuckets_NoCrossBucketContainment(t *testing.T) {
{ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media"}, Class: ClassExcluded}, {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media"}, Class: ClassExcluded},
{ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media/books"}, Class: ClassMandatory}, {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media/books"}, Class: ClassMandatory},
} }
fb := ComputeFabBuckets(binds, true, drv) fb := ComputeFabBuckets(binds, true, drv, "")
if got, want := bucketAbs(fb.Mandatory), []string{udat("media/books")}; !reflect.DeepEqual(got, want) { if got, want := bucketAbs(fb.Mandatory), []string{udat("media/books")}; !reflect.DeepEqual(got, want) {
t.Errorf("mandatory child must survive independently: Mandatory = %v, want %v", got, want) t.Errorf("mandatory child must survive independently: Mandatory = %v, want %v", got, want)
} }
@@ -86,7 +86,7 @@ func TestComputeFabBuckets_EqualAbsMandatoryWins(t *testing.T) {
{ComposeBind: ComposeBind{Root: RootHDD, RelPath: "userdata/media"}, Class: ClassOptional}, {ComposeBind: ComposeBind{Root: RootHDD, RelPath: "userdata/media"}, Class: ClassOptional},
{ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media"}, Class: ClassMandatory}, {ComposeBind: ComposeBind{Root: RootUserdata, RelPath: "media"}, Class: ClassMandatory},
} }
fb := ComputeFabBuckets(binds, true, drv) fb := ComputeFabBuckets(binds, true, drv, "")
if got, want := bucketAbs(fb.Mandatory), []string{udat("media")}; !reflect.DeepEqual(got, want) { if got, want := bucketAbs(fb.Mandatory), []string{udat("media")}; !reflect.DeepEqual(got, want) {
t.Errorf("collapsed path must land in Mandatory, got Mandatory=%v", got) t.Errorf("collapsed path must land in Mandatory, got Mandatory=%v", got)
} }
@@ -0,0 +1,69 @@
package appbackup
import "testing"
// R-203 — the ONE drive-kind rule. Table-driven over BOTH drive kinds on purpose: this defect
// survived because it is invisible on the kind that already worked, so a test that only covers the
// enrolled drive proves nothing about the fix.
func TestNamespaceRootFor_BothDriveKinds(t *testing.T) {
const sys = "/mnt/sys_drive"
cases := []struct {
name, drive, want string
}{
// Scenario B — the enrolled drive must be BYTE-IDENTICAL to pre-R-203 behaviour. The
// in-guest mount already IS the namespace root; appending felhom-data here would recreate
// the .../felhom-data/felhom-data/... double-nest NamespaceRoot's comment exists to prevent.
{"enrolled usb", "/mnt/felhom-usb", "/mnt/felhom-usb"},
{"enrolled hdd", "/mnt/felhom-drives/hdd_1", "/mnt/felhom-drives/hdd_1"},
{"enrolled nvme", "/mnt/felhom-drives/nvme-1tb", "/mnt/felhom-drives/nvme-1tb"},
// Scenario A — the system-data fallback gains the segment. This is the case that was wrong.
{"system drive", "/mnt/sys_drive", "/mnt/sys_drive/felhom-data"},
// A trailing slash is the same drive. Before R-203 the backup package's copy of this rule
// compared WITHOUT Clean while the stacks package's copy compared WITH it — so a config value
// with a trailing slash would have flipped the mode in one package and not the other.
{"system drive, trailing slash", "/mnt/sys_drive/", "/mnt/sys_drive/felhom-data"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
if got := NamespaceRootFor(tc.drive, sys); got != tc.want {
t.Fatalf("NamespaceRootFor(%q, %q) = %q, want %q", tc.drive, sys, got, tc.want)
}
})
}
}
// The rule must survive a trailing slash on the SYSTEM path too — it comes from config.
func TestIsEnrolledDrive_CleansBothSides(t *testing.T) {
if IsEnrolledDrive("/mnt/sys_drive", "/mnt/sys_drive/") {
t.Error("a trailing slash on the system path must not make the system drive look enrolled")
}
if IsEnrolledDrive("/mnt/sys_drive/", "/mnt/sys_drive") {
t.Error("a trailing slash on the drive path must not make the system drive look enrolled")
}
if !IsEnrolledDrive("/mnt/felhom-usb", "/mnt/sys_drive") {
t.Error("an enrolled drive must report enrolled")
}
}
// The consequence the whole item is about: the directory an app binds and the directory the capture
// set looks in must be the SAME on both drive kinds.
//
// RED-PROOF: replace `UserdataDir(NamespaceRootFor(drive, sys))` with `UserdataDir(drive)` — the
// pre-R-203 call — and the system-drive row FAILS with the two paths differing by exactly
// `/felhom-data`. That is production behaviour up to v0.196.0.
func TestAppBindAndCaptureRootAgree(t *testing.T) {
const sys = "/mnt/sys_drive"
for _, drive := range []string{"/mnt/felhom-usb", "/mnt/felhom-drives/hdd_1", "/mnt/sys_drive"} {
nsRoot := NamespaceRootFor(drive, sys)
appBind := UserdataDir(nsRoot) // what the deploy sets as ${USERDATA_PATH}
captureRoot := UserdataDir(nsRoot) // what the capture set resolves RootUserdata against
if appBind != captureRoot {
t.Fatalf("drive %q: the app binds %q while the backup captures %q", drive, appBind, captureRoot)
}
// And it must be the canonical location — the one EnsureUserdataSkeleton creates.
if drive == sys && appBind != "/mnt/sys_drive/felhom-data/userdata" {
t.Fatalf("system drive resolved to %q, want the canonical /mnt/sys_drive/felhom-data/userdata", appBind)
}
}
}
+33 -3
View File
@@ -32,6 +32,35 @@ func NamespaceRoot(drivePath string, inGuestDrive bool) string {
return filepath.Join(drivePath, FelhomDataDir) return filepath.Join(drivePath, FelhomDataDir)
} }
// IsEnrolledDrive reports whether a drive path is an ENROLLED user-data drive (Model A: its in-guest
// mount already IS the namespace root) rather than the system-data fallback. It is the ONE comparison
// that decides which NamespaceRoot mode applies, and it lives here so no package re-derives it.
//
// Both sides are Clean'd: `/mnt/sys_drive/` and `/mnt/sys_drive` are the same drive, and a trailing
// slash arriving from config must not silently flip the mode.
func IsEnrolledDrive(drivePath, systemDataPath string) bool {
return filepath.Clean(drivePath) != filepath.Clean(systemDataPath)
}
// NamespaceRootFor is the resolver every caller should use when it holds a bare DRIVE path and the
// system-data path — i.e. everywhere outside the backup package, which already had this rule.
//
// R-203: FIVE call sites passed a bare drive path straight to UserdataDir (and its siblings), which
// take a NAMESPACE ROOT. On an enrolled drive the two coincide, so nothing showed; on the system-data
// fallback they differ by exactly the felhom-data segment, and the app then bound a directory the
// backup never looked at. The run still reported ok. Measured live on demo-hp 2026-08-04:
// the app wrote to /mnt/sys_drive/userdata/media/books while the off-site capture set looked for
// /mnt/sys_drive/felhom-data/userdata/media/books.
//
// THE CONTRACT, restated because four callers got it wrong and a fifth will: UserdataDir,
// PrimaryBackupPath, RecoveryUnitPath and AppDataDir all take a NAMESPACE ROOT. If you are holding
// something that came out of HDD_PATH or a StoragePath, it is a DRIVE path — put it through here
// first. `UserdataDir(bareDrivePath)` still compiles and is still wrong; TestNoBareDrivePathToUserdataDir
// is the guard that keeps the count from growing.
func NamespaceRootFor(drivePath, systemDataPath string) string {
return NamespaceRoot(drivePath, IsEnrolledDrive(drivePath, systemDataPath))
}
// PrimaryBackupPath returns the root primary backup directory under a felhom-data namespace root. // PrimaryBackupPath returns the root primary backup directory under a felhom-data namespace root.
func PrimaryBackupPath(nsRoot string) string { func PrimaryBackupPath(nsRoot string) string {
return filepath.Join(nsRoot, "backups", "primary") return filepath.Join(nsRoot, "backups", "primary")
@@ -40,9 +69,10 @@ func PrimaryBackupPath(nsRoot string) string {
// RecoveryUnitPath returns the per-app self-contained recovery-unit ROOT under a namespace root. // RecoveryUnitPath returns the per-app self-contained recovery-unit ROOT under a namespace root.
// It is the existing per-app backup dir (`backups/primary/<stack>/`) — the legacy name is kept so the // It is the existing per-app backup dir (`backups/primary/<stack>/`) — the legacy name is kept so the
// db-dumps/ and volume-dumps/ already written there need no migration; the unit gains compose/ and // db-dumps/ and volume-dumps/ already written there need no migration; the unit gains compose/ and
// manifest.json as siblings, making the whole dir a complete, recreatable unit (Phase 2). The unit is // manifest.json as siblings, making the whole dir a complete, recreatable unit (Phase 2). Since D5 the
// secret-free: secrets/data-keys are recovered from the guest's own app.yaml (live or via PBS), never // unit's compose/app.yaml CARRIES the portable secret class (data keys, DB passwords, internal signing
// stored here. See backup.recoveryUnit / restore for the capture + restore flow. // secrets) at mode 0600, so a Tier-1/2 restore needs the drive and nothing else; internet-reachable
// admin logins are still withheld. See backup.recoveryUnit / restore for the capture + restore flow.
func RecoveryUnitPath(nsRoot, stackName string) string { func RecoveryUnitPath(nsRoot, stackName string) string {
return filepath.Join(nsRoot, "backups", "primary", stackName) return filepath.Join(nsRoot, "backups", "primary", stackName)
} }
@@ -0,0 +1,163 @@
package appbackup
import (
"os"
"path/filepath"
"slices"
"strings"
"testing"
)
// R-75 Scenario C — DETERMINISM. This is the P6 gate.
//
// The spike measured the naive map-order derivation producing 20 DISTINCT outputs from 20 identical
// runs. fbNeedsRecreate force-recreates FileBrowser on ANY byte difference in the generated config,
// and SyncFileBrowserMounts has ~14 call sites — so a non-deterministic skeleton is a fleet-wide
// FileBrowser restart loop, the v0.151-class bug. 20 identical generations or this fails.
func TestScenarioC_SkeletonDeterminism(t *testing.T) {
// Deliberately UNSORTED input, with duplicates and a deep path, so the function has real work to
// normalise. A sort applied only to the input would not save a map-ordered implementation.
derived := []string{
"media/podcasts", "roms", "media/books", "downloads", "media",
"media/photos", "media/books", "a/b/c/d",
}
const n = 20
first := BuildUserdataSkeleton(derived)
for i := 1; i < n; i++ {
got := BuildUserdataSkeleton(derived)
if !slices.Equal(got, first) {
t.Fatalf("generation %d/%d differs — a non-deterministic skeleton force-recreates FileBrowser on every sync pass\n first: %v\n got: %v",
i+1, n, first, got)
}
}
if !slices.IsSorted(first) {
t.Errorf("skeleton must be sorted, got %v", first)
}
// Ancestor expansion: a deep derived path implies its whole chain.
for _, want := range []string{"a", "a/b", "a/b/c", "a/b/c/d"} {
if !slices.Contains(first, want) {
t.Errorf("ancestor chain incomplete: %q missing from %v", want, first)
}
}
// Dedup: "media/books" appeared twice in the input and "media" both derived and as an ancestor.
for _, d := range []string{"media", "media/books"} {
if c := countOf(first, d); c != 1 {
t.Errorf("%q appears %d times, want exactly 1", d, c)
}
}
}
func countOf(xs []string, want string) int {
n := 0
for _, x := range xs {
if x == want {
n++
}
}
return n
}
// R-75 Scenario D — ZERO REMOVALS, proven by construction.
//
// The derived set drops `documents` (implied by no catalog app) and, after the R-75 move, the two
// import/* entries. The carry-list is what keeps them. This asserts the merged set is a strict
// SUPERSET of the historical hardcoded skeleton for any derived input — including the empty one, the
// fresh-box case where the catalog has not synced yet.
func TestScenarioD_SkeletonNeverDropsACarriedDir(t *testing.T) {
for _, derived := range [][]string{
nil, // fresh box, catalog not yet synced
{"media/podcasts"}, // the one genuinely new entry
{"roms", "downloads", "media/photos"}, // a partial catalog
} {
got := BuildUserdataSkeleton(derived)
for _, carried := range UserdataSkeletonCarry() {
if !slices.Contains(got, carried) {
t.Errorf("derived=%v: carried dir %q was DROPPED — zero-removals violated", derived, carried)
}
}
}
// And the new entry really is added when the catalog implies it.
if !slices.Contains(BuildUserdataSkeleton([]string{"media/podcasts"}), "media/podcasts") {
t.Error("media/podcasts must be added when the catalog implies it")
}
// `documents` is the specific entry the spike flagged: in the carry-list, in no catalog app.
if !slices.Contains(BuildUserdataSkeleton([]string{"media/podcasts"}), "documents") {
t.Error("`documents` must survive — it exists on both demo boxes and may hold customer files")
}
}
// A traversal or absolute entry reaching the skeleton would make EnsureUserdataSkeleton create a
// directory outside the userdata root. The derived set comes from a compose parser, so this is a
// guard on untrusted-ish catalog input, not defence in depth.
func TestSkeletonRefusesEscapes(t *testing.T) {
got := BuildUserdataSkeleton([]string{"../escape", "..", "", "/abs/path", "ok/dir"})
for _, bad := range []string{"../escape", "..", "", "/abs/path"} {
if slices.Contains(got, bad) {
t.Errorf("escape entry %q must not reach the skeleton: %v", bad, got)
}
}
for _, d := range got {
if filepath.IsAbs(d) || d == ".." || len(d) > 3 && d[:3] == "../" {
t.Errorf("unsafe skeleton entry %q", d)
}
}
if !slices.Contains(got, "ok/dir") {
t.Error("a legitimate entry alongside bad ones must still be kept")
}
// "/abs/path" is not dropped outright — it is normalised to a relative path and kept, which is
// safe (it lands under the userdata root). Pin that so the behaviour is a decision, not a guess.
if !slices.Contains(got, "abs/path") {
t.Errorf("an absolute entry should be normalised to relative, got %v", got)
}
}
// EnsureUserdataSkeleton creates every dir it is given and NOTHING ELSE, and never removes.
func TestEnsureUserdataSkeletonCreatesOnly(t *testing.T) {
ns := t.TempDir()
// A pre-existing customer dir that no catalog app implies and the carry-list does not contain.
stray := filepath.Join(UserdataDir(ns), "sajat-mappa")
if err := os.MkdirAll(stray, 0o755); err != nil {
t.Fatal(err)
}
dirs := BuildUserdataSkeleton([]string{"media/podcasts"})
if err := EnsureUserdataSkeleton(ns, dirs); err != nil {
// chown to gid 1000 fails for a non-root test user; the dirs are still created.
t.Logf("EnsureUserdataSkeleton returned %v (expected when not running as root)", err)
}
for _, d := range dirs {
if fi, err := os.Stat(filepath.Join(UserdataDir(ns), d)); err != nil || !fi.IsDir() {
t.Errorf("skeleton dir %q not created: %v", d, err)
}
}
if _, err := os.Stat(stray); err != nil {
t.Errorf("a pre-existing customer dir was removed — zero-removals violated: %v", err)
}
}
// R-75: a DATA drive must never get a per-drive drop-zone from the skeleton. Carrying the old
// `import/*` entries would have the skeleton re-create a dead lookalike on every drive forever —
// one that is also never backed up, since import paths are class: excluded.
//
// This is NOT a zero-removals violation: nothing deletes the dirs a box already has (see
// TestEnsureUserdataSkeletonCreatesOnly). They stop being maintained and stop appearing on fresh boxes.
func TestSkeletonNeverCreatesAPerDriveDropZone(t *testing.T) {
// The catalog no longer implies any ${USERDATA_PATH}/import path — the binds moved to
// ${IMPORT_PATH} — so the only way one could appear is via the carry-list.
for _, derived := range [][]string{nil, {"media/podcasts", "roms"}} {
for _, d := range BuildUserdataSkeleton(derived) {
if d == "import" || strings.HasPrefix(d, "import/") {
t.Errorf("derived=%v: skeleton created a per-drive drop-zone %q — the canonical root is on the SYSTEM drive", derived, d)
}
}
}
for _, c := range UserdataSkeletonCarry() {
if c == "import" || strings.HasPrefix(c, "import/") {
t.Errorf("the carry-list still holds %q", c)
}
}
// A catalog app that genuinely declares a ${USERDATA_PATH}/import/... bind would still be
// honoured — the rule is "don't carry them", not "filter them out".
if !slices.Contains(BuildUserdataSkeleton([]string{"import/valami"}), "import/valami") {
t.Error("a genuinely derived userdata import path must still be created")
}
}
+84 -5
View File
@@ -2,7 +2,10 @@ package appbackup
import ( import (
"os" "os"
"path"
"path/filepath" "path/filepath"
"sort"
"strings"
) )
// Customer-facing userdata layout + the shared-storage ownership convention (v0.66.0). // Customer-facing userdata layout + the shared-storage ownership convention (v0.66.0).
@@ -28,19 +31,91 @@ func UserdataDir(nsRoot string) string {
return filepath.Join(nsRoot, "userdata") return filepath.Join(nsRoot, "userdata")
} }
// UserdataSkeleton is the standard subtree created on every storage path (relative to UserdataDir). // ImportDirName is the single import (drop-zone) subtree name under a userdata root.
const ImportDirName = "import"
// ImportDir returns the CANONICAL drop-zone root under a namespace root (R-75).
//
// Unlike every other userdata dir, this one is drive-INDEPENDENT: the caller resolves it against the
// SYSTEM drive's namespace root, never against the app's own HDD_PATH, so a multi-drive box has
// exactly ONE import tree. That is the whole point. Each drop-zone app has exactly one ingest bind,
// so a per-drive import/ would put a folder that LOOKS like a drop-zone on every drive while only
// one of them does anything — and because import paths are `class: excluded`, files stranded in a
// dead one are never backed up either.
//
// It deliberately stays INSIDE the userdata tree, so the 2775/setgid/GID-1000 convention, the
// FileBrowser mount and the ownership rules all apply to it unchanged.
func ImportDir(nsRoot string) string {
return filepath.Join(UserdataDir(nsRoot), ImportDirName)
}
// UserdataSkeletonCarry is the explicit NON-DERIVED carry-list: every entry the v0.171.0 hardcoded
// skeleton created, retained verbatim and forever.
//
// It exists so the catalog-derived skeleton (R-75) can only ever ADD. That makes the zero-removals
// invariant true BY CONSTRUCTION rather than by review, and it is not hypothetical:
//
// - `documents` is implied by NO catalog app (SPIKE P0(a)) yet exists on both demo boxes and is
// customer-visible — it may hold customer files. Derivation alone would drop it.
//
// It doubles as the fresh-box floor: on a box whose catalog has not synced yet the derived set is
// empty, and the customer still gets the full standard tree instead of a nearly-empty one.
//
// DELIBERATELY ABSENT: `import`, `import/paperless`, `import/calibre`. They were in the v0.171.0
// hardcoded list, and carrying them would have the skeleton RE-CREATE a per-drive drop-zone on every
// drive forever — the exact dead-lookalike R-75 exists to remove, and one that is never backed up
// (`class: excluded`). Zero-removals is about not DELETING what a box already has, not about
// re-creating it on boxes that never had it: nothing here removes the pre-existing dirs on
// demo-felhom / demo-hp, they simply stop being maintained and stop appearing on fresh boxes.
// Verified before the change: both boxes' old drop-zones held ZERO files (2026-07-26). A box with
// pending files in an old drop-zone would need an operator-run move — see REPORT.md.
//
// ASCII, no spaces (flows through ${} interpolation, shell, and the rsync merge walk). // ASCII, no spaces (flows through ${} interpolation, shell, and the rsync merge walk).
func UserdataSkeleton() []string { func UserdataSkeletonCarry() []string {
return []string{ return []string{
"media", "media/movies", "media/tv", "media/music", "media/audiobooks", "media", "media/movies", "media/tv", "media/music", "media/audiobooks",
"media/books", "media/comics", "media/photos", "media/books", "media/comics", "media/photos",
"downloads", "downloads",
"import", "import/paperless", "import/calibre",
"roms", "roms",
"documents", "documents",
} }
} }
// BuildUserdataSkeleton merges the catalog-derived dirs with the carry-list into the final, SORTED
// set. Each entry is expanded to its ancestor chain ("media/podcasts" implies "media"), deduped, and
// sorted.
//
// SORTING IS A HARD REQUIREMENT, not tidiness. The FileBrowser config is regenerated from this set
// and fbNeedsRecreate force-recreates the container on ANY byte difference. Go randomises map
// iteration, and the spike measured the naive map-order derivation producing 20 DISTINCT outputs from
// 20 identical runs (SPIKE P6) — which across SyncFileBrowserMounts' ~14 call sites is a fleet-wide
// FileBrowser restart loop. TestSkeletonDeterminism pins this.
func BuildUserdataSkeleton(derived []string) []string {
set := make(map[string]bool, len(derived)+16)
addChain := func(rel string) {
rel = path.Clean(strings.TrimPrefix(filepath.ToSlash(rel), "/"))
if rel == "" || rel == "." || rel == ".." || strings.HasPrefix(rel, "../") {
return // never let a traversal or an empty entry become a directory to create
}
parts := strings.Split(rel, "/")
for i := range parts {
set[strings.Join(parts[:i+1], "/")] = true
}
}
for _, d := range UserdataSkeletonCarry() {
addChain(d)
}
for _, d := range derived {
addChain(d)
}
out := make([]string, 0, len(set))
for d := range set { // map order is RANDOM — the sort below is what makes this deterministic
out = append(out, d)
}
sort.Strings(out)
return out
}
// EnsureDirOwned creates path (idempotent) and enforces the convention: mode 2775 via an explicit // EnsureDirOwned creates path (idempotent) and enforces the convention: mode 2775 via an explicit
// Chmod incl. setgid (MkdirAll cannot) + group = gid. Setting an arbitrary group needs CAP_CHOWN — // Chmod incl. setgid (MkdirAll cannot) + group = gid. Setting an arbitrary group needs CAP_CHOWN —
// the in-guest controller runs as root, so this succeeds in production. Returns the first hard error. // the in-guest controller runs as root, so this succeeds in production. Returns the first hard error.
@@ -60,7 +135,11 @@ func EnsureUserdataDir(path string) error { return EnsureDirOwned(path, SharedCo
// EnsureUserdataSkeleton creates the full userdata tree under a namespace root with the convention. // EnsureUserdataSkeleton creates the full userdata tree under a namespace root with the convention.
// It creates ALL dirs even if one errors (so a single chown/chmod hiccup doesn't truncate the tree), // It creates ALL dirs even if one errors (so a single chown/chmod hiccup doesn't truncate the tree),
// returning the first error seen for the caller to log. // returning the first error seen for the caller to log.
func EnsureUserdataSkeleton(nsRoot string) error { //
// dirs is the merged, sorted set from BuildUserdataSkeleton. This function only ever CREATES: there
// is no removal path here or anywhere in R-75, so a directory the current catalog no longer implies
// simply stays where it is (Scenario D).
func EnsureUserdataSkeleton(nsRoot string, dirs []string) error {
base := UserdataDir(nsRoot) base := UserdataDir(nsRoot)
var firstErr error var firstErr error
rec := func(e error) { rec := func(e error) {
@@ -69,7 +148,7 @@ func EnsureUserdataSkeleton(nsRoot string) error {
} }
} }
rec(EnsureUserdataDir(base)) rec(EnsureUserdataDir(base))
for _, sub := range UserdataSkeleton() { for _, sub := range dirs {
rec(EnsureUserdataDir(filepath.Join(base, sub))) rec(EnsureUserdataDir(filepath.Join(base, sub)))
} }
return firstErr return firstErr
+16 -5
View File
@@ -13,21 +13,31 @@ func TestSharedContentGID(t *testing.T) {
} }
} }
// TestUserdataSkeleton_List asserts the locked skeleton subdir set. // TestUserdataSkeleton_List asserts the locked carry-list. R-75 renamed the hardcoded list to
// UserdataSkeletonCarry (it is now the non-derived carry-list) and DELIBERATELY dropped the three
// `import*` entries: carrying them would re-create a per-drive drop-zone on every drive forever, the
// dead lookalike the canonical root exists to remove. That is not a removal — nothing deletes the
// dirs an existing box has; they stop being maintained and stop appearing on fresh boxes. Every other
// entry is unchanged, which is the zero-removals promise.
func TestUserdataSkeleton_List(t *testing.T) { func TestUserdataSkeleton_List(t *testing.T) {
got := map[string]bool{} got := map[string]bool{}
for _, s := range UserdataSkeleton() { for _, s := range UserdataSkeletonCarry() {
got[s] = true got[s] = true
} }
for _, want := range []string{ for _, want := range []string{
"media/movies", "media/tv", "media/music", "media/audiobooks", "media/books", "media/movies", "media/tv", "media/music", "media/audiobooks", "media/books",
"media/comics", "media/photos", "downloads", "import/paperless", "import/calibre", "media/comics", "media/photos", "downloads",
"roms", "documents", "roms", "documents",
} { } {
if !got[want] { if !got[want] {
t.Errorf("skeleton missing %q", want) t.Errorf("skeleton missing %q", want)
} }
} }
for _, gone := range []string{"import", "import/paperless", "import/calibre"} {
if got[gone] {
t.Errorf("carry-list must NOT hold %q — the drop-zone is canonical on the system drive (R-75)", gone)
}
}
} }
// TestUserdataDir confirms the userdata root is a sibling under the namespace. // TestUserdataDir confirms the userdata root is a sibling under the namespace.
@@ -41,9 +51,10 @@ func TestUserdataDir(t *testing.T) {
// is ignored — dirs + setgid still land). Runs cross-platform. // is ignored — dirs + setgid still land). Runs cross-platform.
func TestEnsureUserdataSkeleton_Structure(t *testing.T) { func TestEnsureUserdataSkeleton_Structure(t *testing.T) {
ns := t.TempDir() ns := t.TempDir()
_ = EnsureUserdataSkeleton(ns) // ignore chown error on a non-root CI host dirs := BuildUserdataSkeleton(nil) // no catalog derived → the carry-list floor
_ = EnsureUserdataSkeleton(ns, dirs) // ignore chown error on a non-root CI host
base := UserdataDir(ns) base := UserdataDir(ns)
for _, sub := range append([]string{""}, UserdataSkeleton()...) { for _, sub := range append([]string{""}, dirs...) {
p := filepath.Join(base, sub) p := filepath.Join(base, sub)
if fi, err := os.Stat(p); err != nil || !fi.IsDir() { if fi, err := os.Stat(p); err != nil || !fi.IsDir() {
t.Errorf("skeleton dir missing: %s (%v)", p, err) t.Errorf("skeleton dir missing: %s (%v)", p, err)
@@ -23,6 +23,11 @@ type hddProvider struct {
func (p *hddProvider) GetStackNeedsHDD(string) bool { return true } func (p *hddProvider) GetStackNeedsHDD(string) bool { return true }
func (p *hddProvider) GetStackHDDMounts(string) []string { return p.mounts } func (p *hddProvider) GetStackHDDMounts(string) []string { return p.mounts }
func (p *hddProvider) GetStackHDDPath(string) string { return p.hddPath } func (p *hddProvider) GetStackHDDPath(string) string { return p.hddPath }
func (p *hddProvider) GetImportRoot() string { return "" } // R-75: no import binds in this fixture
// R-203: these fixtures use ENROLLED drive paths, where the namespace root IS the drive path.
// Delegating keeps that identity explicit rather than hardcoding it.
func (p *hddProvider) GetStackNamespaceRoot(name string) string { return p.GetStackHDDPath(name) }
func (p *hddProvider) GetStackClassifiedBinds(string) ([]appbackup.ClassifiedBind, bool) { func (p *hddProvider) GetStackClassifiedBinds(string) ([]appbackup.ClassifiedBind, bool) {
return p.binds, p.hasBinds return p.binds, p.hasBinds
} }
+44 -1
View File
@@ -96,10 +96,26 @@ type Exporter struct {
// computation walks. Nil → the real os.ReadDir-based lister. // computation walks. Nil → the real os.ReadDir-based lister.
dirLister func(dir string) []string dirLister func(dir string) []string
// stopGuard (R-166) marks the stop→export→start window so a controller killed inside it leaves a
// durable record that the app is owed a restart. Declared consumer-side as a two-method interface
// so this package does not import internal/backup; main.go passes the backup manager's guard, so
// BOTH packages write ONE marker file — an exporter with its own file would be a second writer
// racing the same recovery. Nil = not wired (tests): the export runs exactly as it did before.
stopGuard appStopGuard
mu sync.Mutex mu sync.Mutex
activeJob *Job activeJob *Job
} }
// appStopGuard is the app-stop crash-marker seam. The REASON is deliberately not a parameter: it is
// always "app export" from here, and the adapter in main.go supplies it. Passing it as a string
// would duplicate backup.ReasonAppExport's value in a second package with nothing keeping the two in
// step — a drift this codebase has paid for before (the offbox key that was guessed, R-7b).
type appStopGuard interface {
Begin(opID string, stacks []string) error
End()
}
// NewExporter creates a new export/import engine. // NewExporter creates a new export/import engine.
func NewExporter(provider ExportStackProvider, logger *log.Logger, version string) *Exporter { func NewExporter(provider ExportStackProvider, logger *log.Logger, version string) *Exporter {
return &Exporter{ return &Exporter{
@@ -109,6 +125,18 @@ func NewExporter(provider ExportStackProvider, logger *log.Logger, version strin
} }
} }
// SetStopGuard wires the app-stop crash marker. INIT-ONLY — call once at startup, before any export.
func (e *Exporter) SetStopGuard(g appStopGuard) { e.stopGuard = g }
// stopGuardBegin records the app-stop marker before an export stops an app. An unwired guard is a
// no-op (pre-v0.189.0 behaviour), never an error — a test exporter must not be forced to have one.
func (e *Exporter) stopGuardBegin(stackName string) error {
if e.stopGuard == nil {
return nil
}
return e.stopGuard.Begin("app-export:"+stackName, []string{stackName})
}
// SetDebug enables or disables verbose debug logging. // SetDebug enables or disables verbose debug logging.
func (e *Exporter) SetDebug(debug bool) { func (e *Exporter) SetDebug(debug bool) {
e.debug = debug e.debug = debug
@@ -226,6 +254,14 @@ func (e *Exporter) executeExport(req ExportRequest, job *Job) {
// Optionally stop the app // Optionally stop the app
wasRunning := false wasRunning := false
if req.StopApp && e.provider.IsStackRunning(req.StackName) { if req.StopApp && e.provider.IsStackRunning(req.StackName) {
// R-166: mark BEFORE the stop. The defer below covers the graceful exits; it does NOT cover a
// SIGKILL or a power cut, which run no deferred function (Campaign 8 fault 10, on live
// hardware) — only this marker does, and a big export is a long window to be killed in.
if err := e.stopGuardBegin(req.StackName); err != nil {
e.failJob(job, step, "Az alkalmazás leállítása előtti jelölő nem menthető — az exportálás nem indult el.")
e.logger.Printf("[ERROR] Export: could not record the app-stop marker for %s (refusing to stop it unprotected): %v", req.StackName, err)
return
}
wasRunning = true wasRunning = true
e.logger.Printf("[INFO] Export: stopping %s", req.StackName) e.logger.Printf("[INFO] Export: stopping %s", req.StackName)
e.debugf("stopping stack %s before export", req.StackName) e.debugf("stopping stack %s before export", req.StackName)
@@ -246,6 +282,11 @@ func (e *Exporter) executeExport(req ExportRequest, job *Job) {
e.logger.Printf("[WARN] Export: could not restart %s: %v", req.StackName, err) e.logger.Printf("[WARN] Export: could not restart %s: %v", req.StackName, err)
} else { } else {
e.debugf("stack %s restarted successfully", req.StackName) e.debugf("stack %s restarted successfully", req.StackName)
// Cleared only on a restart that succeeded — a failed one keeps the marker so the
// next startup retries.
if e.stopGuard != nil {
e.stopGuard.End()
}
} }
}() }()
} }
@@ -618,7 +659,9 @@ func (e *Exporter) exportHDDData(req ExportRequest, dataDir string, manifest *Ma
// Task 4: the class-scoped plan. Legacy / no-block apps get an EMPTY plan (all mounts kept, root // Task 4: the class-scoped plan. Legacy / no-block apps get an EMPTY plan (all mounts kept, root
// tar with zero excludes) → byte-identical v0.130.0 capture. // tar with zero excludes) → byte-identical v0.130.0 capture.
plan := e.computeFabPlan(req, mounts) plan := e.computeFabPlan(req, mounts)
ud := appbackup.UserdataDir(filepath.Clean(e.provider.GetStackHDDPath(stackName))) // R-203: a NAMESPACE ROOT, not the drive path (identical on an enrolled drive; one segment short
// on the system-data fallback).
ud := appbackup.UserdataDir(filepath.Clean(e.provider.GetStackNamespaceRoot(stackName)))
claimed := make(map[string]string) // subdir → mount that claimed it claimed := make(map[string]string) // subdir → mount that claimed it
for _, mount := range mounts { for _, mount := range mounts {
+12 -5
View File
@@ -34,8 +34,12 @@ func (e *Exporter) computeFabPlan(req ExportRequest, mounts []string) fabPlan {
if !has { if !has {
return fabPlan{} // legacy: byte-identical v0.130.0 capture return fabPlan{} // legacy: byte-identical v0.130.0 capture
} }
hddPath := filepath.Clean(e.provider.GetStackHDDPath(req.StackName)) // R-203: the shared resolver's root parameter is a NAMESPACE ROOT — that is what the off-site
fb := appbackup.ComputeFabBuckets(binds, has, hddPath) // side has always passed (ComputeCaptureSet ← offbox_capture.go). This site passed the bare drive
// path, so on the system-data fallback the export's classified paths and the backup's capture set
// described DIFFERENT directories for the same declared bind. They now agree by construction.
nsRoot := filepath.Clean(e.provider.GetStackNamespaceRoot(req.StackName))
fb := appbackup.ComputeFabBuckets(binds, has, nsRoot, e.provider.GetImportRoot())
deselect := sliceSet(req.DeselectOptional) deselect := sliceSet(req.DeselectOptional)
optIn := sliceSet(req.OptInExcluded) optIn := sliceSet(req.OptInExcluded)
@@ -82,7 +86,10 @@ func (e *Exporter) computeFabPlan(req ExportRequest, mounts []string) fabPlan {
} }
plan := fabPlan{SkipMounts: map[string]bool{}} plan := fabPlan{SkipMounts: map[string]bool{}}
ud := appbackup.UserdataDir(hddPath) // R-203: UserdataDir takes a NAMESPACE ROOT, not the drive path. Identical on an enrolled drive;
// one segment short on the system-data fallback, which is where the export plan then skipped (or
// failed to skip) the wrong directory.
ud := appbackup.UserdataDir(nsRoot)
for _, m := range mounts { for _, m := range mounts {
mc := filepath.Clean(m) mc := filepath.Clean(m)
if mc == filepath.Clean(ud) { if mc == filepath.Clean(ud) {
@@ -116,8 +123,8 @@ func (e *Exporter) fabEstimateSplit(stackName string, est *ExportEstimate, volum
if !has { if !has {
return return
} }
hddPath := filepath.Clean(e.provider.GetStackHDDPath(stackName)) nsRoot := filepath.Clean(e.provider.GetStackNamespaceRoot(stackName)) // R-203, as above
fb := appbackup.ComputeFabBuckets(binds, has, hddPath) fb := appbackup.ComputeFabBuckets(binds, has, nsRoot, e.provider.GetImportRoot())
est.HasClassification = true est.HasClassification = true
toItems := func(cps []appbackup.CapturePath) ([]FabItem, int64) { toItems := func(cps []appbackup.CapturePath) ([]FabItem, int64) {
@@ -23,6 +23,11 @@ type fabProv struct {
} }
func (p *fabProv) GetStackHDDPath(string) string { return p.hddPath } func (p *fabProv) GetStackHDDPath(string) string { return p.hddPath }
func (p *fabProv) GetImportRoot() string { return "" } // R-75: no import binds in this fixture
// R-203: these fixtures use ENROLLED drive paths, where the namespace root IS the drive path.
// Delegating keeps that identity explicit rather than hardcoding it.
func (p *fabProv) GetStackNamespaceRoot(name string) string { return p.GetStackHDDPath(name) }
func (p *fabProv) GetStackHDDMounts(string) []string { return p.mounts } func (p *fabProv) GetStackHDDMounts(string) []string { return p.mounts }
func (p *fabProv) GetStackClassifiedBinds(string) ([]appbackup.ClassifiedBind, bool) { func (p *fabProv) GetStackClassifiedBinds(string) ([]appbackup.ClassifiedBind, bool) {
return p.binds, p.has return p.binds, p.has
@@ -17,6 +17,14 @@ type ExportStackProvider interface {
GetStackHDDMounts(name string) []string GetStackHDDMounts(name string) []string
// GetStackHDDPath returns the raw HDD_PATH env var from app.yaml. // GetStackHDDPath returns the raw HDD_PATH env var from app.yaml.
GetStackHDDPath(name string) string GetStackHDDPath(name string) string
// GetImportRoot returns the CANONICAL drop-zone root (R-75), on the SYSTEM drive. ${IMPORT_PATH}
// binds resolve against THIS, never against GetStackHDDPath. Empty when unresolvable.
GetImportRoot() string
// GetStackNamespaceRoot returns the app's felhom-data NAMESPACE ROOT — the directory that directly
// contains backups/ and userdata/. It is NOT GetStackHDDPath: on an enrolled drive the two are the
// same, and on the system-data fallback the namespace root has one more segment (R-203). Every
// appbackup path helper takes THIS, never the drive path. Empty when the app has no HDD_PATH.
GetStackNamespaceRoot(name string) string
// GetStackClassifiedBinds returns the app's backup-classified compose binds + whether it carries a // GetStackClassifiedBinds returns the app's backup-classified compose binds + whether it carries a
// (valid) backup block (Task 2). Drives the `.fab` class-scoped export plan (Task 4); a legacy app // (valid) backup block (Task 2). Drives the `.fab` class-scoped export plan (Task 4); a legacy app
// (false) exports the v0.130.0 full-root capture unchanged. // (false) exports the v0.130.0 full-root capture unchanged.
@@ -37,6 +37,11 @@ func (p *rtProvider) GetStackComposePath(string) (string, bool) {
} }
func (p *rtProvider) GetStackHDDMounts(string) []string { return nil } func (p *rtProvider) GetStackHDDMounts(string) []string { return nil }
func (p *rtProvider) GetStackHDDPath(string) string { return "" } func (p *rtProvider) GetStackHDDPath(string) string { return "" }
func (p *rtProvider) GetImportRoot() string { return "" } // R-75: no import binds in this fixture
// R-203: these fixtures use ENROLLED drive paths, where the namespace root IS the drive path.
// Delegating keeps that identity explicit rather than hardcoding it.
func (p *rtProvider) GetStackNamespaceRoot(name string) string { return p.GetStackHDDPath(name) }
func (p *rtProvider) GetStackClassifiedBinds(string) ([]appbackup.ClassifiedBind, bool) { func (p *rtProvider) GetStackClassifiedBinds(string) ([]appbackup.ClassifiedBind, bool) {
return nil, false return nil, false
} }
+252
View File
@@ -0,0 +1,252 @@
package backup
import (
"fmt"
"os"
"path/filepath"
"strings"
)
// ── Backup admission (R-181) ─────────────────────────────────────────────────────────────────────
//
// WHAT WAS WRONG. B2's capture floor (v0.192.0, R-165) shipped as the deliberate replacement for the
// bulkhead the `mp1` partition used to give, and it was consulted in exactly ONE place —
// `captureAllRecoveryUnits`, which writes a manifest and three compose files: a few KB. The two legs
// that write the BULK into the same `backups/primary/<app>` tree — the database dump and the volume
// dump — ran FIRST and unguarded. Measured on demo-hp 2026-08-03 06:40:03: opengist's volume dump
// wrote 2.0 GB with no check, free fell to 1.0 GB, and the floor then refused the cheap write it had
// already lost the argument to. Its refusal message said *"the previous unit is untouched"*, which
// was false by then — that app's tar had gone 182,272 B → 2,147,666,432 B under a stale manifest.
//
// WHAT THIS IS. ONE verdict per app per run, taken before that app's FIRST write of the run, covering
// all three legs. The three write under one per-app root (`appbackup.RecoveryUnitPath`), which is
// exactly why one verdict can honestly cover them — and why the message may now claim what it claims.
//
// WHY IT IS DECIDED LAZILY AND NOT ONCE AT THE START OF THE RUN. Space changes during a run: app A's
// 2 GB dump can put app B under the reserve. A verdict taken at run start would wave B through on a
// reading that was true before the disk filled — the same class of mistake as the one being fixed,
// moved one level up.
//
// WHY IT IS REMEMBERED AND NOT RE-DECIDED PER LEG. Re-deciding between an app's own legs reintroduces
// the split this closes: the DB leg admitted, the volume leg admitted, the capture refused — with the
// bulk already written. Decide once, remember, reuse; reset per run, because a set carried between
// runs is a wrong answer with a confident face.
//
// IT REFUSES; IT NEVER DELETES. Unchanged from B2 and load-bearing: nothing on this filesystem is
// generational (one unit per app at one fixed path, refreshed in place), so "prune the oldest" could
// only mean destroying a DIFFERENT app's only local copy. `pruneStalePrimaryDirs` removes ORPHANED
// dirs an app left on a drive it moved off — it has no notion of age or of the current app — and must
// never be repurposed for headroom.
// floorReason records WHICH term bound, so the operator can tell "the disk is full" from "this app's
// backup is too big for what is left". An alert that says only "refused" sends them to read code.
type floorReason int
const (
floorAdmit floorReason = iota // admitted — no term binds
floorHeadroom // the filesystem is ALREADY at/below the reserve
floorSize // there is room now, but this app's own write would cross the reserve
)
func (r floorReason) String() string {
switch r {
case floorHeadroom:
return "headroom"
case floorSize:
return "size"
default:
return "admitted"
}
}
// admissionVerdict is one app's decision for one run. It carries everything the alert needs, so the
// alert is rendered once from the same value every leg consults.
type admissionVerdict struct {
admitted bool
reason floorReason
usage *UnitSpace
estGiB float64 // the estimated write in GiB — the arithmetic unit, matching the reserve's terms
estBytes int64 // the same estimate in bytes — the RENDERING unit; see floorRefusal
hasEst bool // whether an estimate was available at all (§8.2: distinct from "estimated 0")
err error // the refusal, nil when admitted
}
// admissionSet is the per-RUN memo. Deliberately not a field with a lifetime of its own: it is
// created by beginAdmissionRun and cleared by the returned func, so an absent set means "no run is in
// flight" rather than "a stale answer from last night".
type admissionSet struct {
v map[string]admissionVerdict
}
// beginAdmissionRun opens the per-run admission scope and returns the closer. Called once at the top
// of runDBDumpsInternal — which is the single orchestrator of all three legs — so the DB dump, the
// volume dump and the capture of one app all consult the SAME verdict.
//
// A second call while a set is live REPLACES it and the returned closer restores the previous one, so
// nesting cannot silently drop a caller's scope.
func (m *Manager) beginAdmissionRun() func() {
m.admissionMu.Lock()
prev := m.admission
m.admission = &admissionSet{v: map[string]admissionVerdict{}}
m.admissionMu.Unlock()
return func() {
m.admissionMu.Lock()
m.admission = prev
m.admissionMu.Unlock()
}
}
// admitApp is THE gate. It returns true when this app may write, false when the reserve refuses it.
//
// On the first refusal for an app it logs and fires EXACTLY ONE operator alert; every later leg in
// the same run reads the memo and stays silent, so a refused app produces one email and not three.
//
// With no run scope open (the periodic status refresh calls captureAllRecoveryUnits directly) it
// decides fresh. That is not a gap: each app appears once in that sweep, so "once per app" still
// holds — there is simply nothing to remember it across.
func (m *Manager) admitApp(stackName string) bool {
m.admissionMu.Lock()
defer m.admissionMu.Unlock()
if set := m.admission; set != nil {
if v, ok := set.v[stackName]; ok {
return v.admitted // already decided this run — do NOT re-decide, do NOT re-alert
}
}
v := m.decideAdmission(stackName)
if set := m.admission; set != nil {
set.v[stackName] = v
}
if v.admitted {
return true
}
// The claim below is now literally true, and that is the whole point of R-181: the verdict is
// taken before the FIRST of the three writes, so at this moment nothing under
// backups/primary/<app> has been touched by this run. TestAdmission_RefusedAppsTreeIsByteIdentical
// pins the consequence by checksumming the tree, not by reading this line.
m.logger.Printf("[WARN] [backup] App backup REFUSED for %s (%s) — %v; NO database dump, NO volume "+
"dump and NO recovery-unit capture was written for it, the previous unit is untouched and "+
"NOTHING was deleted", stackName, v.reason, v.err)
if m.unitNotify != nil {
m.unitNotify(stackName, v.err, v.usage)
}
// R-182: the digest entry is recorded HERE, where the verdict is taken — once per app per run.
// Not at the three call sites that consult the memo: R-181's whole contract is that ONE verdict
// covers all three legs, so noting it per leg listed a single refused app three times and
// produced counts like "2 of 1 apps failed". The leg name says what actually happened, which is
// that nothing was attempted at all.
m.noteFailure(stackName, "whole app (refused before any write)", v.err.Error())
return false
}
// decideAdmission applies the floor to a fresh reading plus this app's estimated write.
func (m *Manager) decideAdmission(stackName string) admissionVerdict {
estBytes, hasEst := m.estimatedWriteBytes(stackName)
estGiB := float64(estBytes) / (1024 * 1024 * 1024)
usage, reason := m.floorVerdict(m.readUnitSpace(stackName), estGiB)
v := admissionVerdict{
admitted: reason == floorAdmit,
reason: reason,
usage: usage,
estGiB: estGiB,
estBytes: estBytes,
hasEst: hasEst,
}
if !v.admitted {
v.err = floorRefusal(reason, usage, estBytes, hasEst)
}
return v
}
// floorRefusal renders the refusal an operator reads. It names the reserve (not an I/O error — this
// is a deliberate hold, not broken machinery), says WHICH term bound, and states plainly when the
// decision was headroom-only because the app has no previous backup to estimate from (§8.2).
//
// THE ESTIMATE IS RENDERED IN BYTES-HUMANIZED, NOT GiB, and that is not cosmetic. Fixed to two
// decimal GiB, every app under ~10 MB prints `0.00 GiB` — which reads as "no estimate was available"
// and is the opposite of what happened. Observed on the live proof run: opengist's real 178 KB
// estimate rendered as `estimated 0.00 GiB write`. The arithmetic stays in GiB (the reserve's own
// unit); only the rendering changes.
func floorRefusal(reason floorReason, usage *UnitSpace, estBytes int64, hasEst bool) error {
var b strings.Builder
fmt.Fprintf(&b, "%%w (reserve: %.0f%%%% used or %.1f GiB free", FloorUsedPercent, FloorFreeGiB)
switch {
case reason == floorSize:
fmt.Fprintf(&b, "; this app's last backup was %s and writing it again would cross the reserve", humanizeBytes(estBytes))
case hasEst:
fmt.Fprintf(&b, "; the filesystem is already below it, before this app's estimated %s write", humanizeBytes(estBytes))
default:
b.WriteString("; this app has no previous backup on disk, so only current headroom was considered")
}
b.WriteString(") — %s")
return fmt.Errorf(b.String(), ErrCaptureFloor, usage)
}
// estimatedWriteBytes estimates what this app's three legs are about to write, from what the PREVIOUS
// run left in its unit: the `.sql` dumps and the `.tar` volume archives already on disk for this app.
//
// WHY THIS ESTIMATOR. It is free — two ReadDirs of a directory the caller is about to write into — and
// the next write is usually close to the last one. The alternative, a container-based `du` of every
// named volume, was measured on the demo box before being rejected; the figure is in REPORT.md §6.
//
// NO HISTORY → (0, false), and the caller falls back to headroom-only. Refusing an app because it has
// never been backed up would make the first backup the one that can never happen (Scenario E).
//
// It reads the app's CURRENT unit root, so an app that moved drives estimates from its new (probably
// empty) location and is treated as history-less — conservative in the admitting direction, which is
// the right way round for an estimate that only ever tightens a threshold.
func (m *Manager) estimatedWriteBytes(stackName string) (int64, bool) {
drivePath := m.GetAppDrivePath(stackName)
if drivePath == "" {
return 0, false
}
nsRoot := m.namespaceRoot(drivePath)
var total int64
var found bool
for _, d := range []struct {
dir string
ext string
}{
{AppDBDumpPath(nsRoot, stackName), ".sql"},
{AppVolumeDumpPath(nsRoot, stackName), ".tar"},
} {
n, ok := sumFileSizes(d.dir, d.ext)
total += n
found = found || ok
}
if !found {
return 0, false
}
return total, true
}
// sumFileSizes totals the sizes of files with the given suffix in dir. The bool reports whether ANY
// such file was seen — distinct from a zero total, because a 0-byte dump is history (a real, if
// alarming, previous result) while an absent directory is not.
//
// A stat error on one entry is skipped rather than aborting the sum: an estimate built from the
// readable files is worth more than no estimate, and the entry that could not be read is logged
// nowhere because this is a hint, not a measurement — it can only tighten a threshold, never relax
// one below what the headroom term already enforces.
func sumFileSizes(dir, suffix string) (int64, bool) {
entries, err := os.ReadDir(dir)
if err != nil {
return 0, false
}
var total int64
var found bool
for _, e := range entries {
if e.IsDir() || !strings.HasSuffix(e.Name(), suffix) {
continue
}
fi, err := os.Stat(filepath.Join(dir, e.Name()))
if err != nil {
continue
}
found = true
total += fi.Size()
}
return total, found
}
@@ -0,0 +1,741 @@
package backup
import (
"bytes"
"crypto/sha256"
"encoding/hex"
"fmt"
"go/ast"
"go/parser"
"go/token"
"io"
"log"
"os"
"path/filepath"
"sort"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-181 — the reserve guards the write that fills the disk, and its promise is true.
//
// WHAT THESE ASSERT, AND WHY IT IS THE TREE AND NOT THE LOG. The defect being closed is precisely a
// log line that claimed something the filesystem contradicted: B2 printed *"the previous unit is
// untouched"* while the volume leg had already rewritten that unit's tar 182,272 B → 2,147,666,432 B.
// So a test that reads the message and believes it would have passed against the broken code. Every
// refusal test here checksums the whole `backups/primary` tree before and after and compares.
// ── Harness ──────────────────────────────────────────────────────────────────────────────────────
// admissionProvider records the two acts a refused app must never suffer: its recovery info being
// read (a capture that was ATTEMPTED) and its stack being stopped (which DumpAppVolumesSafe does as
// its first act, before any check of its own).
type admissionProvider struct {
stacks []string
volumes map[string][]string
hdd map[string]string // per-app drive path, for the drive-state skip tests
dir string
infoHits []string
stopped []string
}
func (p *admissionProvider) GetStackComposePath(string) (string, bool) { return "", false }
func (p *admissionProvider) ListDeployedStacks() []StackSummary {
out := make([]StackSummary, 0, len(p.stacks))
for _, s := range p.stacks {
out = append(out, StackSummary{Name: s})
}
return out
}
func (p *admissionProvider) GetStackHDDMounts(string) []string { return nil }
func (p *admissionProvider) GetStackHDDPath(n string) string { return p.hdd[n] }
func (p *admissionProvider) GetImportRoot() string { return "" }
func (p *admissionProvider) GetDockerVolumes(name string) []string {
if p.volumes == nil {
return []string{name + "_data"} // every app is volume-bearing unless told otherwise
}
return p.volumes[name]
}
func (p *admissionProvider) StopStack(name string) error {
p.stopped = append(p.stopped, name)
return nil
}
func (p *admissionProvider) StartStack(string) error { return nil }
func (p *admissionProvider) RefreshAndIsRunning(string) bool { return true }
func (p *admissionProvider) GetStackRecoveryInfo(name string) (RecoveryInfo, bool) {
p.infoHits = append(p.infoHits, name)
return RecoveryInfo{StackDir: filepath.Join(p.dir, "stacks", name)}, true
}
func (p *admissionProvider) RecoverStackSecrets(string, []string) map[string]string { return nil }
func (p *admissionProvider) RecreateStackDefinitionFromUnit(string, string, map[string]string) error {
return nil
}
func (p *admissionProvider) StartStackServices(string, []string) error { return nil }
func (p *admissionProvider) GetStackClassifiedBinds(string) ([]ClassifiedBind, bool) {
return nil, false
}
type admissionHarness struct {
m *Manager
prov *admissionProvider
events []unitEvent
usage map[string]*UnitSpace
dir string
logs *bytes.Buffer
volDumped []string
}
func newAdmissionHarness(t *testing.T, stacks ...string) *admissionHarness {
t.Helper()
dir := t.TempDir()
h := &admissionHarness{
prov: &admissionProvider{stacks: stacks, dir: dir, hdd: map[string]string{}},
usage: map[string]*UnitSpace{},
dir: dir,
logs: &bytes.Buffer{},
}
h.m = &Manager{
logger: log.New(h.logs, "", 0),
systemDataPath: dir,
stackProvider: h.prov,
unitSpaceFn: func(name string) *UnitSpace { return h.usage[name] },
}
// The volume-dump seam records the leg that writes the BULK — the one B2 never gated. A refused
// app must not reach it.
h.m.dumpVolumesSafe = func(name string) error {
h.volDumped = append(h.volDumped, name)
// Write what the real leg writes, so an ungated call is visible in the tree checksum too.
dumpDir := AppVolumeDumpPath(h.nsRoot(), name)
if err := os.MkdirAll(dumpDir, 0o755); err != nil {
return err
}
return os.WriteFile(filepath.Join(dumpDir, name+"_data.tar"), []byte("FRESH TAR FROM THIS RUN"), 0o644)
}
h.m.SetUnitNotify(func(name string, err error, u *UnitSpace) {
h.events = append(h.events, unitEvent{app: name, err: err.Error(), usage: u})
})
return h
}
// markDisconnected / markDecommissioned put a real settings row behind the drive-state skips, so
// Scenario F exercises the production guards rather than a stub of them.
func (h *admissionHarness) markDisconnected(app string) {
h.driveState(app, true, false)
}
func (h *admissionHarness) markDecommissioned(app string) {
h.driveState(app, false, true)
}
func (h *admissionHarness) driveState(app string, disconnected, decommissioned bool) {
if h.m.settings == nil {
sett, err := settings.Load(filepath.Join(h.dir, "settings.json"), log.New(io.Discard, "", 0))
if err != nil {
panic(err)
}
h.m.settings = sett
}
// Each such app gets its OWN drive path, or marking one would skip them all.
p := filepath.Join(h.dir, "drives", app)
if err := os.MkdirAll(p, 0o755); err != nil {
panic(err)
}
h.prov.hdd[app] = p
if err := h.m.settings.AddStoragePath(settings.StoragePath{Path: p, Label: app}); err != nil {
panic(err)
}
if disconnected {
if err := h.m.settings.SetDisconnected(p, true, nil); err != nil {
panic(err)
}
}
if decommissioned {
if err := h.m.settings.SetDecommissioned(p, ""); err != nil {
panic(err)
}
}
}
func (h *admissionHarness) nsRoot() string { return filepath.Join(h.dir, "felhom-data") }
// setSpace states the filesystem's occupancy as a test INPUT — the whole point of the unitSpaceFn
// seam, so no test has to manufacture disk pressure on a real disk.
func (h *admissionHarness) setSpace(app string, usedPct, availGB, totalGB float64) {
h.usage[app] = &UnitSpace{
Path: h.dir, UsedPercent: usedPct, AvailGB: availGB,
TotalGB: totalGB, UsedGB: totalGB * usedPct / 100,
}
}
// seedUnit writes a previous recovery unit for an app: a manifest, a captured app.yaml, a DB dump and
// a volume tar of the given size. The tar is SPARSE (Truncate), so a 2 GiB "previous backup" costs no
// disk — the estimator reads st_size, which is what the next write will actually cost.
func (h *admissionHarness) seedUnit(t *testing.T, app string, tarBytes int64) {
t.Helper()
ns := h.nsRoot()
for _, d := range []string{
RecoveryUnitComposePath(ns, app),
AppDBDumpPath(ns, app),
AppVolumeDumpPath(ns, app),
} {
if err := os.MkdirAll(d, 0o755); err != nil {
t.Fatal(err)
}
}
write := func(p string, b []byte, mode os.FileMode) {
if err := os.WriteFile(p, b, mode); err != nil {
t.Fatal(err)
}
}
write(RecoveryUnitManifestPath(ns, app), []byte(`{"app_name":"`+app+`","created_at":"2026-08-02T00:00:00Z"}`), 0o644)
write(filepath.Join(RecoveryUnitComposePath(ns, app), "app.yaml"), []byte("deployed: true\nenv:\n A: previous-good-value\n"), 0o600)
write(filepath.Join(AppDBDumpPath(ns, app), app+"-postgres.sql"), []byte("-- previous good dump\n"), 0o644)
tar := filepath.Join(AppVolumeDumpPath(ns, app), app+"_data.tar")
f, err := os.Create(tar)
if err != nil {
t.Fatal(err)
}
if _, err := f.WriteString("PREVIOUS GOOD TAR"); err != nil {
t.Fatal(err)
}
if tarBytes > 0 {
if err := f.Truncate(tarBytes); err != nil { // sparse — st_size is the estimate, blocks are not spent
t.Fatal(err)
}
}
if err := f.Close(); err != nil {
t.Fatal(err)
}
}
// runOneBackupRun performs exactly the sequence runDBDumpsInternal performs for the two legs that can
// be driven without Docker: the admission scope is opened, the volume leg runs, then the capture leg.
// The DB leg's wiring is pinned structurally by TestAdmission_IsWiredIntoEveryProductionWriteLeg,
// because DiscoverDatabases shells out to `docker` and cannot honestly run here.
func (h *admissionHarness) runOneBackupRun() {
done := h.m.beginAdmissionRun()
defer done()
h.m.runVolumeDumps()
h.m.captureAllRecoveryUnits()
}
// ── The instrument: a checksum of the whole backup tree ──────────────────────────────────────────
// treeFingerprint walks every file under backups/primary and returns "relpath mode sha256" lines,
// sorted. It is the ONLY honest way to check the refusal's claim: it detects a rewritten payload, an
// added file and a deleted one alike, which a log line and an exit code both fail to do.
func treeFingerprint(t *testing.T, root string) string {
t.Helper()
var lines []string
err := filepath.Walk(root, func(p string, fi os.FileInfo, err error) error {
if err != nil {
if os.IsNotExist(err) {
return nil
}
return err
}
if fi.IsDir() {
return nil
}
f, err := os.Open(p)
if err != nil {
return err
}
defer f.Close()
sum := sha256.New()
if _, err := io.Copy(sum, f); err != nil {
return err
}
rel, _ := filepath.Rel(root, p)
lines = append(lines, fmt.Sprintf("%s %o %d %s", rel, fi.Mode().Perm(), fi.Size(), hex.EncodeToString(sum.Sum(nil))))
return nil
})
if err != nil {
t.Fatalf("fingerprinting %s: %v", root, err)
}
sort.Strings(lines)
return strings.Join(lines, "\n")
}
// treeStatFingerprint is the instrument for trees holding a multi-GiB fixture, where hashing every
// byte costs more than it proves: name + mode + SIZE. It still catches the act being tested — the
// volume leg replacing a 2 GiB tar with a freshly written one — because that changes the size, and it
// catches an added or deleted file by name. Content-identical-but-different-bytes is the one thing it
// cannot see, which is why the small-tree tests use treeFingerprint instead.
func treeStatFingerprint(t *testing.T, root string) string {
t.Helper()
var lines []string
_ = filepath.Walk(root, func(p string, fi os.FileInfo, err error) error {
if err != nil || fi.IsDir() {
return nil
}
rel, _ := filepath.Rel(root, p)
lines = append(lines, fmt.Sprintf("%s %o %d", rel, fi.Mode().Perm(), fi.Size()))
return nil
})
sort.Strings(lines)
return strings.Join(lines, "\n")
}
// treeFileList is the weaker instrument used for Scenario F: names only, so the assertion is
// specifically about DELETION and cannot be satisfied or broken by a content change.
func treeFileList(t *testing.T, root string) []string {
t.Helper()
var names []string
_ = filepath.Walk(root, func(p string, fi os.FileInfo, err error) error {
if err != nil || fi.IsDir() {
return nil
}
rel, _ := filepath.Rel(root, p)
names = append(names, rel)
return nil
})
sort.Strings(names)
return names
}
func (h *admissionHarness) primaryRoot() string {
return PrimaryBackupPath(h.nsRoot())
}
// ── Scenario A — one decision, taken before the first byte ───────────────────────────────────────
func TestAdmission_RefusedAppWritesNothingAndIsNotStopped(t *testing.T) {
h := newAdmissionHarness(t, "privatebin", "opengist", "homebox")
h.setSpace("privatebin", 40, 60, 100)
h.setSpace("opengist", 98, 0.4, 70) // below the reserve on BOTH terms
h.setSpace("homebox", 40, 60, 100)
h.seedUnit(t, "opengist", 0)
// Scoped to the REFUSED app's own unit: its two siblings are admitted and legitimately write
// theirs, so a whole-tree fingerprint would change for the right reason and prove nothing here.
// Scenario F below takes the whole-tree view, where every app is refused.
refusedUnit := RecoveryUnitPath(h.nsRoot(), "opengist")
before := treeFingerprint(t, refusedUnit)
if before == "" {
t.Fatal("the fixture seeded no previous unit, so 'byte-identical' would be vacuously true")
}
h.runOneBackupRun()
after := treeFingerprint(t, refusedUnit)
// 1. NOT ONE of the three legs ran for the refused app.
for _, got := range h.volDumped {
if got == "opengist" {
t.Fatal("the VOLUME leg ran for a refused app — this is the R-181 defect exactly: the leg " +
"that writes the bulk was never gated, so the reserve it protects was consumed by the " +
"very step it exists to bound")
}
}
for _, got := range h.prov.infoHits {
if got == "opengist" {
t.Fatal("the CAPTURE leg was attempted for a refused app — the verdict must be taken before " +
"any write is prepared, not partway through one")
}
}
// 2. The tree is byte-identical. This is the assertion the broken code could not pass.
if after != before {
t.Fatalf("the backup tree CHANGED across a refusal.\n--- before ---\n%s\n--- after ---\n%s\n"+
"A refusal that has already rewritten the payload is the defect, not the fix", before, after)
}
// 3. The app was never stopped. DumpAppVolumesSafe stops the stack as its FIRST act, so a gate
// placed inside it would bounce the app it is refusing to back up.
for _, got := range h.prov.stopped {
if got == "opengist" {
t.Fatal("the refused app was STOPPED — the reserve check has drifted behind the stop")
}
}
// 4. Exactly ONE alert, for that app, carrying the space figures. Three legs must not mean three
// emails about one disk.
if len(h.events) != 1 {
t.Fatalf("got %d alerts, want exactly 1 (one app refused, three legs): %+v", len(h.events), h.events)
}
if h.events[0].app != "opengist" {
t.Fatalf("alert names %q, want opengist", h.events[0].app)
}
if h.events[0].usage == nil || h.events[0].usage.AvailGB != 0.4 {
t.Fatalf("the alert carries no/incorrect space figures: %+v", h.events[0].usage)
}
}
// ── Scenario B — the other apps are unaffected ───────────────────────────────────────────────────
func TestAdmission_SiblingAppsProceedAndOnlyTheRefusedOneAlerts(t *testing.T) {
h := newAdmissionHarness(t, "privatebin", "opengist", "homebox")
h.setSpace("privatebin", 40, 60, 100)
h.setSpace("opengist", 99, 0.2, 70)
h.setSpace("homebox", 40, 60, 100)
h.runOneBackupRun()
for _, app := range []string{"privatebin", "homebox"} {
if !hasStr(h.volDumped, app) {
t.Errorf("%s was not volume-dumped (dumped=%v) — one app's refusal silenced its siblings", app, h.volDumped)
}
if !hasStr(h.prov.infoHits, app) {
t.Errorf("%s was not captured (attempted=%v) — the loop did not continue past the refusal", app, h.prov.infoHits)
}
if _, err := os.Stat(RecoveryUnitManifestPath(h.nsRoot(), app)); err != nil {
t.Errorf("%s has no manifest after the run: %v — an admitted app must be backed up normally", app, err)
}
}
if len(h.events) != 1 {
t.Fatalf("got %d alerts, want exactly 1: %+v", len(h.events), h.events)
}
}
// ── Scenario C — the promise is true ─────────────────────────────────────────────────────────────
// Every claim the shipped message makes is checked against the tree it describes. The wording is NOT
// weakened to fit the behaviour; the behaviour was moved so the wording became true (§8.3).
func TestAdmission_EveryClaimInTheRefusalMessageHoldsAgainstTheTree(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
h.setSpace("opengist", 98, 0.5, 70)
h.seedUnit(t, "opengist", 0)
beforeFP := treeFingerprint(t, h.primaryRoot())
beforeList := treeFileList(t, h.primaryRoot())
h.runOneBackupRun()
msg := h.logs.String()
if !strings.Contains(msg, "REFUSED for opengist") {
t.Fatalf("no refusal was logged for opengist; log was:\n%s", msg)
}
// Claim 1: "NO database dump, NO volume dump and NO recovery-unit capture was written for it".
for _, claim := range []string{"NO database dump", "NO volume dump", "NO recovery-unit capture"} {
if !strings.Contains(msg, claim) {
t.Fatalf("the message no longer claims %q — if a leg cannot be brought under the verdict the "+
"wording must be narrowed deliberately and the gap named, not dropped silently.\n%s", claim, msg)
}
}
if len(h.volDumped) != 0 || len(h.prov.infoHits) != 0 {
t.Fatalf("the message claims no leg ran, but volume=%v capture=%v", h.volDumped, h.prov.infoHits)
}
// Claim 2: "the previous unit is untouched" — the claim that was MEASURED FALSE in R-181.
if !strings.Contains(msg, "the previous unit is untouched") {
t.Fatalf("the message dropped the untouched claim: %s", msg)
}
if got := treeFingerprint(t, h.primaryRoot()); got != beforeFP {
t.Fatalf("the message says the previous unit is untouched; the tree says otherwise.\n"+
"--- before ---\n%s\n--- after ---\n%s", beforeFP, got)
}
// Claim 3: "NOTHING was deleted".
if !strings.Contains(msg, "NOTHING was deleted") {
t.Fatalf("the message dropped the no-deletion claim: %s", msg)
}
if got := treeFileList(t, h.primaryRoot()); !equalStrs(got, beforeList) {
t.Fatalf("files disappeared across a refusal: before=%v after=%v", beforeList, got)
}
// Claim 4: the reason is named, so the operator can tell which term bound.
if !strings.Contains(msg, "headroom") {
t.Fatalf("the message does not name WHICH term bound — an operator cannot tell 'the disk is "+
"full' from 'this app's backup is too big for what is left':\n%s", msg)
}
}
// ── Scenario D — size-aware, not just headroom-aware ─────────────────────────────────────────────
// The live R-181 sequence, reproduced as a unit: the filesystem is ABOVE the reserve on both terms
// when the run reaches the app, and the app's own write is what crosses it. Under B2 this app was
// admitted at 96% and then allowed to write 2 GB.
func TestAdmission_SizeTermRefusesAnAppWhoseOwnWriteWouldCrossTheReserve(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
// 96% used of 70 GiB, 3.0 GiB free — BOTH reserve terms deliberately still clear (97% / 1.0 GiB),
// exactly as on demo-hp at 06:40:03, so a headroom-only rule starts the run.
h.setSpace("opengist", 96, 3.0, 70)
if _, r := h.m.floorVerdict(h.usage["opengist"], 0); r != floorAdmit {
t.Fatalf("fixture is wrong: the headroom term already refuses (%v), so this test would pass "+
"without a size term and prove nothing", r)
}
h.seedUnit(t, "opengist", 2<<30) // its last backup was 2 GiB — the figure measured live
before := treeStatFingerprint(t, h.primaryRoot())
h.runOneBackupRun()
if len(h.events) != 1 {
t.Fatalf("got %d alerts, want 1 — the app was admitted at 96%% and would have been allowed to "+
"write 2 GiB, which is the R-181 sequence: %+v", len(h.events), h.events)
}
if !strings.Contains(h.logs.String(), "(size)") {
t.Fatalf("the refusal was not attributed to the SIZE term:\n%s", h.logs.String())
}
if !strings.Contains(h.events[0].err, "last backup was 2.0 GB") {
t.Fatalf("the alert does not carry the estimate that produced the refusal: %q", h.events[0].err)
}
if len(h.volDumped) != 0 {
t.Fatalf("the volume leg ran anyway: %v", h.volDumped)
}
if got := treeStatFingerprint(t, h.primaryRoot()); got != before {
t.Fatalf("the tree changed despite the size-term refusal.\nbefore=%s\nafter =%s", before, got)
}
}
// ── Scenario E — a first-ever backup is not blocked by having no history ─────────────────────────
func TestAdmission_FirstEverBackupIsAdmitted(t *testing.T) {
h := newAdmissionHarness(t, "brandnew")
h.setSpace("brandnew", 40, 600, 1000) // ample room, and NO previous unit on disk
if est, ok := h.m.estimatedWriteBytes("brandnew"); ok || est != 0 {
t.Fatalf("estimatedWriteBytes = (%v, %v) for an app with no history, want (0, false)", est, ok)
}
h.runOneBackupRun()
if len(h.events) != 0 {
t.Fatalf("a brand-new app was refused: %+v — refusing every app that has no size to estimate "+
"from would make the FIRST backup the one that can never happen", h.events)
}
if !hasStr(h.volDumped, "brandnew") || !hasStr(h.prov.infoHits, "brandnew") {
t.Fatalf("the app was not backed up (volume=%v capture=%v)", h.volDumped, h.prov.infoHits)
}
}
// ── Scenario F — the reserve still never deletes ─────────────────────────────────────────────────
func TestAdmission_NothingUnderBackupsIsEverRemoved(t *testing.T) {
h := newAdmissionHarness(t, "privatebin", "opengist", "homebox")
for _, app := range []string{"privatebin", "opengist", "homebox"} {
h.setSpace(app, 99, 0.1, 70) // every app refused — maximum pressure to "make room"
h.seedUnit(t, app, 0)
}
before := treeFileList(t, h.primaryRoot())
h.runOneBackupRun()
after := treeFileList(t, h.primaryRoot())
if !equalStrs(before, after) {
t.Fatalf("the file list changed under the reserve.\nbefore=%v\nafter =%v\n"+
"Nothing here is generational — a unit is ONE fixed path per app — so 'prune the oldest' "+
"could only mean destroying a DIFFERENT app's only local recovery unit", before, after)
}
if len(before) == 0 {
t.Fatal("the fixture seeded no files, so this test would pass against code that deleted everything")
}
}
// ── §8.1 — one verdict per app per run, and it resets between runs ───────────────────────────────
// The verdict must not be re-taken between an app's own legs. Re-deciding is how the split this fixes
// came about: DB leg admitted, volume leg admitted, capture refused — with the bulk already written.
func TestAdmission_VerdictIsTakenOncePerAppPerRunAndNotRedecidedBetweenLegs(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
reads := 0
h.m.unitSpaceFn = func(string) *UnitSpace {
reads++
if reads == 1 {
return &UnitSpace{Path: h.dir, UsedPercent: 99, AvailGB: 0.1, TotalGB: 70, UsedGB: 69.3}
}
// The disk "recovers" mid-run. A re-decided verdict would admit the capture leg here — which
// is precisely the split R-181 closes, arriving from the other direction.
return &UnitSpace{Path: h.dir, UsedPercent: 10, AvailGB: 60, TotalGB: 70, UsedGB: 7}
}
h.runOneBackupRun()
if reads != 1 {
t.Fatalf("the filesystem was read %d times for ONE app in ONE run — the verdict is being "+
"re-decided between legs, which reintroduces the split (bulk written, capture refused)", reads)
}
if len(h.prov.infoHits) != 0 {
t.Fatal("the capture leg ran after the app was refused earlier in the same run")
}
if len(h.events) != 1 {
t.Fatalf("got %d alerts, want exactly 1 per app per run: %+v", len(h.events), h.events)
}
}
// A set carried between runs is a wrong answer with a confident face: tonight's question answered
// with last night's disk.
func TestAdmission_TheRememberedSetResetsBetweenRuns(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
h.setSpace("opengist", 99, 0.1, 70)
h.runOneBackupRun()
if len(h.events) != 1 {
t.Fatalf("run 1: want 1 alert, got %+v", h.events)
}
h.setSpace("opengist", 20, 55, 70) // space freed between runs
h.runOneBackupRun()
if !hasStr(h.volDumped, "opengist") {
t.Fatal("the second run still refused the app — the previous run's verdict was carried over, " +
"so freeing space could never take effect")
}
if len(h.events) != 1 {
t.Fatalf("the second (admitted) run alerted again: %+v", h.events)
}
}
// ── §8.4 — a nil reading neither refuses nor warns, across ALL THREE legs ────────────────────────
// Unchanged behaviour, re-pinned because the decision now governs three legs instead of one: an
// unreadable filesystem must not silently stop an app being backed up at all.
func TestAdmission_UnreadableFilesystemAdmitsEveryLegAndDoesNotWarn(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
// No usage entry → the reader returns nil, which is what system.GetDiskUsage does on error.
h.runOneBackupRun()
if len(h.events) != 0 {
t.Fatalf("an unreadable filesystem produced %d alert(s): %+v — that is the drive gate's "+
"business and has its own alert", len(h.events), h.events)
}
if !hasStr(h.volDumped, "opengist") {
t.Fatal("the VOLUME leg was refused on an unreadable read — a drive that merely blipped would " +
"now stop the bulk of the backup, not just the capture")
}
if !hasStr(h.prov.infoHits, "opengist") {
t.Fatal("the CAPTURE leg was refused on an unreadable read")
}
}
// ── The estimator, through the production path (no seam) ─────────────────────────────────────────
func TestEstimatedWriteBytes_SumsTheAppsPreviousDumpsFromRealFiles(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
h.seedUnit(t, "opengist", 3<<30) // 3 GiB sparse tar + a small .sql
est, ok := h.m.estimatedWriteBytes("opengist")
if !ok {
t.Fatal("history on disk was not recognised as history")
}
if est < 3<<30 || est > (3<<30)+4096 {
t.Fatalf("estimate = %d B, want ~%d (the .tar plus the small .sql)", est, int64(3)<<30)
}
// An app whose unit exists but holds no dumps yet is history-LESS, not a zero-byte estimate.
other := AppVolumeDumpPath(h.nsRoot(), "empty")
if err := os.MkdirAll(other, 0o755); err != nil {
t.Fatal(err)
}
if est, ok := h.m.estimatedWriteBytes("empty"); ok || est != 0 {
t.Fatalf("an empty unit reported history (%v, %v) — an absent dump is not a 0-byte one", est, ok)
}
}
// ── The seam is WIRED — walked as an AST, not grepped ────────────────────────────────────────────
// FOUR mechanisms in this project have been built and left disconnected (REUSE.md's seam register).
// The behavioural tests above drive the two legs that can run without Docker; the DB leg cannot, so
// its gate is pinned HERE, structurally. `strings.Contains` is deliberately not used: a commented-out
// call still contains the string, and so does a call inside dead code.
func TestAdmission_IsWiredIntoEveryProductionWriteLeg(t *testing.T) {
fset := token.NewFileSet()
file, err := parser.ParseFile(fset, "backup.go", nil, 0) // comments dropped — only real calls survive
if err != nil {
t.Fatal(err)
}
calls := map[string][]string{} // enclosing func → called names, in source order
var current string
ast.Inspect(file, func(n ast.Node) bool {
switch v := n.(type) {
case *ast.FuncDecl:
current = v.Name.Name
case *ast.CallExpr:
name := ""
switch fn := v.Fun.(type) {
case *ast.Ident:
name = fn.Name
case *ast.SelectorExpr:
name = fn.Sel.Name
}
if name != "" && current != "" {
calls[current] = append(calls[current], name)
}
}
return true
})
// 1. The run scope is opened by the orchestrator of all three legs.
if !hasStr(calls["runDBDumpsInternal"], "beginAdmissionRun") {
t.Fatal("runDBDumpsInternal does not open the admission scope — without it every leg decides " +
"independently and the per-run memo never exists, which is the pre-R-181 behaviour")
}
// 2. The DB leg consults it BEFORE the dump. Order is the whole point: a gate after the write is
// the defect, relocated.
assertGateBefore(t, calls["runDBDumpsInternal"], "admitApp", "DumpOne",
"the DATABASE leg dumps before consulting the reserve")
// 3. The volume leg consults it BEFORE the dump seam — which stops the stack as its first act.
assertGateBefore(t, calls["runVolumeDumps"], "admitApp", "dump",
"the VOLUME leg — the one that writes the bulk, and the one B2 never gated — dumps before "+
"consulting the reserve")
// 4. The capture leg, in its own file.
rfset := token.NewFileSet()
rfile, err := parser.ParseFile(rfset, "recovery_unit.go", nil, 0)
if err != nil {
t.Fatal(err)
}
capCalls := map[string][]string{}
current = ""
ast.Inspect(rfile, func(n ast.Node) bool {
switch v := n.(type) {
case *ast.FuncDecl:
current = v.Name.Name
case *ast.CallExpr:
if sel, ok := v.Fun.(*ast.SelectorExpr); ok && current != "" {
capCalls[current] = append(capCalls[current], sel.Sel.Name)
}
}
return true
})
assertGateBefore(t, capCalls["captureAllRecoveryUnits"], "admitApp", "CaptureRecoveryUnit",
"the CAPTURE leg captures before consulting the reserve")
}
// assertGateBefore checks that `gate` appears in the call list before `act`.
func assertGateBefore(t *testing.T, calls []string, gate, act, why string) {
t.Helper()
gi, ai := -1, -1
for i, c := range calls {
if c == gate && gi < 0 {
gi = i
}
if c == act && ai < 0 {
ai = i
}
}
if gi < 0 {
t.Fatalf("%s: %q is never called there at all (calls=%v)", why, gate, calls)
}
if ai < 0 {
t.Fatalf("fixture drift: %q is no longer called in that function (calls=%v) — this test can no "+
"longer see the act it is ordering the gate against", act, calls)
}
if gi > ai {
t.Fatalf("%s: %q first appears at %d, after %q at %d", why, gate, gi, act, ai)
}
}
func hasStr(hay []string, needle string) bool {
for _, s := range hay {
if s == needle {
return true
}
}
return false
}
func equalStrs(a, b []string) bool {
if len(a) != len(b) {
return false
}
for i := range a {
if a[i] != b[i] {
return false
}
}
return true
}
@@ -121,6 +121,11 @@ func NamespaceRoot(drivePath string, inGuestDrive bool) string {
return appbackup.NamespaceRoot(drivePath, inGuestDrive) return appbackup.NamespaceRoot(drivePath, inGuestDrive)
} }
// NamespaceRootFor re-exports the ONE drive-kind-aware resolver (R-203).
func NamespaceRootFor(drivePath, systemDataPath string) string {
return appbackup.NamespaceRootFor(drivePath, systemDataPath)
}
func PrimaryBackupPath(nsRoot string) string { func PrimaryBackupPath(nsRoot string) string {
return appbackup.PrimaryBackupPath(nsRoot) return appbackup.PrimaryBackupPath(nsRoot)
} }
@@ -0,0 +1,229 @@
package backup
import (
"errors"
"fmt"
"io"
"log"
"path/filepath"
"strings"
"testing"
)
// R-174 — the app-stop guard's crash recovery must not start an app onto a MISSING drive.
//
// The defect these pin, found by review on 2026-08-02 in code shipped 2026-08-01 (v0.189.0):
// `appStopGuard.SetStarter(stackMgr)` handed Recover the raw stack manager, whose `StartStack` has
// no drive gate. Recover runs at STARTUP — exactly when an external drive may not have come back —
// so a backup that stopped an app, followed by a power cut and a drive that did not remount, ended
// with the app started onto a missing drive. R-171 one path over.
//
// THE SEAM UNDER TEST IS THE STARTER, not the gate: `internal/backup` must not import `stacks` or
// `settings`, so the production gate lives in `cmd/controller`. What is pinned here is the contract
// between them — that a starter returning ErrStartRefused produces a REFUSAL (marker kept, no alarm)
// and not a FAILURE. The production wiring itself is pinned by TestMainWiresGatedAppStopStarter.
// gatingStarter is a starter whose gate refuses a named set of apps, in the shape the production
// `gatedAppStopStarter` uses: refuse BEFORE calling through, and wrap ErrStartRefused with a reason.
type gatingStarter struct {
inner *fakeStarter
refuse map[string]string // app → reason
refused []string
}
func (s *gatingStarter) StartStack(name string) error {
if why, ok := s.refuse[name]; ok {
s.refused = append(s.refused, name)
return fmt.Errorf("%w: %s", ErrStartRefused, why)
}
return s.inner.StartStack(name)
}
func newGatedGuard(t *testing.T, dir string, refuse map[string]string) (*AppStopGuard, *gatingStarter) {
t.Helper()
s := &gatingStarter{inner: &fakeStarter{}, refuse: refuse}
g := NewAppStopGuard(filepath.Join(dir, "appstop-state.json"), log.New(io.Discard, "", 0))
g.SetStarter(s)
return g, s
}
// --- Scenario A — the guard does not start an app onto a missing drive ---------------------------
func TestRecover_DriveAbsent_RefusesTheStartAndKEEPSTheMarker(t *testing.T) {
dir := t.TempDir()
// process 1: a volume dump stops immich, then the box loses power. No End(), no defer.
g1, _ := newGatedGuard(t, dir, nil)
if err := g1.Begin("volume-dump:immich", ReasonVolumeDump, []string{"immich"}); err != nil {
t.Fatalf("Begin: %v", err)
}
// <power cut> — and immich's drive does NOT come back.
// process 2: a fresh controller starts. The drive is absent.
g2, starter := newGatedGuard(t, dir, map[string]string{
"immich": "drive /mnt/felhom-drives/hdd_1 is not a live mountpoint",
})
res := g2.Recover()
if len(starter.inner.starts) != 0 {
t.Fatalf("started %v — the app was started onto a MISSING drive, which is the whole defect",
starter.inner.starts)
}
if res == nil {
t.Fatal("Recover returned nil — the refusal is invisible to the caller, so nothing can report it")
}
if len(res.Refused) != 1 || res.Refused[0] != "immich" {
t.Fatalf("refused=%v, want [immich]", res.Refused)
}
if len(res.Failed) != 0 {
t.Fatalf("failed=%v — a deliberate hold was recorded as a FAILURE. That bucket reaches "+
"NotifyBackupFailed, which is customer-enabled by default, so the customer would be "+
"emailed \"A biztonsági mentés sikertelen!\" about an app nothing is wrong with (R-171's "+
"false-alarm shape one path over)", res.Failed)
}
if !markerExists(t, dir) {
t.Fatal("the marker was CLEARED after a refused start — the operation is genuinely " +
"unfinished, and clearing it erases the only durable record that immich is owed a restart")
}
// The refusal must name the app AND the reason, or an operator cannot act on it.
if d := res.Detail(); !strings.Contains(d, "held_by_drive") || !strings.Contains(d, "immich") {
t.Fatalf("detail %q does not name the held app", d)
}
if msg := res.Message(); !strings.Contains(msg, "HELD") || !strings.Contains(msg, "drive") {
t.Fatalf("operator message %q does not say the app is held by an absent drive", msg)
}
}
// A refusal-only recovery MUST NOT alarm. This is the assertion that keeps the fix from being the
// bug it fixes: the drive gate doing its job is not a backup failure.
func TestRecover_RefusalOnly_IsNotAlarming(t *testing.T) {
dir := t.TempDir()
g1, _ := newGatedGuard(t, dir, nil)
if err := g1.Begin("volume-dump:immich", ReasonVolumeDump, []string{"immich"}); err != nil {
t.Fatal(err)
}
g2, _ := newGatedGuard(t, dir, map[string]string{"immich": "drive /mnt/felhom-drives/hdd_1 is not a live mountpoint"})
res := g2.Recover()
if res.Alarming() {
t.Fatal("a recovery that only REFUSED starts reports as alarming — main.go would push it " +
"through NotifyBackupFailed and email the customer about a working drive gate")
}
}
// A genuine failure alongside a refusal still alarms, and the two stay in different buckets.
func TestRecover_FailureAlongsideRefusal_StillAlarmsAndKeepsThemApart(t *testing.T) {
dir := t.TempDir()
g1, _ := newGatedGuard(t, dir, nil)
if err := g1.Begin("volume-dump:batch", ReasonVolumeDump, []string{"immich", "nextcloud", "homebox"}); err != nil {
t.Fatal(err)
}
g2, starter := newGatedGuard(t, dir, map[string]string{"immich": "drive /mnt/felhom-drives/hdd_1 is not a live mountpoint"})
starter.inner.failWith = map[string]error{"nextcloud": errors.New("compose up: no such image")}
res := g2.Recover()
if len(res.Refused) != 1 || res.Refused[0] != "immich" {
t.Fatalf("refused=%v, want [immich]", res.Refused)
}
if len(res.Failed) != 1 || res.Failed[0] != "nextcloud" {
t.Fatalf("failed=%v, want [nextcloud]", res.Failed)
}
if len(res.Restarted) != 1 || res.Restarted[0] != "homebox" {
t.Fatalf("restarted=%v, want [homebox] — neither a refusal nor a failure may abort the loop",
res.Restarted)
}
if !res.Alarming() {
t.Fatal("a genuine restart FAILURE alongside a refusal no longer alarms — the refusal " +
"swallowed a real fault")
}
if !markerExists(t, dir) {
t.Fatal("the marker was cleared with work still owed")
}
// The message must not let the held app inflate the failure count.
msg := res.Message()
if !strings.Contains(msg, "1 of 2 app(s) could NOT be restarted") {
t.Fatalf("operator message %q miscounts: the held app must not be counted as a failure", msg)
}
if !strings.Contains(msg, "not counted as failures") {
t.Fatalf("operator message %q does not disclose the held app at all", msg)
}
}
// --- Scenario B — a live drive still recovers normally, byte-identical to before -----------------
func TestRecover_DriveLive_RecoversExactlyAsBefore(t *testing.T) {
dir := t.TempDir()
g1, _ := newGatedGuard(t, dir, nil)
if err := g1.Begin("volume-dump:immich", ReasonVolumeDump, []string{"immich", "nextcloud"}); err != nil {
t.Fatal(err)
}
// Nothing refused — the gate says yes for both.
g2, starter := newGatedGuard(t, dir, nil)
res := g2.Recover()
if len(starter.inner.starts) != 2 {
t.Fatalf("started %v, want both apps — the new gate refused a LEGITIMATE recovery",
starter.inner.starts)
}
if len(res.Refused) != 0 || len(res.Failed) != 0 {
t.Fatalf("refused=%v failed=%v, want neither on a live drive", res.Refused, res.Failed)
}
if len(res.Restarted) != 2 {
t.Fatalf("restarted=%v, want both", res.Restarted)
}
if markerExists(t, dir) {
t.Fatal("the marker survived a fully successful recovery — the next boot would restart the apps again")
}
if !res.Alarming() {
t.Fatal("a successful recovery no longer reports to the operator — the interrupted operation " +
"itself is what §2.4 wants reported, and it went silent")
}
}
// The next startup, with the drive back, completes the recovery and clears the marker. This is what
// makes "keep the marker" a recovery rather than a leak.
func TestRecover_HeldAppIsRestartedOnceTheDriveReturns(t *testing.T) {
dir := t.TempDir()
g1, _ := newGatedGuard(t, dir, nil)
if err := g1.Begin("volume-dump:immich", ReasonVolumeDump, []string{"immich"}); err != nil {
t.Fatal(err)
}
// Boot 1 — drive absent: refused, marker kept.
g2, _ := newGatedGuard(t, dir, map[string]string{"immich": "drive /mnt/felhom-drives/hdd_1 is not a live mountpoint"})
if res := g2.Recover(); len(res.Refused) != 1 {
t.Fatalf("boot 1 refused=%v, want [immich]", res.Refused)
}
if !markerExists(t, dir) {
t.Fatal("boot 1 cleared the marker — boot 2 has nothing to act on and immich stays down forever")
}
// Boot 2 — the drive is back.
g3, starter := newGatedGuard(t, dir, nil)
res := g3.Recover()
if len(starter.inner.starts) != 1 || starter.inner.starts[0] != "immich" {
t.Fatalf("boot 2 started %v, want [immich] — the held app was never picked up again",
starter.inner.starts)
}
if len(res.Restarted) != 1 {
t.Fatalf("boot 2 restarted=%v, want [immich]", res.Restarted)
}
if markerExists(t, dir) {
t.Fatal("boot 2 kept the marker after a fully successful recovery")
}
}
// ErrStartRefused must be matched with errors.Is, i.e. it survives wrapping. A starter that returns
// a bare string reason would land in Failed and alarm — the exact collapse this type prevents.
func TestErrStartRefused_SurvivesWrapping(t *testing.T) {
err := fmt.Errorf("%w: drive /mnt/felhom-drives/hdd_1 is not a live mountpoint", ErrStartRefused)
if !errors.Is(err, ErrStartRefused) {
t.Fatal("a wrapped ErrStartRefused is no longer matched by errors.Is — every refusal would " +
"be recorded as a restart failure and alarm the customer")
}
if errors.Is(errors.New("compose up: no such image"), ErrStartRefused) {
t.Fatal("an ordinary restart failure matches ErrStartRefused — real faults would go silent")
}
}
@@ -0,0 +1,365 @@
package backup
import (
"encoding/json"
"errors"
"fmt"
"log"
"os"
"path/filepath"
"sort"
"time"
)
// ── The app-stop marker (R-166 part 2, decision D-b "in-flight operations") ───────────────────────
//
// Several operations stop a customer's app, do something to its data, and start it again. Between
// the stop and the start, NOTHING ON DISK RECORDED THAT AN APP WAS OWED A RESTART. A controller that
// died in that window left the app down with no explanation anywhere — and because a stopped app has
// zero containers, the boot reconciler read it as a deliberate customer stop and deliberately left
// it alone. Silently, indefinitely.
//
// A `defer` is NOT the fix and must never be described as one. Campaign 8 fault 10 established this
// on live hardware: a SIGKILL runs no deferred function, and what brought the quiesce loop's stacks
// back was its persisted marker read by Recover() one second after restart. The defer covers the
// graceful exits; the marker covers the hard crash and the power cut. This file is that marker for
// the app-data path, modelled directly on internal/quiesce's.
//
// WHY ITS OWN FILE, not quiesce's: one file, one writer. Quiesce's marker records a whole-guest
// backup window and is written by the quiesce loop; this one records an app-data operation and is
// written by the backup manager and the exporter. Sharing the file would give it two writers with
// two lifetimes, and one clearing the other's record is a stranded app by a different route.
//
// SAFETY (D-b's binding rule): losing this file must never be worse than not having it. A lost or
// corrupt marker means the app is not auto-restarted by THIS mechanism — which is precisely the
// pre-v0.189.0 position, not a new hazard. It never deletes, restores, or touches a backup artifact.
// AppStopReason names WHY an app was stopped, so the recovery log tells an operator which operation
// was interrupted rather than merely that something was.
type AppStopReason string
const (
// ReasonVolumeDump — DumpAppVolumesSafe: stop, tar the volumes consistently, start.
ReasonVolumeDump AppStopReason = "volume_dump"
// ReasonOffboxReconstitute — a full offsite restore overwriting the app's files.
ReasonOffboxReconstitute AppStopReason = "offbox_reconstitute"
// ReasonAppExport — a .fab export taken with "stop the app first".
ReasonAppExport AppStopReason = "app_export"
)
// humanReason is the operator-facing phrasing for each reason.
func (r AppStopReason) humanReason() string {
switch r {
case ReasonVolumeDump:
return "an app-data backup (volume dump)"
case ReasonOffboxReconstitute:
return "an off-site restore"
case ReasonAppExport:
return "an app export"
default:
return string(r)
}
}
// AppStopMarker is the persisted "these apps were stopped by an operation that has not reported
// finishing — they are owed a restart" note.
type AppStopMarker struct {
Active bool `json:"active"`
OpID string `json:"op_id"`
Reason AppStopReason `json:"reason"`
Stacks []string `json:"stacks"`
StartedAt time.Time `json:"started_at"`
}
// AppStopStarter is the one thing recovery needs: the ability to start a stack. StartStack must be
// idempotent (it is — `compose up -d` on a running stack is a no-op).
//
// R-174: production MUST pass a GATED starter, never the raw stack manager. Recover runs at STARTUP —
// exactly when an external drive may not have come back — and `Manager.StartStack` has no drive gate
// of its own. See `gatedAppStopStarter` in cmd/controller/main.go.
type AppStopStarter interface {
StartStack(name string) error
}
// ErrStartRefused is what a gated starter returns when a DELIBERATE HOLDER — today the drive gate —
// says an app must not be started. Wrap it (`fmt.Errorf("%w: …", ErrStartRefused)`) so the reason
// survives; Recover matches with errors.Is.
//
// IT IS NOT A FAILURE, AND THE DISTINCTION IS THE WHOLE POINT OF THE TYPE. A refusal means the
// holder is doing its job and owns the restart; a failure means the restart was attempted and broke.
// Collapsing the two would put a deliberately-held app into `Failed`, which main.go reports through
// `NotifyBackupFailed` — a type that is customer-enabled by default (`settings.DefaultEnabledEvents`)
// and carries the Hungarian "A biztonsági mentés sikertelen!". That is R-171's defect one path over:
// a false alarm about an app the drive gate is deliberately holding. Both buckets keep the marker;
// only `Failed` alarms.
var ErrStartRefused = errors.New("start refused by a deliberate holder")
// AppStopGuard owns one marker file. Construct with NewAppStopGuard; the zero value is inert (every
// method is a no-op on a nil guard), so a caller that was never wired degrades to pre-v0.189.0
// behaviour instead of panicking.
type AppStopGuard struct {
path string
logger *log.Logger
now func() time.Time
// starter is only needed by Recover; Begin/End work without one.
starter AppStopStarter
}
// AppStopRecovery is what Recover found and did. Returned rather than pushed through a notifier
// seam, because of a hard ordering constraint: Recover must COMPLETE before the boot reconciler is
// launched (§8.4, main.go:236) and the hub notifier is not constructed until main.go:307. A seam
// wired after the fact would be a seam that never fires — the "built but never wired" shape this
// project has now hit four times. Returning the outcome lets main.go report it the moment the
// notifier exists, and makes the reporting decision visible at the call site instead of buried here.
type AppStopRecovery struct {
Reason AppStopReason
OpID string
StartedAt time.Time
Restarted []string // apps started again by this recovery
Failed []string // apps whose restart was ATTEMPTED and broke (the marker was kept for these)
// Refused are apps a deliberate holder said must not start — today, an absent data drive
// (R-174). The marker is kept for these too, but they are NOT a fault and MUST NOT alarm: the
// holder owns the restart. Separate from Failed for the reason recorded on ErrStartRefused.
Refused []string
}
// Alarming reports whether this recovery is worth paging an operator about. A recovery that only
// REFUSED starts is the drive gate working as designed, and reporting it through the customer-enabled
// `backup_failed` type would be the R-171 false alarm one path over.
func (r *AppStopRecovery) Alarming() bool {
if r == nil {
return false
}
return len(r.Failed) > 0 || len(r.Restarted) > 0
}
// Message is the operator-facing headline for an interrupted operation.
func (r *AppStopRecovery) Message() string {
if r == nil {
return ""
}
if len(r.Failed) > 0 {
m := fmt.Sprintf("%s was interrupted by a controller restart and %d of %d app(s) could NOT be restarted",
r.Reason.humanReason(), len(r.Failed), len(r.Restarted)+len(r.Failed))
if len(r.Refused) > 0 {
m += fmt.Sprintf(" (a further %d are held by an absent drive and are not counted as failures)", len(r.Refused))
}
return m
}
if len(r.Refused) > 0 && len(r.Restarted) == 0 {
return fmt.Sprintf("%s was interrupted by a controller restart — %d app(s) are left stopped and HELD: their data drive is not available, so the drive gate restarts them when it returns",
r.Reason.humanReason(), len(r.Refused))
}
m := fmt.Sprintf("%s was interrupted by a controller restart — %d app(s) were left stopped and have been restarted",
r.Reason.humanReason(), len(r.Restarted))
if len(r.Refused) > 0 {
m += fmt.Sprintf("; %d more are held by an absent drive", len(r.Refused))
}
return m
}
// Detail is the machine-readable tail. App/stack NAMES only — never env values (§9.5).
func (r *AppStopRecovery) Detail() string {
if r == nil {
return ""
}
d := fmt.Sprintf("op=%s reason=%s started_at=%s restarted=%v", r.OpID, r.Reason,
r.StartedAt.UTC().Format(time.RFC3339), r.Restarted)
if len(r.Failed) > 0 {
d += fmt.Sprintf(" restart_failed=%v", r.Failed)
}
if len(r.Refused) > 0 {
d += fmt.Sprintf(" held_by_drive=%v", r.Refused)
}
return d
}
// NewAppStopGuard builds a guard over the given marker path.
func NewAppStopGuard(path string, logger *log.Logger) *AppStopGuard {
if logger == nil {
logger = log.Default()
}
return &AppStopGuard{path: path, logger: logger, now: time.Now}
}
// SetStarter wires the stack-start seam used by Recover. INIT-ONLY — call once at startup, before
// Recover. Separate from the constructor because the guard is built alongside the backup manager,
// which learns its stack provider later (the same shape as SetStackProvider).
func (g *AppStopGuard) SetStarter(s AppStopStarter) {
if g == nil {
return
}
g.starter = s
}
// Begin records that `stacks` are about to be stopped by `reason`. It MUST be called BEFORE the
// first stop — an error here means the marker could not be written, and the caller must not proceed
// to stop an app it cannot promise to restart.
func (g *AppStopGuard) Begin(opID string, reason AppStopReason, stackNames []string) error {
if g == nil || g.path == "" {
return nil // not wired — pre-v0.189.0 behaviour, never a hard failure
}
if len(stackNames) == 0 {
return nil
}
return g.write(AppStopMarker{
Active: true,
OpID: opID,
Reason: reason,
Stacks: append([]string(nil), stackNames...),
StartedAt: g.now(),
})
}
// End clears the marker after a successful restart. Best-effort by contract: a failure to clear is
// logged, never returned as the operation's error — a stale marker costs one idempotent StartStack
// on the next boot, which is exactly D-b's "worst acceptable outcome" and far cheaper than failing
// a backup that actually succeeded.
func (g *AppStopGuard) End() {
if g == nil || g.path == "" {
return
}
if err := os.Remove(g.path); err != nil && !os.IsNotExist(err) {
g.logger.Printf("[ERROR] [appstop] could not clear the app-stop marker at %s: %v (a stale marker costs one idempotent restart at next startup)", g.path, err)
}
}
// Recover restarts any apps left stopped by an operation that died before restarting them, then
// clears the marker. Call ONCE at startup, and — critically — call it to COMPLETION before the boot
// reconciler is launched, so an app this marker explains is not also reported as an unexplained boot
// orphan (§8.4).
//
// Idempotent: StartStack on a running stack is tolerated, and an absent or inactive marker is a
// no-op. On a restart FAILURE the marker is deliberately LEFT IN PLACE — the next startup retries,
// and in the meantime the app is down with desired_state:running, so the boot reconciler sees it as
// an orphan and the dead-app alarm owns it. Clearing a marker whose restart failed would erase the
// only durable record that an app is owed one.
//
// Returns nil when there was nothing to recover — so "no interrupted operation" and "the recovery
// never ran" are distinguishable to the caller, not only in a log (standing rule 3).
func (g *AppStopGuard) Recover() *AppStopRecovery {
if g == nil || g.path == "" {
return nil
}
m, ok := g.read()
if !ok || !m.Active || len(m.Stacks) == 0 {
return nil
}
if g.starter == nil {
g.logger.Printf("[ERROR] [appstop] crash recovery: %d app(s) were stopped by %s and are owed a restart, but no stack starter is wired — leaving the marker for the next startup: %v",
len(m.Stacks), m.Reason.humanReason(), m.Stacks)
return nil
}
g.logger.Printf("[WARN] [appstop] crash recovery: %s (op %q) was interrupted and left %d app(s) stopped — restarting them: %v",
m.Reason.humanReason(), m.OpID, len(m.Stacks), m.Stacks)
res := &AppStopRecovery{Reason: m.Reason, OpID: m.OpID, StartedAt: m.StartedAt}
for _, name := range m.Stacks {
if err := g.starter.StartStack(name); err != nil {
// R-174: a REFUSAL is not a failure. The starter's gate has said this app must not be
// started (an absent data drive), so the app is left down deliberately and the holder
// owns the restart. Logged at WARN with the reason, and kept out of Failed so it never
// reaches the customer-enabled backup_failed alarm — see ErrStartRefused.
if errors.Is(err, ErrStartRefused) {
g.logger.Printf("[WARN] [appstop] crash recovery: NOT restarting %s — %v; the marker is KEPT and the holder owns the restart", name, err)
res.Refused = append(res.Refused, name)
continue
}
g.logger.Printf("[ERROR] [appstop] crash recovery: restart %s failed: %v", name, err)
res.Failed = append(res.Failed, name)
continue
}
g.logger.Printf("[INFO] [appstop] crash recovery: restarted %s after the interrupted %s", name, m.Reason.humanReason())
res.Restarted = append(res.Restarted, name)
}
sort.Strings(res.Failed)
sort.Strings(res.Refused)
sort.Strings(res.Restarted)
// The marker is kept for BOTH unfinished outcomes, for the same reason and with different
// urgency: a failed restart is retried next startup, and a refused one is genuinely unfinished
// until its drive returns. Clearing it in either case would erase the only durable record that
// an app is owed a restart.
if len(res.Failed) > 0 {
g.logger.Printf("[ERROR] [appstop] crash recovery: %d app(s) could not be restarted — KEEPING the marker so the next startup retries; the dead-app alarm owns them meanwhile: %v",
len(res.Failed), res.Failed)
return res
}
if len(res.Refused) > 0 {
g.logger.Printf("[WARN] [appstop] crash recovery: %d app(s) were deliberately NOT restarted (drive absent) — KEEPING the marker; this is the gate working, not a fault: %v",
len(res.Refused), res.Refused)
return res
}
g.End()
return res
}
// HeldStacks returns the stacks an app-data operation is CURRENTLY holding down, or nil.
//
// Read-only and nil-safe. It exists for the boot reconciler (§8.2): once R-157 mechanism A widened
// the boot window, the sweep could overlap a running volume dump or export and "recover" an app that
// is deliberately stopped mid-operation — restarting it under a tar, which is the inconsistency the
// stop was taken to avoid. Recover() has already run to completion by then, so a marker seen through
// this method belongs to an operation running NOW, not to a crashed one.
func (g *AppStopGuard) HeldStacks() []string {
if g == nil || g.path == "" {
return nil
}
m, ok := g.read()
if !ok || !m.Active {
return nil
}
return append([]string(nil), m.Stacks...)
}
// ---- marker persistence (atomic, 0600) — the quiesce shape ------------------------------------
func (g *AppStopGuard) write(m AppStopMarker) error {
data, err := json.MarshalIndent(m, "", " ")
if err != nil {
return err
}
if err := os.MkdirAll(filepath.Dir(g.path), 0o755); err != nil {
return err
}
tmp := g.path + ".tmp"
f, err := os.OpenFile(tmp, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600)
if err != nil {
return err
}
if _, err := f.Write(data); err != nil {
f.Close()
os.Remove(tmp)
return err
}
// fsync before rename: the whole point is surviving a power cut, and a rename that lands ahead
// of the bytes it points at is a marker that reads as corrupt at exactly the wrong moment.
if err := f.Sync(); err != nil {
f.Close()
os.Remove(tmp)
return err
}
if err := f.Close(); err != nil {
os.Remove(tmp)
return err
}
return os.Rename(tmp, g.path)
}
func (g *AppStopGuard) read() (AppStopMarker, bool) {
data, err := os.ReadFile(g.path)
if err != nil {
return AppStopMarker{}, false
}
var m AppStopMarker
if err := json.Unmarshal(data, &m); err != nil {
// Never a silent skip (§9.4): a corrupt marker is LOUD and the bad file is quarantined, so a
// genuinely interrupted operation leaves a trace instead of vanishing. Still returns false —
// "no usable marker ⇒ no recovery" is the correct contract, and matches quiesce's.
g.logger.Printf("[WARN] [appstop] the app-stop marker at %s is corrupt (%v) — quarantining; apps are NOT auto-restarted from it", g.path, err)
_ = os.Rename(g.path, fmt.Sprintf("%s.corrupt-%d", g.path, g.now().Unix()))
return AppStopMarker{}, false
}
return m, true
}
@@ -0,0 +1,395 @@
package backup
import (
"encoding/json"
"errors"
"io"
"log"
"os"
"path/filepath"
"strings"
"testing"
)
// R-166 part 2 — the app-stop crash marker.
//
// THE DISCIPLINE THAT MATTERS HERE (§10): a `defer` is not crash-safety, so a test that lets the
// deferred cleanup run proves nothing about a crash. Every "interrupted" test below simulates a
// SIGKILL by never reaching the restart — the marker is written, the process conceptually dies, and
// a FRESH guard over the SAME file does the recovering. That is exactly what Campaign 8 fault 10
// established on live hardware: a SIGKILL runs no deferred function, and what brought the stacks
// back was the marker read at startup.
type fakeStarter struct {
starts []string
failWith map[string]error
}
func (f *fakeStarter) StartStack(name string) error {
f.starts = append(f.starts, name)
if err := f.failWith[name]; err != nil {
return err
}
return nil
}
func newGuard(t *testing.T, dir string) (*AppStopGuard, *fakeStarter) {
t.Helper()
s := &fakeStarter{}
g := NewAppStopGuard(filepath.Join(dir, "appstop-state.json"), log.New(io.Discard, "", 0))
g.SetStarter(s)
return g, s
}
func markerPath(dir string) string { return filepath.Join(dir, "appstop-state.json") }
func markerExists(t *testing.T, dir string) bool {
t.Helper()
_, err := os.Stat(markerPath(dir))
if err != nil && !os.IsNotExist(err) {
t.Fatal(err)
}
return err == nil
}
// --- Scenario E — a crash mid-backup brings the app back -----------------------------------------
func TestRecover_InterruptedVolumeDump_RestartsTheAppAndClearsTheMarker(t *testing.T) {
dir := t.TempDir()
// --- process 1: an operation stops the app and is KILLED. No End(), no defer, no cleanup. ---
g1, _ := newGuard(t, dir)
if err := g1.Begin("volume-dump:immich", ReasonVolumeDump, []string{"immich"}); err != nil {
t.Fatalf("Begin: %v", err)
}
if !markerExists(t, dir) {
t.Fatal("Begin did not write a marker — nothing would survive the kill")
}
// <SIGKILL here> — g1 is abandoned deliberately; nothing else is called on it.
// --- process 2: a fresh controller starts and recovers from the file alone. ---
g2, starter := newGuard(t, dir)
res := g2.Recover()
if len(starter.starts) != 1 || starter.starts[0] != "immich" {
t.Fatalf("started %v, want exactly [immich] — the app was left stranded by the interrupted backup", starter.starts)
}
if res == nil || len(res.Restarted) != 1 || res.Restarted[0] != "immich" {
t.Fatalf("recovery result = %+v, want immich restarted", res)
}
if res.Reason != ReasonVolumeDump {
t.Fatalf("reason = %q, want %q — the operator must be told WHICH operation was interrupted", res.Reason, ReasonVolumeDump)
}
if markerExists(t, dir) {
t.Fatal("the marker survived a successful recovery — the next boot would restart the app again")
}
// The operator-facing text must name the interruption, not merely report a restart.
if msg := res.Message(); msg == "" || !strings.Contains(msg, "interrupted") {
t.Fatalf("operator message %q does not say the operation was interrupted", msg)
}
}
func TestRecover_NoMarker_IsASilentNoOp(t *testing.T) {
dir := t.TempDir()
g, starter := newGuard(t, dir)
if res := g.Recover(); res != nil {
t.Fatalf("Recover reported %+v on a box with no marker", res)
}
if len(starter.starts) != 0 {
t.Fatalf("started %v with no marker present", starter.starts)
}
}
func TestRecover_FailedRestart_KEEPSTheMarkerForTheNextStartup(t *testing.T) {
// The single most important failure behaviour: clearing a marker whose restart failed would
// erase the only durable record that an app is owed one. The app is genuinely still down.
dir := t.TempDir()
g1, _ := newGuard(t, dir)
if err := g1.Begin("volume-dump:immich", ReasonVolumeDump, []string{"immich", "nextcloud"}); err != nil {
t.Fatal(err)
}
g2, starter := newGuard(t, dir)
starter.failWith = map[string]error{"immich": errors.New("compose up: no such image")}
res := g2.Recover()
if len(res.Failed) != 1 || res.Failed[0] != "immich" {
t.Fatalf("failed=%v, want [immich]", res.Failed)
}
if len(res.Restarted) != 1 || res.Restarted[0] != "nextcloud" {
t.Fatalf("restarted=%v, want [nextcloud] — one app failing must not abort the others", res.Restarted)
}
if !markerExists(t, dir) {
t.Fatal("the marker was cleared even though a restart FAILED — the next startup would not retry")
}
if msg := res.Message(); !strings.Contains(msg, "NOT be restarted") {
t.Fatalf("operator message %q does not report the failure", msg)
}
if d := res.Detail(); !strings.Contains(d, "restart_failed") || !strings.Contains(d, "immich") {
t.Fatalf("detail %q does not name which app failed", d)
}
}
func TestRecover_IsIdempotentAcrossRepeatedStartups(t *testing.T) {
dir := t.TempDir()
g1, _ := newGuard(t, dir)
if err := g1.Begin("op", ReasonOffboxReconstitute, []string{"immich"}); err != nil {
t.Fatal(err)
}
g2, s2 := newGuard(t, dir)
g2.Recover()
g3, s3 := newGuard(t, dir)
g3.Recover()
if len(s2.starts) != 1 {
t.Fatalf("first recovery started %v", s2.starts)
}
if len(s3.starts) != 0 {
t.Fatalf("a SECOND startup restarted %v again — the marker was not cleared", s3.starts)
}
}
func TestRecover_CorruptMarkerIsQuarantinedNotSilentlySkipped(t *testing.T) {
// §9.4: never a silent skip. A corrupt marker cannot be acted on, but it must leave a trace —
// otherwise a genuinely interrupted operation vanishes without evidence.
dir := t.TempDir()
if err := os.WriteFile(markerPath(dir), []byte("{not json"), 0o600); err != nil {
t.Fatal(err)
}
g, starter := newGuard(t, dir)
if res := g.Recover(); res != nil {
t.Fatalf("a corrupt marker produced a recovery result %+v", res)
}
if len(starter.starts) != 0 {
t.Fatalf("apps were started from a corrupt marker: %v", starter.starts)
}
if markerExists(t, dir) {
t.Fatal("the corrupt marker was left in place — it would be re-read forever")
}
quarantined, _ := filepath.Glob(markerPath(dir) + ".corrupt-*")
if len(quarantined) != 1 {
t.Fatalf("the corrupt marker was not quarantined (found %d) — it was silently dropped", len(quarantined))
}
}
func TestRecover_NoStarterWiredKeepsTheMarker(t *testing.T) {
// D-b's safety rule: never worse than not having the file. With no starter the guard cannot act,
// so it must keep the record for a startup that can, rather than clear it and lose the app.
dir := t.TempDir()
g1, _ := newGuard(t, dir)
if err := g1.Begin("op", ReasonVolumeDump, []string{"immich"}); err != nil {
t.Fatal(err)
}
g2 := NewAppStopGuard(markerPath(dir), log.New(io.Discard, "", 0)) // deliberately no SetStarter
if res := g2.Recover(); res != nil {
t.Fatalf("recovered without a starter: %+v", res)
}
if !markerExists(t, dir) {
t.Fatal("the marker was cleared with no starter wired — the app would never come back")
}
}
func TestNilGuardIsInert(t *testing.T) {
// A caller that was never wired must degrade to pre-v0.189.0 behaviour, not panic.
var g *AppStopGuard
if err := g.Begin("op", ReasonVolumeDump, []string{"x"}); err != nil {
t.Fatalf("nil guard Begin returned %v", err)
}
g.End()
if res := g.Recover(); res != nil {
t.Fatalf("nil guard recovered %+v", res)
}
}
func TestMarkerContentsAreDiagnosable(t *testing.T) {
dir := t.TempDir()
g, _ := newGuard(t, dir)
if err := g.Begin("volume-dump:immich", ReasonVolumeDump, []string{"immich"}); err != nil {
t.Fatal(err)
}
raw, err := os.ReadFile(markerPath(dir))
if err != nil {
t.Fatal(err)
}
var m AppStopMarker
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatalf("the marker on disk is not readable JSON: %v", err)
}
if !m.Active || m.OpID != "volume-dump:immich" || m.Reason != ReasonVolumeDump ||
len(m.Stacks) != 1 || m.Stacks[0] != "immich" || m.StartedAt.IsZero() {
t.Fatalf("the marker does not record enough to diagnose the interruption: %+v", m)
}
// 0600 — it names customer apps.
fi, err := os.Stat(markerPath(dir))
if err != nil {
t.Fatal(err)
}
if fi.Mode().Perm() != 0o600 {
t.Fatalf("marker mode = %v, want 0600", fi.Mode().Perm())
}
}
func TestBeginWithNoStacksWritesNothing(t *testing.T) {
dir := t.TempDir()
g, _ := newGuard(t, dir)
if err := g.Begin("op", ReasonVolumeDump, nil); err != nil {
t.Fatal(err)
}
if markerExists(t, dir) {
t.Fatal("a marker was written for an operation that stops nothing")
}
}
// --- Scenarios E/F — DumpAppVolumesSafe, the primary site ----------------------------------------
// inspectingProvider is the StackDataProvider slice DumpAppVolumesSafe touches. It records whether
// the marker file EXISTED at each step — the positive observable for the ordering property. An
// absent log line is not evidence (standing rule 3); the file's presence at the moment of the stop
// is.
//
// GetDockerVolumes returns nothing, so the dump itself is a no-op and no Docker is involved — the
// stop/start bracket around it is what is under test.
type inspectingProvider struct {
StackDataProvider
markerFile string
events []string
stopErr error
startErr error
markerPresentAtStop bool
markerAtStartCall bool
// panicOnVolumes simulates a hard abort (SIGKILL/power cut) at the point the dump begins: the
// unwind skips the restart statement, exactly as a kill would.
panicOnVolumes bool
}
func (p *inspectingProvider) GetDockerVolumes(string) []string {
if p.panicOnVolumes {
panic("simulated hard abort mid-dump")
}
return nil
}
func (p *inspectingProvider) StopStack(name string) error {
_, err := os.Stat(p.markerFile)
p.markerPresentAtStop = err == nil
p.events = append(p.events, "stop:"+name)
return p.stopErr
}
func (p *inspectingProvider) StartStack(name string) error {
_, err := os.Stat(p.markerFile)
p.markerAtStartCall = err == nil
p.events = append(p.events, "start:"+name)
return p.startErr
}
func newDumpManager(t *testing.T, dir string, p *inspectingProvider) *Manager {
t.Helper()
lg := log.New(io.Discard, "", 0)
m := &Manager{logger: lg, stackProvider: p, systemDataPath: dir}
m.appStop = NewAppStopGuard(markerPath(dir), lg)
return m
}
func TestDumpAppVolumesSafe_MarkerCoversTheWholeStopStartWindow(t *testing.T) {
// Scenario F, the happy path: the marker is on disk BEFORE the stop, still on disk for the whole
// time the app is down, and GONE once the restart succeeds.
dir := t.TempDir()
p := &inspectingProvider{markerFile: markerPath(dir)}
m := newDumpManager(t, dir, p)
if err := m.DumpAppVolumesSafe("immich"); err != nil {
t.Fatalf("DumpAppVolumesSafe: %v", err)
}
if !p.markerPresentAtStop {
t.Fatal("the marker was NOT on disk when the app was stopped — a crash one instruction later " +
"strands the app, which is the entire failure this marker exists to prevent")
}
if !p.markerAtStartCall {
t.Fatal("the marker was already gone while the app was still down")
}
if markerExists(t, dir) {
t.Fatal("the marker survived a dump whose restart succeeded — the next boot would restart the app again")
}
if len(p.events) != 2 || p.events[0] != "stop:immich" || p.events[1] != "start:immich" {
t.Fatalf("events=%v, want [stop:immich start:immich]", p.events)
}
}
func TestDumpAppVolumesSafe_Interrupted_RecoveryBringsTheAppBack(t *testing.T) {
// Scenario E end-to-end THROUGH THE PRODUCTION PATH, and WITHOUT running any cleanup.
//
// The abort is real: GetDockerVolumes panics, which unwinds out of DumpAppVolumesSafe AFTER the
// marker was written and the app stopped, and BEFORE the restart statement — and because that
// restart is a plain statement, not a defer, it never runs. That is the shape of a hard kill.
//
// The earlier version of this test called m.appStop.Begin itself, which meant it proved the
// marker type worked and NOT that DumpAppVolumesSafe uses it — it survived the red-proof that
// deleted the production Begin call. Driving the real function is what makes the proof bite.
//
// RED-PROOF: delete the `m.appStop.Begin(...)` call from DumpAppVolumesSafe and this test fails —
// nothing is written, so nothing is recovered. Demonstrated in REPORT.md §5.
dir := t.TempDir()
p := &inspectingProvider{markerFile: markerPath(dir), panicOnVolumes: true}
m := newDumpManager(t, dir, p)
func() {
defer func() {
if recover() == nil {
t.Error("the simulated abort did not fire — this test proves nothing")
}
}()
_ = m.DumpAppVolumesSafe("immich")
}()
if !p.markerPresentAtStop {
t.Fatal("the app was stopped before any marker existed")
}
if p.markerAtStartCall {
t.Fatal("the restart ran despite the abort — the simulation is wrong, not the code")
}
// <the controller is gone> — a fresh one starts and recovers from the file alone.
g, starter := newGuard(t, dir)
res := g.Recover()
if len(starter.starts) != 1 || starter.starts[0] != "immich" {
t.Fatalf("started %v — the app stopped by the interrupted dump was not brought back", starter.starts)
}
if res == nil || res.Reason != ReasonVolumeDump {
t.Fatalf("recovery did not name the volume dump as the interrupted operation: %+v", res)
}
if markerExists(t, dir) {
t.Fatal("the marker was not cleared after a successful recovery")
}
}
func TestDumpAppVolumesSafe_FailedRestartKeepsTheMarker(t *testing.T) {
dir := t.TempDir()
p := &inspectingProvider{markerFile: markerPath(dir), startErr: errors.New("compose up failed")}
m := newDumpManager(t, dir, p)
if err := m.DumpAppVolumesSafe("immich"); err == nil {
t.Fatal("a failed restart must surface as an error")
}
if !markerExists(t, dir) {
t.Fatal("the marker was cleared even though the restart FAILED — the app is still down and " +
"nothing records that it is owed a restart")
}
}
func TestDumpAppVolumesSafe_FailedStopClearsTheMarker(t *testing.T) {
// Nothing was stopped, so nothing is owed a restart. A stranded marker here would cost a
// spurious restart at the next startup AND a false "a backup was interrupted" alert.
dir := t.TempDir()
p := &inspectingProvider{markerFile: markerPath(dir), stopErr: errors.New("stack is protected")}
m := newDumpManager(t, dir, p)
if err := m.DumpAppVolumesSafe("traefik"); err == nil {
t.Fatal("a failed stop must surface as an error")
}
if markerExists(t, dir) {
t.Fatal("a marker was left behind for an app that was never stopped")
}
}
+162 -1
View File
@@ -9,6 +9,7 @@ import (
"path/filepath" "path/filepath"
"strings" "strings"
"sync" "sync"
"sync/atomic"
"time" "time"
"gitea.dooplex.hu/admin/felhom-controller/internal/config" "gitea.dooplex.hu/admin/felhom-controller/internal/config"
@@ -32,6 +33,54 @@ type Manager struct {
// tier2Notify, if set, is called after each Tier 2 copy (success: err==nil) for notifications. // tier2Notify, if set, is called after each Tier 2 copy (success: err==nil) for notifications.
tier2Notify func(stackName, destLabel string, dur time.Duration, err error) tier2Notify func(stackName, destLabel string, dur time.Duration, err error)
// unitNotify (R-158 / R-167), if set, is called ONCE PER APP whose Tier-1 recovery-unit capture
// FAILED, and the capture loop continues to the next app. Wired in cmd/controller/main.go.
//
// WHY IT EXISTS. `/backups/apps` is the page a person opens to ask whether ONE app is backed up,
// and until now it was the one page that never said: a per-app capture failure was a `[WARN]`
// line and went no further. The manager had three notify seams and none for the unit capture —
// the FIFTH instance in this project of a mechanism built and left disconnected.
//
// IT CARRIES THE SPACE FIGURES DELIBERATELY. The overwhelmingly likely cause is a full
// filesystem, and an operator who has the used/free bytes at the moment of failure can act
// without logging in. It is the same pair of numbers the customer-facing fill warning reports,
// which is why the two ship together.
//
// OPERATOR-TIER. Routed to a hub event type that is in `notify.operatorOnlyEvents` — a customer
// can take no action on a capture failure. Deliberately NOT `backup_failed`, which is
// customer-enabled by default and would email them in Hungarian about it (D-c).
//
// NO CONTROLLER-SIDE COOLDOWN — the hub owns cooldown, per the offboxEnlargeBlockedNotify
// precedent.
unitNotify func(stackName string, err error, usage *UnitSpace)
// unitSpaceFn (R-165 / B2), if set, replaces the real statfs behind the capture floor so a test
// can state a filesystem's occupancy as an input. Nil in production → `unitTargetSpace`.
unitSpaceFn func(stackName string) *UnitSpace
// admission (R-181) is the per-RUN memo of the reserve's per-app verdict, guarded by admissionMu.
// Non-nil only for the duration of a backup run (beginAdmissionRun → its closer). One verdict per
// app covers all THREE write legs — DB dump, volume dump, unit capture — because all three write
// under one per-app root; see admission.go for why it is decided lazily and never re-decided.
admissionMu sync.Mutex
admission *admissionSet
// summary (R-182) is the per-RUN digest collector, guarded by summaryMu. Same lifetime as
// `admission` and for the same reason: an absent collector means "no run in flight", never a
// stale answer from last night. runSummaryNotify is the operator digest seam, wired in main.go.
summaryMu sync.Mutex
summary *runSummary
runSummaryNotify func(RunSummary)
// manualRun tags the NEXT run as operator-triggered (cleared as the run starts), so the digest
// can say which kind it was and the hub can decline to collapse a manual run into a nightly one.
manualRun atomic.Bool
// appStop (R-166) is the crash marker for operations that stop an app, work on its data, and
// start it again. Written BEFORE the stop and cleared AFTER the restart, so a SIGKILL or a power
// cut in that window leaves a durable record that Recover honours at the next startup. Built in
// NewManager from cfg.Paths.DataDir — see appstop_marker.go for why it is not quiesce's file.
appStop *AppStopGuard
// offbox (Part B): the restic-SFTP exec seam (nil → real restic) + the failure→operator-alert hook. // offbox (Part B): the restic-SFTP exec seam (nil → real restic) + the failure→operator-alert hook.
offboxRunner offboxRunner offboxRunner offboxRunner
offboxNotify func(dur time.Duration, snapshots int, err error) offboxNotify func(dur time.Duration, snapshots int, err error)
@@ -44,6 +93,9 @@ type Manager struct {
// ("offbox_repo_orphaned" / "offbox_repo_reset"); renamedTo names the move-aside path (reset only). // ("offbox_repo_orphaned" / "offbox_repo_reset"); renamedTo names the move-aside path (reset only).
// Wired in main.go to the notifier. Nil-safe. // Wired in main.go to the notifier. Nil-safe.
offboxOrphanEvent func(eventType, renamedTo string) offboxOrphanEvent func(eventType, renamedTo string)
// offboxGapNotify (R-203) fires when a COMPLETED offsite run could not capture a directory an
// app declares MANDATORY — a coverage gap, not a failed run. nil → no signal.
offboxGapNotify func(gaps map[string][]string)
// offboxSSH (v0.142.0) is the raw-ssh exec seam for the orphaned-repo move-aside (restic has no // offboxSSH (v0.142.0) is the raw-ssh exec seam for the orphaned-repo move-aside (restic has no
// rename); tests inject a fake. Nil → the real ssh invocation (defaultOffboxSSH). // rename); tests inject a fake. Nil → the real ssh invocation (defaultOffboxSSH).
offboxSSH func(ctx context.Context, host, user string, port int, keyPath, knownHosts, remoteCmd string) ([]byte, error) offboxSSH func(ctx context.Context, host, user string, port int, keyPath, knownHosts, remoteCmd string) ([]byte, error)
@@ -51,6 +103,13 @@ type Manager struct {
// offboxSizer (3a) — the mandatory-set byte estimator for the pre-push enlargement gate, overridable // offboxSizer (3a) — the mandatory-set byte estimator for the pre-push enlargement gate, overridable
// in tests so the gate is unit-testable without a real du. Nil → the real dirSizeBytes (du -sb). // in tests so the gate is unit-testable without a real du. Nil → the real dirSizeBytes (du -sb).
offboxSizer func(path string) int64 offboxSizer func(path string) int64
// offboxNow (v0.206.0, R-241) is the abandonment countdown's clock. Nil → time.Now.
//
// IT EXISTS SO THE TERMINAL STEP IS TESTABLE WITHOUT SHORTENING A LIVE TIMER (§7.4). The sweep is
// the only thing in the product that deletes a customer's off-site history; driving it with a
// clock keeps that step exercised on every run of the suite instead of once, on real data, by an
// operator who then has to hope.
offboxNow func() time.Time
// offboxEnlargeBlockedNotify (3a), if set, is called ONCE per app that NEWLY enters the // offboxEnlargeBlockedNotify (3a), if set, is called ONCE per app that NEWLY enters the
// quota-blocked (enlargement-refused) state — edge-triggered against the persisted EnlargedBlocked // quota-blocked (enlargement-refused) state — edge-triggered against the persisted EnlargedBlocked
// set so a nightly schedule can't re-notify a persistently-blocked app (the hub owns cooldown; the // set so a nightly schedule can't re-notify a persistently-blocked app (the hub owns cooldown; the
@@ -216,10 +275,30 @@ func NewManager(cfg *config.Config, sett *settings.Settings, logger *log.Logger)
settings: sett, settings: sett,
systemDataPath: cfg.Paths.SystemDataPath, systemDataPath: cfg.Paths.SystemDataPath,
} }
// R-166: its OWN file next to quiesce-state.json, never inside it — one file, one writer.
m.appStop = NewAppStopGuard(filepath.Join(cfg.Paths.DataDir, "appstop-state.json"), logger)
m.reconcileCrashedRun() m.reconcileCrashedRun()
return m return m
} }
// AppStopGuard exposes the app-stop crash marker so the exporter (a different package with the same
// stop-work-start shape) can share the one marker file rather than opening a second one.
func (m *Manager) AppStopGuard() *AppStopGuard { return m.appStop }
// SetAppStopGuard injects the guard instead of using the one NewManager built. INIT-ONLY — call once
// during single-threaded startup, before any backup runs.
//
// It exists because of a startup ORDERING constraint, not for testing: the guard's Recover must
// complete before the boot reconciler is launched (main.go:~236) and this manager is not constructed
// until ~line 272. So main.go builds the guard early, recovers, and hands the SAME object here —
// rather than a second guard over the same file, which would be one file with two owners, the exact
// shape this marker was kept out of quiesce's file to avoid.
func (m *Manager) SetAppStopGuard(g *AppStopGuard) {
if g != nil {
m.appStop = g
}
}
// reconcileCrashedRun makes the persisted offbox status truthful after a crash (campaign C1): a controller // reconcileCrashedRun makes the persisted offbox status truthful after a crash (campaign C1): a controller
// that died mid-run left LastStatus="running" on disk (the in-memory single-flight mutex is gone with the // that died mid-run left LastStatus="running" on disk (the in-memory single-flight mutex is gone with the
// process, but the persisted status keeps lying "running" forever). Flip it to error with a Hungarian // process, but the persisted status keeps lying "running" forever). Flip it to error with a Hungarian
@@ -260,7 +339,10 @@ func (m *Manager) GetAppDrivePath(stackName string) string {
// as-is; only the SSD-only system-data fallback gets the felhom-data subdir appended. This is what // as-is; only the SSD-only system-data fallback gets the felhom-data subdir appended. This is what
// keeps a drive-resident app's backups single-nested instead of .../felhom-data/felhom-data/... . // keeps a drive-resident app's backups single-nested instead of .../felhom-data/felhom-data/... .
func (m *Manager) namespaceRoot(drivePath string) string { func (m *Manager) namespaceRoot(drivePath string) string {
return NamespaceRoot(drivePath, drivePath != m.systemDataPath) // R-203: delegates to the ONE expression of the rule (appbackup.NamespaceRootFor). This used to
// hold its own copy — `drivePath != m.systemDataPath`, without Clean on either side — while
// stacks.Manager.inGuest held a second copy WITH Clean. Two copies that already differed.
return NamespaceRootFor(drivePath, m.systemDataPath)
} }
// AppNamespaceRoot returns the felhom-data namespace root for a stack's keep-side backups, resolving // AppNamespaceRoot returns the felhom-data namespace root for a stack's keep-side backups, resolving
@@ -361,6 +443,21 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
start := time.Now() start := time.Now()
m.logger.Printf("[INFO] [backup] Starting database dump run") m.logger.Printf("[INFO] [backup] Starting database dump run")
// R-181: open the per-run admission scope HERE, because this function is the single orchestrator
// of all three write legs. Each app's reserve verdict is taken at its first write of this run and
// then reused by the other two legs, so a refused app writes nothing at all and is alerted once.
// The scope is closed on every exit path — a set that outlived its run would answer tonight's
// question with last night's disk.
defer m.beginAdmissionRun()()
// R-182: the digest scope has the same lifetime. `emitRunSummary` runs BEFORE the closer (defers
// unwind last-in-first-out), so the summary is still populated when it is sent, and it sends
// nothing at all when the run was clean.
kind := m.runKindFor()
m.manualRun.Store(false) // tags exactly ONE run; a stale flag would mislabel every later nightly
defer m.beginRunSummary(kind, newRunID())()
defer m.emitRunSummary()
dbs, err := DiscoverDatabases(ctx, m.logger, m.isDebug(), m.knownStackNames()) dbs, err := DiscoverDatabases(ctx, m.logger, m.isDebug(), m.knownStackNames())
if err != nil { if err != nil {
m.logger.Printf("[ERROR] [backup] Database discovery failed: %v", err) m.logger.Printf("[ERROR] [backup] Database discovery failed: %v", err)
@@ -396,6 +493,16 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
continue continue
} }
// R-181: the reserve, BEFORE the first byte of this app's backup is written. This is usually
// where an app's verdict is taken, because the DB leg runs first; the volume leg and the
// capture then read the same memo. SKIP, not FAIL — a deliberate hold is not a broken dump,
// and the operator alert (fired once, inside admitApp) is the signal that it happened.
m.noteAttempted(db.StackName)
if !m.admitApp(db.StackName) {
summary = append(summary, fmt.Sprintf("SKIP %s (reserve — app backup refused)", db.ContainerName))
continue
}
dumpDir := AppDBDumpPath(m.namespaceRoot(drivePath), db.StackName) dumpDir := AppDBDumpPath(m.namespaceRoot(drivePath), db.StackName)
result := DumpOne(ctx, db, dumpDir, m.logger, m.isDebug()) result := DumpOne(ctx, db, dumpDir, m.logger, m.isDebug())
@@ -404,6 +511,7 @@ func (m *Manager) runDBDumpsInternal(ctx context.Context) error {
if result.Error != nil { if result.Error != nil {
allOK = false allOK = false
summary = append(summary, fmt.Sprintf("FAIL %s: %v", result.DB.ContainerName, result.Error)) summary = append(summary, fmt.Sprintf("FAIL %s: %v", result.DB.ContainerName, result.Error))
m.noteFailure(db.StackName, "database dump", result.Error.Error())
m.logger.Printf("[ERROR] [backup] DB dump failed for %s: %v", result.DB.ContainerName, result.Error) m.logger.Printf("[ERROR] [backup] DB dump failed for %s: %v", result.DB.ContainerName, result.Error)
} else { } else {
totalSize += result.Size totalSize += result.Size
@@ -490,6 +598,12 @@ func failedSummaryLines(summary []string) []string {
// variant stops the stack before its own volume check — calling it unconditionally would bounce // variant stops the stack before its own volume check — calling it unconditionally would bounce
// every volume-less app on every nightly run. Per-stack isolation mirrors the DB loop: one app's // every volume-less app on every nightly run. Per-stack isolation mirrors the DB loop: one app's
// failure is recorded and does not abort the others. // failure is recorded and does not abort the others.
//
// R-181 adds the reserve to that order, and for the SAME reason: it sits ahead of DumpAppVolumesSafe,
// so a refused app is never stopped. A refusal decided inside the Safe variant would already have
// bounced the app it was refusing to back up. It sits AFTER the volume-less check because an app with
// no named volumes writes nothing in this leg — there is no first write here to gate, and consulting
// the reserve for it would only decide a verdict early on a stale reading.
func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) { func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) {
allOK = true allOK = true
if m.stackProvider == nil { if m.stackProvider == nil {
@@ -525,9 +639,19 @@ func (m *Manager) runVolumeDumps() (summary []string, dumped int, allOK bool) {
continue continue
} }
// R-181: the reserve, ahead of DumpAppVolumesSafe so a refused app is NOT stopped. For an app
// that already has a DB this is a memo lookup taken before its DB dump; for a volume-only app
// this is where its verdict is taken, still before its first byte.
m.noteAttempted(stack.Name)
if !m.admitApp(stack.Name) {
summary = append(summary, fmt.Sprintf("SKIP %s volumes (reserve — app backup refused)", stack.Name))
continue
}
if err := dump(stack.Name); err != nil { if err := dump(stack.Name); err != nil {
allOK = false allOK = false
summary = append(summary, fmt.Sprintf("FAIL %s volumes: %v", stack.Name, err)) summary = append(summary, fmt.Sprintf("FAIL %s volumes: %v", stack.Name, err))
m.noteFailure(stack.Name, "volume dump", err.Error())
m.logger.Printf("[ERROR] [backup] Volume dump failed for %s: %v", stack.Name, err) m.logger.Printf("[ERROR] [backup] Volume dump failed for %s: %v", stack.Name, err)
continue continue
} }
@@ -679,13 +803,28 @@ func atomicPromoteTar(tmpPath, finalPath string) error {
// DumpAppVolumesSafe stops the stack before dumping volumes and restarts after. // DumpAppVolumesSafe stops the stack before dumping volumes and restarts after.
// Prevents inconsistent tars of live database volumes (e.g. PostgreSQL). // Prevents inconsistent tars of live database volumes (e.g. PostgreSQL).
// Protected stacks that reject StopStack will return an error — callers handle as warning. // Protected stacks that reject StopStack will return an error — callers handle as warning.
//
// R-166: the stop→dump→start window is marked. Before this, a controller killed between the stop
// and the start left the app down with NOTHING on disk saying why or that it was owed a restart —
// and a stopped app has zero containers, which the boot reconciler then read as a deliberate
// customer stop and left alone. The marker is the mechanism, not the restart call below: a SIGKILL
// runs no deferred function (Campaign 8 fault 10, on live hardware), so only something already
// written to disk can survive it.
func (m *Manager) DumpAppVolumesSafe(stackName string) error { func (m *Manager) DumpAppVolumesSafe(stackName string) error {
if m.stackProvider == nil { if m.stackProvider == nil {
return fmt.Errorf("no stack provider") return fmt.Errorf("no stack provider")
} }
// Intent before the act: refuse to stop an app we cannot promise to restart.
if err := m.appStop.Begin("volume-dump:"+stackName, ReasonVolumeDump, []string{stackName}); err != nil {
return fmt.Errorf("could not record the app-stop marker for %s (refusing to stop it unprotected): %w", stackName, err)
}
m.logger.Printf("[INFO] [backup] Stopping %s for safe volume dump", stackName) m.logger.Printf("[INFO] [backup] Stopping %s for safe volume dump", stackName)
if err := m.stackProvider.StopStack(stackName); err != nil { if err := m.stackProvider.StopStack(stackName); err != nil {
// Nothing was stopped, so nothing is owed a restart — clear rather than strand a marker that
// would cost a spurious (if harmless) restart at the next startup.
m.appStop.End()
return fmt.Errorf("could not stop %s for volume dump: %w", stackName, err) return fmt.Errorf("could not stop %s for volume dump: %w", stackName, err)
} }
@@ -695,6 +834,10 @@ func (m *Manager) DumpAppVolumesSafe(stackName string) error {
startErr := m.stackProvider.StartStack(stackName) startErr := m.stackProvider.StartStack(stackName)
if startErr != nil { if startErr != nil {
m.logger.Printf("[ERROR] [backup] Failed to restart %s after volume dump: %v", stackName, startErr) m.logger.Printf("[ERROR] [backup] Failed to restart %s after volume dump: %v", stackName, startErr)
} else {
// Cleared ONLY on a restart that succeeded. A failed restart keeps the marker so the next
// startup retries — the app really is still owed one.
m.appStop.End()
} }
// Surface both errors — callers must know if the app is left stopped // Surface both errors — callers must know if the app is left stopped
@@ -721,6 +864,12 @@ func (m *Manager) IsRunning() bool {
return m.running return m.running
} }
// AcquireRunningForTest / ReleaseRunningForTest occupy the single-flight from another package's
// test, so the "a run is already in flight" branch can be exercised without racing a real run.
// Test-only seam, in the same spirit as SetOffboxRunner; nothing in production calls them.
func (m *Manager) AcquireRunningForTest() error { return m.acquireRunning() }
func (m *Manager) ReleaseRunningForTest() { m.releaseRunning() }
// acquireRunning atomically sets the running flag. Returns error if already running. // acquireRunning atomically sets the running flag. Returns error if already running.
func (m *Manager) acquireRunning() error { func (m *Manager) acquireRunning() error {
m.mu.Lock() m.mu.Lock()
@@ -894,7 +1043,19 @@ func (m *Manager) RefreshCache(nextDBDump time.Time) {
// Phase 2: keep each app's recovery unit current with its definition. Idempotent // Phase 2: keep each app's recovery unit current with its definition. Idempotent
// (checksum-skip), so this periodic refresh only writes when the config actually changed, // (checksum-skip), so this periodic refresh only writes when the config actually changed,
// and ensures units exist shortly after startup without waiting for the daily DB dump. // and ensures units exist shortly after startup without waiting for the daily DB dump.
//
// R-182: this sweep gets its OWN digest scope. It has to, and the reason is the whole
// balance of this change. The per-app event is now record-only, so without a digest here a
// capture failure detected between runs would be recorded and NEVER notified — a new
// silence introduced while closing one. But this path can fire on every status poll, so its
// digest deliberately carries NO run id: the hub's ordinary 1-hour operator cooldown then
// applies, which caps it at one mail an hour exactly as before, while the mail now lists
// EVERY failing app instead of whichever one happened to be first.
func() {
defer m.beginRunSummary(runKindRefresh, "")()
defer m.emitRunSummary()
m.captureAllRecoveryUnits() m.captureAllRecoveryUnits()
}()
} }
// Fill in dynamic fields under lock. // Fill in dynamic fields under lock.
@@ -0,0 +1,305 @@
package backup
import (
"crypto/sha256"
"encoding/hex"
"errors"
"io"
"log"
"os"
"path/filepath"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/fillwatch"
)
// R-165 / decision B2 — the capture floor that replaces the `mp1` bulkhead.
//
// Before the merge, the 20 G backup partition kept a runaway capture from reaching
// `/var/lib/docker`, because it was a different filesystem. After the merge it is the same one, and a
// full Docker data-root is a stopped box. These pin the replacement.
// floorProvider lists stacks and always resolves recovery info — the floor must refuse BEFORE any of
// that is consulted, so a capture that gets as far as GetStackRecoveryInfo has already lost.
type floorProvider struct {
stacks []string
dir string
infoHits []string // records every app whose recovery info was read = a capture that was ATTEMPTED
}
func (p *floorProvider) GetStackComposePath(string) (string, bool) { return "", false }
func (p *floorProvider) ListDeployedStacks() []StackSummary {
out := make([]StackSummary, 0, len(p.stacks))
for _, s := range p.stacks {
out = append(out, StackSummary{Name: s})
}
return out
}
func (p *floorProvider) GetStackHDDMounts(string) []string { return nil }
func (p *floorProvider) GetStackHDDPath(string) string { return "" }
func (p *floorProvider) GetImportRoot() string { return "" }
func (p *floorProvider) GetDockerVolumes(string) []string { return nil }
func (p *floorProvider) StopStack(string) error { return nil }
func (p *floorProvider) StartStack(string) error { return nil }
func (p *floorProvider) RefreshAndIsRunning(string) bool { return true }
func (p *floorProvider) GetStackRecoveryInfo(name string) (RecoveryInfo, bool) {
p.infoHits = append(p.infoHits, name)
return RecoveryInfo{StackDir: filepath.Join(p.dir, "stacks", name)}, true
}
func (p *floorProvider) RecoverStackSecrets(string, []string) map[string]string { return nil }
func (p *floorProvider) RecreateStackDefinitionFromUnit(string, string, map[string]string) error {
return nil
}
func (p *floorProvider) StartStackServices(string, []string) error { return nil }
func (p *floorProvider) GetStackClassifiedBinds(string) ([]ClassifiedBind, bool) {
return nil, false
}
type floorHarness struct {
m *Manager
prov *floorProvider
events []unitEvent
usage map[string]*UnitSpace
dir string
}
// newFloorHarness injects the usage read, so the filesystem's occupancy is a test input rather than
// something the test has to manufacture on a real disk.
func newFloorHarness(t *testing.T, stacks ...string) *floorHarness {
t.Helper()
dir := t.TempDir()
h := &floorHarness{
prov: &floorProvider{stacks: stacks, dir: dir},
usage: map[string]*UnitSpace{},
dir: dir,
}
h.m = &Manager{
logger: log.New(io.Discard, "", 0),
systemDataPath: dir,
stackProvider: h.prov,
unitSpaceFn: func(name string) *UnitSpace { return h.usage[name] },
}
h.m.SetUnitNotify(func(name string, err error, u *UnitSpace) {
h.events = append(h.events, unitEvent{app: name, err: err.Error(), usage: u})
})
return h
}
func (h *floorHarness) setSpace(app string, usedPct, availGB float64) {
h.usage[app] = &UnitSpace{
Path: h.dir, UsedPercent: usedPct, AvailGB: availGB,
TotalGB: 100, UsedGB: usedPct,
}
}
// --- Scenario D — the floor refuses, per app, and says so ----------------------------------------
func TestFloor_RefusesTheAppAndLeavesItsPreviousUnitByteIdentical(t *testing.T) {
h := newFloorHarness(t, "homebox", "immich", "nextcloud")
h.setSpace("homebox", 40, 60)
h.setSpace("immich", 98, 0.4) // below the floor on BOTH terms
h.setSpace("nextcloud", 40, 60)
// A previous unit exists for the app about to be refused. Checksum it before and after.
unitDir := filepath.Join(h.dir, "felhom-data", "backups", "primary", "immich", "compose")
if err := os.MkdirAll(unitDir, 0o755); err != nil {
t.Fatal(err)
}
prev := filepath.Join(unitDir, "app.yaml")
if err := os.WriteFile(prev, []byte("deployed: true\nenv:\n A: previous-good-value\n"), 0o600); err != nil {
t.Fatal(err)
}
before := checksumFile(t, prev)
h.m.captureAllRecoveryUnits()
// The refused app must NOT have been attempted at all — the floor is checked BEFORE any write.
for _, hit := range h.prov.infoHits {
if hit == "immich" {
t.Fatal("the refused app's recovery info was read — the capture was ATTEMPTED rather than " +
"refused up front, so a write could have started and failed partway")
}
}
if after := checksumFile(t, prev); after != before {
t.Fatalf("the previous unit changed (%s → %s) — a refused capture must leave the last good "+
"copy byte-identical", before, after)
}
if _, err := os.Stat(prev); err != nil {
t.Fatalf("the previous unit is gone: %v — the floor REFUSES, it never deletes", err)
}
// Exactly one alert, for the refused app, carrying the space figures.
if len(h.events) != 1 {
t.Fatalf("got %d alerts, want exactly 1: %+v", len(h.events), h.events)
}
e := h.events[0]
if e.app != "immich" {
t.Fatalf("alert names %q, want immich", e.app)
}
if e.usage == nil || e.usage.AvailGB != 0.4 {
t.Fatalf("the alert carries no/incorrect space figures: %+v", e.usage)
}
if !strings.Contains(e.err, "reserve") {
t.Fatalf("the alert message %q does not say it was a reserve refusal — an operator would read "+
"it as a broken capture rather than a deliberate hold", e.err)
}
// The other two must have been captured normally — one app's refusal must not silence its siblings.
got := strings.Join(h.prov.infoHits, ",")
if !strings.Contains(got, "homebox") || !strings.Contains(got, "nextcloud") {
t.Fatalf("attempted=%v — the loop did not continue past the refusal", h.prov.infoHits)
}
}
// Nothing may be deleted to make room, under any threshold. Nothing on this filesystem is
// generational, so "the oldest" is always a DIFFERENT app's only local copy.
func TestFloor_NeverDeletesAnotherAppsUnit(t *testing.T) {
h := newFloorHarness(t, "immich", "nextcloud")
h.setSpace("immich", 99, 0.1)
h.setSpace("nextcloud", 99, 0.1)
other := filepath.Join(h.dir, "felhom-data", "backups", "primary", "nextcloud")
if err := os.MkdirAll(other, 0o755); err != nil {
t.Fatal(err)
}
keep := filepath.Join(other, "manifest.json")
if err := os.WriteFile(keep, []byte(`{"app_name":"nextcloud"}`), 0o644); err != nil {
t.Fatal(err)
}
before := checksumFile(t, keep)
h.m.captureAllRecoveryUnits()
if _, err := os.Stat(keep); err != nil {
t.Fatalf("another app's unit was DELETED to make room: %v — nothing here is generational, so "+
"pruning could only destroy an app's only local copy", err)
}
if after := checksumFile(t, keep); after != before {
t.Fatal("another app's unit was modified while the filesystem was under the floor")
}
}
// --- Scenario E — the floor is not a wall by another name ----------------------------------------
// The floor is about the FILESYSTEM's remaining headroom, never the unit's size. A per-unit cap would
// be R-163 rebuilt inside one volume.
func TestFloor_LargeUnitWithAmpleSpaceIsCaptured(t *testing.T) {
h := newFloorHarness(t, "immich")
// A huge app on a huge, mostly-empty filesystem: 40% used, 600 GB free.
h.usage["immich"] = &UnitSpace{Path: h.dir, UsedPercent: 40, AvailGB: 600, TotalGB: 1000, UsedGB: 400}
h.m.captureAllRecoveryUnits()
if len(h.events) != 0 {
t.Fatalf("a capture was refused on a filesystem with 600 GB free (%+v) — the floor has become "+
"a per-unit size cap, which is exactly the ceiling R-165 removed", h.events)
}
if len(h.prov.infoHits) != 1 || h.prov.infoHits[0] != "immich" {
t.Fatalf("attempted=%v, want [immich] — the capture was not even tried", h.prov.infoHits)
}
}
// The old 20 G ceiling must not survive anywhere: a unit far larger than the retired partition is
// captured when the filesystem has room.
func TestFloor_TheOld20GCeilingIsGone(t *testing.T) {
h := newFloorHarness(t, "immich")
// 180 GB free, and the app's own data is 120 GB — SIX TIMES the retired 20 G area. The figure is
// deliberately far above 20 so that a literal `UsedGB > 20` cap cannot survive this test: a
// fixture sitting exactly on the old boundary would pass under the very shape it forbids.
h.usage["immich"] = &UnitSpace{Path: h.dir, UsedPercent: 40, AvailGB: 180, TotalGB: 300, UsedGB: 120}
h.m.captureAllRecoveryUnits()
if len(h.events) != 0 {
t.Fatalf("refused with 180 GB free: %+v — a fixed per-area limit survives somewhere", h.events)
}
}
// --- Group E — the floor sits BELOW the critical warning band -------------------------------------
// A floor that fires before its own warning is a silent failure wearing a threshold: the customer
// would get a refusal with no prior notice that anything was wrong. The customer's `disk_critical`
// must always come first.
func TestFloorSitsBelowTheCriticalWarningBand(t *testing.T) {
if FloorUsedPercent <= fillwatch.CritUsedPercent {
t.Fatalf("FloorUsedPercent (%.1f) must be strictly ABOVE fillwatch.CritUsedPercent (%.1f) — "+
"otherwise a capture can be refused before the customer was ever warned that the disk was "+
"filling, which is a silent failure wearing a threshold",
FloorUsedPercent, fillwatch.CritUsedPercent)
}
if FloorFreeGiB >= fillwatch.CritFreeGiB {
t.Fatalf("FloorFreeGiB (%.1f) must be strictly BELOW fillwatch.CritFreeGiB (%.1f) — the "+
"free-byte term needs the same ordering as the percentage term, or the free-byte path "+
"refuses before it warns", FloorFreeGiB, fillwatch.CritFreeGiB)
}
// And below the WARNING band too, transitively — stated explicitly so the chain is readable.
if FloorUsedPercent <= fillwatch.WarnUsedPercent || FloorFreeGiB >= fillwatch.WarnFreeGiB {
t.Fatal("the floor is not beyond the warning band — the customer must be warned, then warned " +
"critically, and only then can a capture be refused")
}
// Both terms must be able to refuse INDEPENDENTLY — that is why there are two. estGiB=0 is the
// history-less case, which exercises the headroom term alone.
if _, r := (&Manager{}).floorVerdict(&UnitSpace{UsedPercent: 50, AvailGB: 0.5}, 0); r != floorHeadroom {
t.Fatal("a filesystem with 0.5 GiB free at only 50% used was NOT refused — the free-byte term " +
"does not trip on its own, so a very large volume can run out without the floor engaging")
}
if _, r := (&Manager{}).floorVerdict(&UnitSpace{UsedPercent: 98, AvailGB: 40}, 0); r != floorHeadroom {
t.Fatal("a filesystem 98% used was NOT refused — the percentage term does not trip on its own")
}
if _, r := (&Manager{}).floorVerdict(&UnitSpace{UsedPercent: 50, AvailGB: 50}, 0); r != floorAdmit {
t.Fatal("a healthy filesystem was refused")
}
}
// --- §8.4 — a nil usage read neither refuses nor warns --------------------------------------------
func TestFloor_UnreadableFilesystemNeitherRefusesNorWarns(t *testing.T) {
h := newFloorHarness(t, "immich")
// No entry → the injected reader returns nil, which is what system.GetDiskUsage does on error.
h.m.captureAllRecoveryUnits()
if len(h.events) != 0 {
t.Fatalf("an UNREADABLE filesystem produced %d alert(s): %+v — an absent, unmounted or "+
"unreadable filesystem is the drive gate's business and already has its own alert; "+
"refusing here would block every capture on a box whose drive merely blipped", len(h.events), h.events)
}
if len(h.prov.infoHits) != 1 {
t.Fatalf("the capture was not attempted on an unreadable read (attempted=%v) — a nil reading "+
"must not refuse", h.prov.infoHits)
}
}
// ErrCaptureFloor must be matchable, so a caller can tell a deliberate refusal from a broken capture.
func TestErrCaptureFloor_IsMatchable(t *testing.T) {
h := newFloorHarness(t, "immich")
h.setSpace("immich", 99, 0.2)
h.m.captureAllRecoveryUnits()
if len(h.events) != 1 {
t.Fatalf("want 1 event, got %d", len(h.events))
}
// The seam hands a string, so assert on the sentinel's own text being present and distinct.
if !errors.Is(errWrapForTest(), ErrCaptureFloor) {
t.Fatal("ErrCaptureFloor does not survive wrapping")
}
if !strings.Contains(h.events[0].err, "Refused") && !strings.Contains(h.events[0].err, "refused") {
t.Fatalf("the alert %q does not identify itself as a refusal", h.events[0].err)
}
}
func errWrapForTest() error { return errors.Join(ErrCaptureFloor, errors.New("ctx")) }
func checksumFile(t *testing.T, path string) string {
t.Helper()
f, err := os.Open(path)
if err != nil {
t.Fatal(err)
}
defer f.Close()
h := sha256.New()
if _, err := io.Copy(h, f); err != nil {
t.Fatal(err)
}
return hex.EncodeToString(h.Sum(nil))
}
+626 -12
View File
@@ -54,6 +54,14 @@ func (m *Manager) SetOffboxNotify(fn func(dur time.Duration, snapshots int, err
m.offboxNotify = fn m.offboxNotify = fn
} }
// SetOffboxGapNotify wires the R-203 operator signal: a run that completed but could NOT capture a
// directory an app declares MANDATORY. Distinct from offboxNotify, which fires only on a hard run
// failure — a coverage gap is not a failed run, and until v0.197.0 it reached nobody at all.
// gaps is app → the relative paths that were missed. nil → no signal (pre-R-203 behaviour).
func (m *Manager) SetOffboxGapNotify(fn func(gaps map[string][]string)) {
m.offboxGapNotify = fn
}
// SetOffboxOrphanEvent wires the offsite-repo continuity event push (main.go → notifier). // SetOffboxOrphanEvent wires the offsite-repo continuity event push (main.go → notifier).
func (m *Manager) SetOffboxOrphanEvent(fn func(eventType, renamedTo string)) { func (m *Manager) SetOffboxOrphanEvent(fn func(eventType, renamedTo string)) {
m.offboxOrphanEvent = fn m.offboxOrphanEvent = fn
@@ -69,6 +77,16 @@ func (m *Manager) SetOffboxSSH(fn func(ctx context.Context, host, user string, p
// the orphan card instead of the raw restic error. // the orphan card instead of the raw restic error.
var ErrOffboxOrphaned = fmt.Errorf("offbox repo orphaned: exists but keyed under a previous, no-longer-available passphrase") var ErrOffboxOrphaned = fmt.Errorf("offbox repo orphaned: exists but keyed under a previous, no-longer-available passphrase")
// ErrOffboxRunInFlight is returned to the MANUAL caller only, when the single-flight dropped the
// request because a run was already going (R-234). It is not a failure of anything — the run in
// flight is doing the work — but it IS a request that did nothing, and the page must say so instead
// of showing the previous run's verdict under a „started" message.
// offboxWholeUnitGap is the pseudo-path used to report a WHOLE-unit gap through the mandatory-gap
// notification, so a skipped app and a skipped directory reach the operator in one vocabulary.
const offboxWholeUnitGap = "(a teljes alkalmazás — nincs helyi mentési egysége)"
var ErrOffboxRunInFlight = fmt.Errorf("an off-box backup is already running; this request did not start a new one")
// classifyResticProbe maps a `restic cat config` failure to a repo class. The signatures are the exact // classifyResticProbe maps a `restic cat config` failure to a repo class. The signatures are the exact
// restic stderr matched in the 2026-07-17 diagnosis + restic's no-repo message: // restic stderr matched in the 2026-07-17 diagnosis + restic's no-repo message:
// - "orphaned": repo present, wrong key ("wrong password or no key found") — the definitive signal // - "orphaned": repo present, wrong key ("wrong password or no key found") — the definitive signal
@@ -92,6 +110,126 @@ func classifyResticProbe(out []byte, err error) string {
} }
} }
// ── F-DIAG: four causes, four messages, and no secrets ──────────────────────────────────────────
//
// The offsite failure notification was a raw passthrough:
//
// "a NAS-ra mentés hibázott (<dur>): " + err.Error()
//
// One string for every cause, so an operator could not tell a full disk from a dead network without
// reading logs — AND a raw restic/ssh error carries the repo URL, which is built as
// `sftp:<user>@<host>:<path>` (offboxBaseArgs). That breaks this project's keys-not-values rule at the
// one place the text leaves the box.
//
// OffsiteFailureClass names the causes that are genuinely DISTINGUISHABLE where the error is produced.
// Nothing is invented: each maps to a signal the code already has.
type OffsiteFailureClass string
const (
OffsiteFailQuota OffsiteFailureClass = "quota" // the pre-run soft-quota gate refused (offbox.go quota state)
OffsiteFailOrphaned OffsiteFailureClass = "orphaned" // ErrOffboxOrphaned — repo keyed under a lost passphrase
OffsiteFailNoRepo OffsiteFailureClass = "no_repo" // classifyResticProbe "norepo" — nothing at the location
OffsiteFailNoUnits OffsiteFailureClass = "no_units" // apps toggled but no recovery unit found on any drive
OffsiteFailTransport OffsiteFailureClass = "transport" // network / SFTP auth / host key / timeout
OffsiteFailUnknown OffsiteFailureClass = "unknown" // genuinely unclassified — say so rather than guess
)
// offsiteRepoURLRe matches the `sftp:user@host:/path` repo reference restic echoes back in its errors.
// It is the BACKSTOP, not the primary defence — see sanitiseOffsiteErrorFor.
var offsiteRepoURLRe = regexp.MustCompile(`sftp:[^\s"']+`)
// sanitiseOffsiteErrorFor strips anything that could carry a secret or a customer-identifying location
// out of an error before it reaches a message, an event or a report.
//
// IT REDACTS THE KNOWN TARGET VALUES, not a guessed pattern. The first version of this function
// regex-matched `sftp:…` and `user@host` and looked complete; its own test caught it leaking on
// `ssh: connect to host <host> port 23: Connection refused`, which contains a BARE hostname in neither
// shape. Guessing at what a secret looks like fails exactly where it matters — the target's host, user
// and repo path are known here, so they are removed literally and the regex stays only as a backstop
// for forms built before the target is loaded.
//
// Whole-token replacement, not masking: a partially-masked host still identifies the customer, and
// "it looked masked" is how a leak survives review.
func sanitiseOffsiteErrorFor(t *settings.OffboxTarget, err error) string {
if err == nil {
return ""
}
out := offsiteRepoURLRe.ReplaceAllString(err.Error(), "<repo>")
if t != nil {
// Longest first, so the repo path is not half-eaten by the host replacement.
for _, v := range []string{t.RepoPath, t.Host, t.User} {
if len(strings.TrimSpace(v)) >= 3 {
out = strings.ReplaceAll(out, v, "<repo>")
}
}
}
if len(out) > 300 {
out = out[:300] + "…"
}
return out
}
// ClassifyOffsiteFailure maps a run error to its cause.
//
// Order matters: the explicit sentinels first, then the text signatures. A cause that cannot be told
// apart here returns OffsiteFailUnknown rather than being folded into a neighbour — inventing a
// precision the code does not have is how a confident-but-wrong diagnosis ships.
func ClassifyOffsiteFailure(err error) OffsiteFailureClass {
if err == nil {
return ""
}
if errors.Is(err, ErrOffboxOrphaned) {
return OffsiteFailOrphaned
}
s := strings.ToLower(err.Error())
switch {
case strings.Contains(s, "tárhelykeretet"):
return OffsiteFailQuota
case strings.Contains(s, "produced no snapshots"):
return OffsiteFailNoUnits
case strings.Contains(s, "unable to open config file"),
strings.Contains(s, "is there a repository at the following location"):
return OffsiteFailNoRepo
case strings.Contains(s, "connection refused"), strings.Contains(s, "connection reset"),
strings.Contains(s, "no route to host"), strings.Contains(s, "i/o timeout"),
strings.Contains(s, "timed out"), strings.Contains(s, "permission denied"),
strings.Contains(s, "host key"), strings.Contains(s, "handshake"),
strings.Contains(s, "could not resolve"), strings.Contains(s, "network is unreachable"):
return OffsiteFailTransport
default:
return OffsiteFailUnknown
}
}
// OffsiteFailureMessage returns the operator-facing Hungarian message for a run failure: a distinct
// cause line plus the SANITISED detail. The detail is kept because an operator needs something to act
// on; it is sanitised because this text leaves the box.
//
// A method, not a function, so it can reach the target and redact its ACTUAL host/user/path rather
// than pattern-matching at what those might look like.
func (m *Manager) OffsiteFailureMessage(err error, dur time.Duration) string {
var t *settings.OffboxTarget
if m != nil && m.settings != nil {
t = m.settings.GetOffboxTarget()
}
return offsiteFailureMessage(t, err, dur)
}
func offsiteFailureMessage(t *settings.OffboxTarget, err error, dur time.Duration) string {
head := map[OffsiteFailureClass]string{
OffsiteFailQuota: "A távoli mentés nem fért el a tárhelykereten belül",
OffsiteFailOrphaned: "A távoli tárhely egy korábbi, már nem elérhető kulccsal készült",
OffsiteFailNoRepo: "A távoli tárhelyen nincs mentési adattár",
OffsiteFailNoUnits: "Nem volt mit menteni: egyetlen kijelölt alkalmazásnak sem található mentése",
OffsiteFailTransport: "A távoli tárhely nem érhető el (hálózat vagy bejelentkezés)",
OffsiteFailUnknown: "A távoli mentés ismeretlen okból nem sikerült",
}[ClassifyOffsiteFailure(err)]
if head == "" {
head = "A távoli mentés nem sikerült"
}
return fmt.Sprintf("%s (%s): %s", head, dur.Round(time.Second), sanitiseOffsiteErrorFor(t, err))
}
func defaultOffboxSSH(ctx context.Context, host, user string, port int, keyPath, knownHosts, remoteCmd string) ([]byte, error) { func defaultOffboxSSH(ctx context.Context, host, user string, port int, keyPath, knownHosts, remoteCmd string) ([]byte, error) {
if port == 0 { if port == 0 {
port = 22 port = 22
@@ -206,7 +344,21 @@ func (m *Manager) ResetOrphanedRepo(ctx context.Context) error {
} }
t := m.settings.GetOffboxTarget() t := m.settings.GetOffboxTarget()
base, env := m.offboxBaseArgs(t) base, env := m.offboxBaseArgs(t)
return m.resetOrphanedRepo(ctx, base, env, "operator-confirmed (claimed)") if err := m.resetOrphanedRepo(ctx, base, env, "operator-confirmed (claimed)"); err != nil {
return err
}
// R-241: THE COUNTDOWN STARTS HERE AND NOT IN THE SHARED HELPER, deliberately. The helper is also
// the UNCLAIMED auto-reset path (Scenario B in ensureOffboxRepo), where nobody decided anything —
// an as-delivered box tidying a stranger's leftover store must not put a customer's 14-day
// deletion clock on it. Only the confirmed, claimed choice is a decision.
//
// The path is read back from OrphanedRenamedTo, which the helper has just written.
if cur := m.settings.GetOffboxTarget(); cur != nil && cur.OrphanedRenamedTo != "" {
m.startAbandonCountdown(cur.OrphanedRenamedTo)
} else {
m.logger.Printf("[WARN] [offbox] the reset succeeded but no set-aside path was recorded — no countdown started; the old history stays indefinitely")
}
return nil
} }
// shellQuote single-quotes a path for the remote shell (our repo paths have no single quotes). // shellQuote single-quotes a path for the remote shell (our repo paths have no single quotes).
@@ -243,10 +395,46 @@ func (m *Manager) offboxKeyPath() string { return filepath.Join(m.offboxDir()
func (m *Manager) offboxPwPath() string { return filepath.Join(m.offboxDir(), "repo_password") } func (m *Manager) offboxPwPath() string { return filepath.Join(m.offboxDir(), "repo_password") }
func (m *Manager) offboxKnownHosts() string { return filepath.Join(m.offboxDir(), "known_hosts") } func (m *Manager) offboxKnownHosts() string { return filepath.Join(m.offboxDir(), "known_hosts") }
// ErrOffboxSealedPackageHeld is the R-241 mint refusal: this box has no repository password and the
// hub is holding a sealed recovery package for it, so minting one would write a key the package does
// not cover — orphaning the very history the customer's recovery code protects.
//
// It is a SENTINEL, not a failure. `ApplyOffsiteTarget` catches it and still configures the transport
// (SSH key, known_hosts, host/user/path), because the transport is not the problem and having it is
// what lets the recovery screen bring the tier up the moment the key arrives (R-219). What it does
// NOT do is let the tier come up under a key nobody escrowed.
var ErrOffboxSealedPackageHeld = fmt.Errorf("offbox: the hub holds a sealed recovery package for this box — not minting a repository password over it")
// ErrOffboxSealedPackageHeld reports whether err is the mint refusal (errors.Is-friendly for callers
// that wrap it).
func IsOffboxSealedPackageHeld(err error) bool { return errors.Is(err, ErrOffboxSealedPackageHeld) }
// WriteOffboxSecrets persists the SSH private key + (auto-generated if empty) repo password + the pinned // WriteOffboxSecrets persists the SSH private key + (auto-generated if empty) repo password + the pinned
// known-host line as 0600/0644 files in the data dir. The key is provided out-of-band by the operator // known-host line as 0600/0644 files in the data dir. The key is provided out-of-band by the operator
// (UI), never logged. Returns the repo password so the caller need not read the file. Idempotent: an empty // (UI), never logged. Idempotent: an empty sshKey/knownHosts leaves the existing file untouched (a
// sshKey/knownHosts leaves the existing file untouched (a re-save of just the target shouldn't wipe keys). // re-save of just the target shouldn't wipe keys).
//
// ⚠ R-241 (v0.206.0) — IT NO LONGER MINTS OVER A SEALED PACKAGE, AND THAT IS THE WHOLE FIX.
//
// Until now the auto-generate branch consulted **one** input: does the file exist. Not the settings,
// not the hub's ACK — nothing about whether anything already depended on a different key. Its two
// neighbours in this very file, `OffsiteRecoveryOffer` (:1412) and `needsOffsiteCredential` (:1377),
// BOTH consult `GetHubEscrowIdentityPresent()`. The same fact was available on three paths and used
// on two.
//
// WHAT THAT COST, measured on the final walk (2026-08-06/07, SPIKE-r241-recovery-offer-2026-08-07):
// a rebuilt box's credential self-heal reached here at 03:18:06Z and minted `9b4a9a9d…` over a hub
// package sealing `30ef574f…`. The recovery screen then looked, found a key present and no orphan
// recorded, and correctly said there was nothing to recover. **The screen was telling the truth; the
// lie happened thirty minutes earlier, here.** And the flag was not merely available at that moment —
// it was the PRECONDITION of the chain that reached this function: the credential retry only runs
// while `needsOffsiteCredential` is true, which requires this exact flag, and the venue logged it at
// 02:48:03Z, six ticks before the mint.
//
// THE GUARD IS DELIBERATELY NARROW — see the Scenario B test. It fires ONLY when a package is held AND
// no password exists. A box the hub holds nothing for mints exactly as before, which is every
// first-time box in the fleet; widening this to "never mint" would leave a new customer unable to
// start, waiting for a package that will never exist.
func (m *Manager) WriteOffboxSecrets(sshKey, knownHosts string) error { func (m *Manager) WriteOffboxSecrets(sshKey, knownHosts string) error {
if err := os.MkdirAll(m.offboxDir(), 0o700); err != nil { if err := os.MkdirAll(m.offboxDir(), 0o700); err != nil {
return fmt.Errorf("offbox dir: %w", err) return fmt.Errorf("offbox dir: %w", err)
@@ -269,8 +457,15 @@ func (m *Manager) WriteOffboxSecrets(sshKey, knownHosts string) error {
return fmt.Errorf("offbox known_hosts: %w", err) return fmt.Errorf("offbox known_hosts: %w", err)
} }
} }
// Auto-generate the repo password once (0600), never log it. // Auto-generate the repo password once (0600), never log it — UNLESS the hub is holding a sealed
// package for us (R-241). The transport files above are already written and that is deliberate.
if _, err := os.Stat(m.offboxPwPath()); os.IsNotExist(err) { if _, err := os.Stat(m.offboxPwPath()); os.IsNotExist(err) {
if m.sealedPackageHeld() {
m.logger.Printf("[WARN] [offbox] NOT minting a repository password: the hub holds a sealed recovery package for this box, " +
"and a fresh key would orphan the history that package protects (R-241). The transport is configured; " +
"the tier stays down until the customer's recovery code places the escrowed key.")
return ErrOffboxSealedPackageHeld
}
pw, gerr := generateOffboxPassword() pw, gerr := generateOffboxPassword()
if gerr != nil { if gerr != nil {
return gerr return gerr
@@ -282,6 +477,37 @@ func (m *Manager) WriteOffboxSecrets(sshKey, knownHosts string) error {
return nil return nil
} }
// sealedPackageHeld reports the ACK-cached fact that the hub is holding a sealed recovery package for
// this box. It is the SAME call `OffsiteRecoveryOffer` and `needsOffsiteCredential` already make —
// deliberately, so the three paths can never disagree about it. A nil settings store reads as "no
// package": the mint guard must never block a box whose settings could not be read, because that
// would turn a transient read failure into a tier that never comes up.
func (m *Manager) sealedPackageHeld() bool {
return m.settings != nil && m.settings.GetHubEscrowIdentityPresent()
}
// OffboxAwaitingRecoveryKey reports the R-241 holding state: a transport target exists, but no
// repository password does, because the hub holds a sealed package and the mint was refused.
//
// DERIVED, NOT STORED, and that is the §2.1 ruling applied to this field too: a stored flag would be a
// second copy of a fact the three inputs already carry, and a second copy is a thing that can drift.
// The moment a recovery places the escrowed key, this goes false on its own with nothing to clear.
//
// ⚠ `t.Enabled` IS LOAD-BEARING, and it was missing in the first draft — caught by the existing
// TestOffsiteDeclare_DisabledTargetIsNotStranded rather than by review. A customer who switched
// off-site OFF is not awaiting anything, and a box that declares a holding state for a tier nobody
// asked for is the R-215 shape (a screen about data the customer did not ask to protect). This is the
// SAME Scenario-E carve-out `needsOffsiteCredential` makes two functions above; the two must agree,
// and now do.
func (m *Manager) OffboxAwaitingRecoveryKey() bool {
t := m.settings.GetOffboxTarget()
if t == nil || !t.Enabled || !m.sealedPackageHeld() {
return false
}
_, hasPw := m.OffboxRepoPasswordHash()
return !hasPw
}
// generateOffboxPassword returns a 256-bit hex repo password. // generateOffboxPassword returns a 256-bit hex repo password.
func generateOffboxPassword() (string, error) { func generateOffboxPassword() (string, error) {
b := make([]byte, 32) b := make([]byte, 32)
@@ -351,9 +577,20 @@ var offboxRepoPwPattern = regexp.MustCompile(`^[0-9a-fA-F]{64}$`)
// the repo password to the agent for escrow — the SAME fork-4 enable path a manual config takes. `stage` is // the repo password to the agent for escrow — the SAME fork-4 enable path a manual config takes. `stage` is
// the agent escrow-stage push (nil skips it, e.g. when the agent is unreachable — the run gate still holds). // the agent escrow-stage push (nil skips it, e.g. when the agent is unreachable — the run gate still holds).
func (m *Manager) ApplyOffsiteTarget(ctx context.Context, tgt *settings.OffboxTarget, sshKeyPEM, knownHosts string, stage func(ctx context.Context, pw string) error) error { func (m *Manager) ApplyOffsiteTarget(ctx context.Context, tgt *settings.OffboxTarget, sshKeyPEM, knownHosts string, stage func(ctx context.Context, pw string) error) error {
// R-241: the mint refusal is a HOLDING state, not a failure. The transport files were written
// before the refusal, so we still record the target — `OffboxConfigured()` stays false because the
// password file is absent, which is what keeps runs gated, and the recovery screen can bring the
// tier up the instant the escrowed key is placed (R-219's synchronous tier-up).
//
// Returning the error here instead would leave `needsOffsiteCredential` true forever, so the hub
// would re-stage a credential the box had already consumed, on every cycle, for ever.
awaitingKey := false
if err := m.WriteOffboxSecrets(sshKeyPEM, knownHosts); err != nil { if err := m.WriteOffboxSecrets(sshKeyPEM, knownHosts); err != nil {
if !IsOffboxSealedPackageHeld(err) {
return fmt.Errorf("apply offsite secrets: %w", err) return fmt.Errorf("apply offsite secrets: %w", err)
} }
awaitingKey = true
}
// Re-apply (v0.109.1 live finding): the bridge rebuilds the target from the descriptor, but the // Re-apply (v0.109.1 live finding): the bridge rebuilds the target from the descriptor, but the
// EXISTING target's custody + runtime status must carry over — EscrowState tracks the REPO PASSWORD // EXISTING target's custody + runtime status must carry over — EscrowState tracks the REPO PASSWORD
// (preserved by WriteOffboxSecrets above, never rotated by this path), not the target coords; and the // (preserved by WriteOffboxSecrets above, never rotated by this path), not the target coords; and the
@@ -363,7 +600,12 @@ func (m *Manager) ApplyOffsiteTarget(ctx context.Context, tgt *settings.OffboxTa
tgt.EscrowState = cur.EscrowState tgt.EscrowState = cur.EscrowState
tgt.LastRun, tgt.LastStatus, tgt.LastError = cur.LastRun, cur.LastStatus, cur.LastError tgt.LastRun, tgt.LastStatus, tgt.LastError = cur.LastRun, cur.LastStatus, cur.LastError
tgt.LastDuration, tgt.LastWarning = cur.LastDuration, cur.LastWarning tgt.LastDuration, tgt.LastWarning = cur.LastDuration, cur.LastWarning
// R-100: carry the staleness anchor across a hub re-apply, for the same reason as the rest of
// this block — a re-apply is not a new tier. Dropping it would reset an established tier to
// "never succeeded" every time the hub re-pushes the descriptor.
tgt.LastSuccess = cur.LastSuccess
tgt.RepoSizeHuman, tgt.RepoSizeBytes, tgt.SnapshotCount = cur.RepoSizeHuman, cur.RepoSizeBytes, cur.SnapshotCount tgt.RepoSizeHuman, tgt.RepoSizeBytes, tgt.SnapshotCount = cur.RepoSizeHuman, cur.RepoSizeBytes, cur.SnapshotCount
tgt.StatsKnown = cur.StatsKnown // R-225: carry the KNOWN-ness with the numbers
} }
if tgt.EscrowState != "escrowed" { if tgt.EscrowState != "escrowed" {
tgt.EscrowState = "pending" tgt.EscrowState = "pending"
@@ -371,6 +613,14 @@ func (m *Manager) ApplyOffsiteTarget(ctx context.Context, tgt *settings.OffboxTa
if err := m.settings.SetOffboxTarget(tgt); err != nil { if err := m.settings.SetOffboxTarget(tgt); err != nil {
return fmt.Errorf("apply offsite target: %w", err) return fmt.Errorf("apply offsite target: %w", err)
} }
// R-241: nothing to stage — there is no repository password, by design. Say so once, plainly, and
// return without touching the escrow. `PushOffboxPasswordForEscrow` would fail on the absent file
// anyway; naming the situation beats a misleading "agent unreachable?" warning.
if awaitingKey {
m.logger.Printf("[INFO] [offbox] apply-offsite: transport configured for %s@%s:%s, tier HELD awaiting the escrowed key "+
"(the hub holds a sealed package; no key was minted — R-241)", tgt.User, tgt.Host, tgt.RepoPath)
return nil
}
if stage != nil { if stage != nil {
// Best-effort: the offbox is configured + pending regardless. A stage-push failure (agent momentarily // Best-effort: the offbox is configured + pending regardless. A stage-push failure (agent momentarily
// unreachable) is logged, not fatal — the escrow can be (re-)staged later (operator ceremony / re-enable). // unreachable) is logged, not fatal — the escrow can be (re-)staged later (operator ceremony / re-enable).
@@ -613,7 +863,19 @@ func (m *Manager) runOffboxBackup(ctx context.Context, withProgress bool) error
} }
if err := m.acquireRunning(); err != nil { if err := m.acquireRunning(); err != nil {
m.logger.Printf("[INFO] [offbox] skipped — another backup is running") m.logger.Printf("[INFO] [offbox] skipped — another backup is running")
return nil // single-flight: don't race; the next scheduled run retries // R-234 (the MEASURED cause). The nightly path is unchanged: returning nil is right for it —
// nobody asked, and the next scheduled run retries.
//
// The MANUAL path is a different question, and answering it the same way is what produced the
// 2026-08-06 sequence. The customer pressed „Távoli mentés most" and was told
// „A távoli mentés elindult"; the run was dropped here and returned nil; the card then showed
// the PREVIOUS run's „✓ Rendben", which they read as covering the app they had just selected.
// It did not — the restore refused for that app minutes later. A request that did nothing must
// not be reported as one that started, so the manual caller is told.
if withProgress {
return ErrOffboxRunInFlight
}
return nil
} }
defer m.releaseRunning() defer m.releaseRunning()
@@ -714,6 +976,10 @@ func (m *Manager) runOffboxBackup(ctx context.Context, withProgress bool) error
} }
if perr := m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) { if perr := m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
o.LastRun = time.Now().UTC().Format(time.RFC3339) o.LastRun = time.Now().UTC().Format(time.RFC3339)
// R-100: LastRun above records the ATTEMPT; this records the RESULT. The hub's staleness
// verdict counts from the anchor, never from the attempt. INVARIANT: a failed run neither
// advances nor clears it — pinned by TestOffboxAnchorAfterRun_* , not asserted in prose.
o.LastSuccess = offboxAnchorAfterRun(o.LastSuccess, o.LastRun, runErr)
o.LastDuration = dur.Round(time.Second).String() o.LastDuration = dur.Round(time.Second).String()
if errors.Is(runErr, ErrOffboxOrphaned) { if errors.Is(runErr, ErrOffboxOrphaned) {
// First-detection of the orphaned repo: RepoState (set by markOrphaned) drives the orphan // First-detection of the orphaned repo: RepoState (set by markOrphaned) drives the orphan
@@ -725,10 +991,57 @@ func (m *Manager) runOffboxBackup(ctx context.Context, withProgress bool) error
o.LastStatus = "error" o.LastStatus = "error"
o.LastError = runErr.Error() o.LastError = runErr.Error()
o.LastWarning = "" o.LastWarning = ""
} else {
// R-203 — THE VERDICT. A run that could not capture a directory the app declares MANDATORY
// is not a successful run. Until v0.197.0 it reported `ok` with a warning beside it, and a
// warning beside a success is read as a success: that is how calibre-web's declared book
// directory stayed out of every off-site snapshot on demo-hp while the card, the counters
// and the hub all said the backup worked.
//
// NOT "error": the rest of the run worked and the data that WAS captured is real. The
// snapshot count and the LastSuccess anchor are deliberately left to record it — half a
// backup is not no backup, and reporting it as none would be its own lie. `incomplete` is
// minted here because the existing vocabulary ("ok" | "error" | "running") has nothing that
// means "it ran, and this app is not fully protected".
// R-234 EXTENDS THE SAME RULE TO THE BIGGER CASE. Until v0.205.0 the paragraph above was
// applied to ONE of the two shapes it describes: an app missing a declared mandatory
// FOLDER made the run incomplete, while an app skipped ENTIRELY — no recovery unit, so
// nothing of it in the snapshot at all — still reported ok with a warning beside it. The
// smaller gap moved the verdict and the bigger one did not. Measured 2026-08-06: a run
// reported „✓ Rendben · 1 pillanatkép" and the restore then refused for the app the
// customer had just selected.
gaps := len(runResult.mandatoryGaps) > 0
unprotected := len(runResult.missingUnprotected) > 0
if gaps || unprotected {
o.LastStatus = "incomplete"
if m.offboxGapNotify != nil {
// Reuse, not mirror: the operator signal for "this run left an app less protected
// than the customer asked for" is the same signal. A skipped app is reported as a
// whole-unit gap so one notification shape covers both, and the recipient does not
// have to learn a second vocabulary for the worse case.
notify := runResult.mandatoryGaps
if unprotected {
if notify == nil {
notify = map[string][]string{}
} else {
cp := make(map[string][]string, len(notify)+len(runResult.missingUnprotected))
for k, v := range notify {
cp[k] = v
}
notify = cp
}
for _, a := range runResult.missingUnprotected {
notify[a] = append(notify[a], offboxWholeUnitGap)
}
}
m.offboxGapNotify(notify)
}
} else { } else {
o.LastStatus = "ok" o.LastStatus = "ok"
}
o.LastError = "" o.LastError = ""
o.SnapshotCount = snapshots o.SnapshotCount = snapshots
o.StatsKnown = true // R-225: measured, even if the answer is zero
o.EnlargedBlocked = blockedNames // replace each run (sorted); empty slice clears it o.EnlargedBlocked = blockedNames // replace each run (sorted); empty slice clears it
var warns []string var warns []string
// Zero-toggle honesty (take-two obs.): a configured target with NOTHING selected reports // Zero-toggle honesty (take-two obs.): a configured target with NOTHING selected reports
@@ -739,9 +1052,18 @@ func (m *Manager) runOffboxBackup(ctx context.Context, withProgress bool) error
if len(apps) == 0 && !runResult.sharesBackedUp { if len(apps) == 0 && !runResult.sharesBackedUp {
warns = append(warns, "Sikeres — nincs mentésre jelölt alkalmazás") warns = append(warns, "Sikeres — nincs mentésre jelölt alkalmazás")
} }
if len(missing) > 0 { // R-234 §7.4 — WHICH apps, WHY, and WHEN. The old sentence said only that N apps "had no
warns = append(warns, fmt.Sprintf("Figyelmeztetés: %d alkalmazásnak nincs elérhető mentése, ezek kimaradtak: %s", // available backup and were left out", which names a problem with no next step and reads
len(missing), strings.Join(missing, ", "))) // the same whether the customer must act or simply wait.
if len(runResult.missingUnprotected) > 0 {
warns = append(warns, fmt.Sprintf(
"Ezek az alkalmazások NEM kerültek be a távoli mentésbe, mert még nincs helyi mentési egységük: %s. A következő mentés általában már elkészíti — ha a második futás után is itt szerepelnek, szólj az üzemeltetőnek.",
strings.Join(runResult.missingUnprotected, ", ")))
}
if len(runResult.missingNotDeployed) > 0 {
warns = append(warns, fmt.Sprintf(
"Ezek az alkalmazások ki vannak jelölve távoli mentésre, de nincsenek telepítve, ezért nem menthetők: %s. Ha már nincs rájuk szükséged, vedd ki a kijelölésüket a Távoli mentés oldalon.",
strings.Join(runResult.missingNotDeployed, ", ")))
} }
// 3a: capture-gap warnings (structurally-refused / on-disk-missing mandatory paths, undeployed). // 3a: capture-gap warnings (structurally-refused / on-disk-missing mandatory paths, undeployed).
warns = append(warns, runResult.warns...) warns = append(warns, runResult.warns...)
@@ -899,6 +1221,43 @@ type offboxRunResult struct {
// the zero-toggle honesty notice honest: a box with no app toggled but shares in the cloud is NOT // the zero-toggle honesty notice honest: a box with no app toggled but shares in the cloud is NOT
// "nothing is covered". // "nothing is covered".
sharesBackedUp bool sharesBackedUp bool
// mandatoryGaps (R-203) is app → the relative paths of its declared MANDATORY data directories
// that could NOT be captured. It is the STRUCTURED form of the warnings above, and it is what
// decides the run's verdict: a run that dropped a mandatory directory is not a successful run.
mandatoryGaps map[string][]string
// missingUnprotected / missingNotDeployed (R-234) split `missing` by WHY, because only one of the
// two may move the verdict. See the classification comment at the skip site: an app the customer
// selected and that IS deployed but has no unit is unprotected and counts; an app that is no
// longer installed is named but does not, so a removed app cannot leave the box amber forever.
missingUnprotected []string
missingNotDeployed []string
}
// stackDeployed reports whether the stack is currently deployed on this box. Used only to classify a
// skip (R-234) — never to decide whether to back something up.
func (m *Manager) stackDeployed(stack string) bool {
if m.stackProvider == nil {
return false
}
for _, st := range m.stackProvider.ListDeployedStacks() {
if st.Name == stack {
return true
}
}
return false
}
// driveUnavailableFor reports whether the app's drive is disconnected or decommissioned — states that
// already have their own customer-facing signal, so a skip caused by them is not re-reported here.
func (m *Manager) driveUnavailableFor(stack string) bool {
if m.settings == nil {
return false
}
d := m.GetAppDrivePath(stack)
if d == "" {
return false
}
return m.settings.IsDisconnected(d) || m.settings.IsDecommissioned(d)
} }
// runOffboxInternal does the repo-ensure + per-app DISCOVER → capture-set → gate → multi-path backup + // runOffboxInternal does the repo-ensure + per-app DISCOVER → capture-set → gate → multi-path backup +
@@ -919,12 +1278,43 @@ func (m *Manager) runOffboxInternal(ctx context.Context, apps, base, env []strin
if !ok { if !ok {
m.logger.Printf("[WARN] [offbox] %s: no recovery unit found on any connected drive — skipping", stack) m.logger.Printf("[WARN] [offbox] %s: no recovery unit found on any connected drive — skipping", stack)
res.missing = append(res.missing, stack) res.missing = append(res.missing, stack)
// R-234 §7.2 — WHICH skips make the run not-successful. The list above is prose for the
// customer; this classification is what the VERDICT may consult, and the two are not the
// same question. Established by measurement on demo-hp 2026-08-06, not assumed:
//
// * DEPLOYED, no unit — the run's own pre-dump phase (captureAllRecoveryUnits) writes a
// unit for every deployed stack before the push, so this state does not normally
// survive a run. Reaching here means the capture was refused (the reserve) or failed.
// The app the customer selected is NOT protected: it COUNTS.
// * NOT DEPLOYED — nothing can protect an app that is not there, and the remedy is to
// deselect it. It is NAMED so the customer can act, but it does NOT count: a box left
// permanently amber over an app somebody removed is a status that stops being read,
// which is how this whole class of defect starts.
// * drive disconnected/decommissioned — has its own signal and its own card; not ours to
// re-report as a backup gap.
switch {
case !m.stackDeployed(stack):
res.missingNotDeployed = append(res.missingNotDeployed, stack)
case m.driveUnavailableFor(stack):
// counted as neither: the drive card is the honest surface for this one.
default:
res.missingUnprotected = append(res.missingUnprotected, stack)
}
continue continue
} }
// Task 3-core TierOffsite capture set: mandatory userdata paths added to the unit snapshot, // Task 3-core TierOffsite capture set: mandatory userdata paths added to the unit snapshot,
// plus loud warnings for structurally-refused / on-disk-missing mandatory paths (SP-3.4). // plus loud warnings for structurally-refused / on-disk-missing mandatory paths (SP-3.4).
extra, capWarns := m.offboxCaptureSet(stack) extra, capWarns, capGaps := m.offboxCaptureSet(stack)
res.warns = append(res.warns, capWarns...) res.warns = append(res.warns, capWarns...)
// R-203: the gaps are recorded STRUCTURALLY, not only as prose, because the run's verdict now
// depends on them. A warning standing beside a success is read as a success — which is exactly
// how a customer-declared mandatory directory stayed out of the snapshot while the run said ok.
if len(capGaps) > 0 {
if res.mandatoryGaps == nil {
res.mandatoryGaps = map[string][]string{}
}
res.mandatoryGaps[stack] = append(res.mandatoryGaps[stack], capGaps...)
}
// Pre-push enlargement gate (§9): if last-known repo raw-data bytes + the mandatory-set estimate // Pre-push enlargement gate (§9): if last-known repo raw-data bytes + the mandatory-set estimate
// would cross the soft quota, push UNIT-ONLY (protection never regresses) and record the block. // would cross the soft quota, push UNIT-ONLY (protection never regresses) and record the block.
if len(extra) > 0 && t != nil && t.QuotaGB > 0 { if len(extra) > 0 && t != nil && t.QuotaGB > 0 {
@@ -994,28 +1384,251 @@ func (m *Manager) runOffboxInternal(ctx context.Context, apps, base, env []strin
// multiplied by the retained-snapshot count (SP-1). The displayed size drops one-time after deploy. // multiplied by the retained-snapshot count (SP-1). The displayed size drops one-time after deploy.
const offboxGiB = int64(1) << 30 const offboxGiB = int64(1) << 30
// offboxAnchorAfterRun returns the last-SUCCESS anchor after a run that finished at `at` with
// `runErr`, given the anchor value `prev` from before the run. R-100.
//
// THE RULE THIS ENCODES: a timestamp recording an ATTEMPT is not evidence of a RESULT. `LastRun` is
// written unconditionally at the end of every run, failures included, so "how long since LastRun"
// answers "how long since we last TRIED" — and the hub's staleness verdict was asking exactly that of
// exactly that field, so a tier failing on every run read as perfectly fresh forever.
//
// Both directions matter and each is a different bug if got wrong:
// - a FAILURE must not ADVANCE it → otherwise the original defect survives;
// - a FAILURE must not CLEAR it → otherwise one bad night makes an established tier read as
// never-succeeded, which is the mirror-image over-correction (and on the hub, the newborn-box path).
//
// It is a function rather than two lines inside the status closure so the rule can be red-proofed
// directly; the first version of this fix modelled the rule in its own test and was therefore hollow.
func offboxAnchorAfterRun(prev, at string, runErr error) string {
if runErr != nil {
return prev // failures neither advance nor clear the anchor
}
return at
}
// OffboxReportStatus is the NON-SECRET offsite summary carried on the hub report (SLICE 4) — the input // OffboxReportStatus is the NON-SECRET offsite summary carried on the hub report (SLICE 4) — the input
// to the hub's OffsiteChecker (fill + staleness alerts). nil when no offbox target is configured. // to the hub's OffsiteChecker (fill + staleness alerts). nil when no offbox target is configured.
type OffboxReportStatus struct { type OffboxReportStatus struct {
Enabled bool `json:"enabled"` Enabled bool `json:"enabled"`
EscrowState string `json:"escrow_state"` EscrowState string `json:"escrow_state"`
// State (v0.199.0, R-204 item 4 / R-193) is a DECLARED condition — the box naming its own
// situation rather than the hub deducing it from a silence. Empty on every configured box, so a
// healthy report's JSON is byte-identical to v0.198.0's.
//
// WHY A DECLARATION AND NOT AN INFERENCE (the operator ruling, 2026-08-05). An ABSENT off-site
// object has FOUR meanings — never configured, mid-restart, a transient config read failure, and
// rebuilt-and-stranded — and the hub cannot tell them apart. The BOX can, from two local facts it
// holds with certainty. So it says so.
State string `json:"state,omitempty"`
// AbandonPurgeRequested (v0.206.0, R-241) — the customer's abandonment countdown has run out, the
// set-aside off-site history HAS been deleted, and the hub is asked to drop the sealed package
// that protected it so the two halves go together (Scenario F).
//
// It is a DECLARATION, on the same principle as State: the box knows it has deleted the store; the
// hub cannot see that and must not infer it. It keeps being sent until the ACK stops reporting a
// superseded package, so a lost request retries by itself rather than leaving the pair half-gone.
// Absent/false on every other box, so a healthy report is byte-identical to v0.205.0's.
AbandonPurgeRequested bool `json:"abandon_purge_requested,omitempty"`
LastRun string `json:"last_run,omitempty"` // RFC3339 LastRun string `json:"last_run,omitempty"` // RFC3339
LastStatus string `json:"last_status,omitempty"` // "ok" | "error" | "running" LastStatus string `json:"last_status,omitempty"` // "ok" | "incomplete" (R-203) | "error" | "running"
// LastSuccess (R-100) is the last run that SUCCEEDED — the hub's staleness anchor. Absent on a
// pre-v0.181.0 controller, which the hub must degrade on explicitly rather than by accident:
// treating absence as failure alarms every un-upgraded box, treating it as success keeps the bug.
LastSuccess string `json:"last_success,omitempty"` // RFC3339
SnapshotCount int `json:"snapshot_count"` SnapshotCount int `json:"snapshot_count"`
RepoSizeBytes int64 `json:"repo_size_bytes"` RepoSizeBytes int64 `json:"repo_size_bytes"`
QuotaGB int `json:"quota_gb"` QuotaGB int `json:"quota_gb"`
} }
// OffboxReportStatus returns the offsite summary for the hub report (nil = not configured; the hub's // OffsiteStateNeedsCredential is the ONE declared state (v0.199.0, R-204 item 4 / R-193): this box
// checker treats absence as "nothing to watch" — pre-v0.109 reports look the same). // has no off-site tier, holds no repository password, and the hub says it is keeping a sealed
// recovery package for it — i.e. it is a REBUILT box whose predecessor spent the one-time provider
// password, and it cannot configure its off-site tier without a credential it has no way to obtain.
// That was the last of the four manual interventions the 2026-08-04 drill needed.
const OffsiteStateNeedsCredential = "needs_credential"
// OffsiteStateAwaitingRecoveryKey (v0.206.0, R-241) is the declared HOLDING state: the transport is
// configured, but no repository password exists because the hub holds a sealed package and minting
// one would orphan the history it protects. The box is not stranded (it has its credential) and not
// healthy (it cannot run) — it is waiting for a person with a recovery code.
//
// WHY IT IS INERT TO EVERY EXISTING HUB READER, established from their code rather than assumed —
// the same discipline `OffsiteStateNeedsCredential`'s own note applies:
//
// - `offsiteheal` acts on EXACTLY ONE string, `needs_credential` ("Everything else … is a no-op"),
// so it will not re-stage a credential this box already has;
// - `monitor.OffsiteChecker.isStale` returns false unless `Enabled && EscrowState == "escrowed"`,
// and this object carries Enabled=false;
// - `monitor/offsite_delivery.go` keys on the delivery shape, which is `applied` here (the secret
// WAS consumed), and that branch is skipped;
// - an unknown `state` string is ignored by encoding/json on an older hub.
//
// So this needs NO hub change to be safe. It does mean a held box raises no alarm — which is R-243,
// filed and deliberately not widened here; the difference from R-241 is that this state is now
// VISIBLE to the customer instead of silent.
const OffsiteStateAwaitingRecoveryKey = "awaiting_recovery_key"
// needsOffsiteCredential is the stranded-rebuild predicate. BOTH facts are required and neither is
// sufficient on its own — this is the whole correctness of the feature:
//
// - the data area is FRESH (no repository password on disk). Alone, this is simply a box that never
// had off-site backups, and declaring on it would make every un-configured box in the fleet ask
// for a credential.
// - the HUB holds a sealed recovery package (the ACK's identity_blob_present, cached in settings).
// Alone, this is a healthy box that has run its ceremony.
//
// Only together do they mean "this box HAD an off-site tier, and no longer has what it needs to use
// it". A target that exists but is DISABLED is not stranded either — the customer switched it off —
// so the caller only consults this when there is no enabled target, and a non-nil disabled target
// short-circuits to false here.
//
// ⚠ THE DECLARATION STOPS WHEN THE TIER WORKS, NOT WHEN A KEY EXISTS (R-218, v0.201.0).
//
// This used to carry a third condition: hold a repository password ⇒ not stranded, stop declaring.
// It reads as a sound freshness test and it is the exact opposite on the one path that matters,
// because installing a repository password is the RECOVERY SCREEN'S WHOLE JOB. Measured live on
// 2026-08-05 (CAMPAIGN-11 Phase 1):
//
// 13:39:54 needs_credential the box asks
// 13:42:43 needs_credential the hub's 2-report debounce is satisfied
// 13:47:03 hub re-stages the credential — "the box re-consumes on its next cycle"
// 13:47:35 the customer's unlock succeeds and places the recovered key
// 13:53:42 (silence) …and never asks again
//
// Thirty-two seconds after the remedy fired, the customer's own success switched off the mechanism
// that would have delivered the coordinates for the key they had just recovered. The hub held an
// unconsumed credential the box had no reason to collect; the box held a correct key and nowhere to
// use it; the screen said "a few minutes"; nothing was ever going to happen. Two features, each
// correct alone, cancelling on the path both were built for.
//
// The two remaining conditions are the honest ones: no target at all, and the hub is keeping a sealed
// package for us. Both stay true exactly until the tier is configured — which is the moment the box
// genuinely no longer needs a credential — and `OffboxReportStatus` stops consulting this predicate
// the instant a target exists. **Scenario E is unaffected and is pinned by its own test**: a disabled
// target is non-nil and still short-circuits at the first line, so a box whose customer switched
// off-site off stays silent.
func (m *Manager) needsOffsiteCredential(t *settings.OffboxTarget) bool {
if t != nil {
return false // a target exists (merely disabled) — the customer's own choice, not a rebuild
}
if m.settings == nil || !m.settings.GetHubEscrowIdentityPresent() {
return false // the hub holds nothing for us: never had off-site backups
}
return true
}
// OffsiteRecoveryOffer reports whether the customer should be OFFERED the recovery screen (R-193):
// the hub is holding a sealed recovery package for this box, and this box cannot open what that
// package protects.
//
// TWO FACTS, BOTH REQUIRED — and this is the whole correctness of the screen:
//
// 1. **the hub holds a sealed package** (the ACK's identity_blob_present, cached in settings). Without
// it there is nothing to recover, and a box that never had off-site backups must never be greeted
// by a recovery screen for data it never had. Dropping this condition is the plausible wrong fix.
// 2. **this box cannot open the history that package protects** — see the two shapes below.
//
// ⚠ WHY SHAPE (b) EXISTS, recorded because the task specified only shape (a) and the difference is
// load-bearing. The literal reading of "the data area is fresh" is *no repository password on disk*,
// which is true of a rebuilt box — but only until it re-applies its off-site target, because
// `WriteOffboxSecrets` AUTO-GENERATES a repository password when none is present. That is precisely
// R-193's orphaning mechanism, and since hub v0.96.0's credential self-heal the re-apply happens by
// itself within ~1530 minutes. So shape (a) alone would make this screen appear only inside a
// half-hour window that closes on its own, and the customer who logs in the next morning — the actual
// customer — would never see it. Shape (b) is the state they are in: a repository password exists, but
// it is a NEW one and the inherited history cannot be opened with it, which the box has already
// measured and recorded as `RepoState == "orphaned"`.
//
// Shape (b) also happens to be the state the existing move-aside requires (`ResetOrphanedRepo` refuses
// unless orphaned), which is what lets "I do not want the old data" reach the shipped handler rather
// than needing a new one.
//
// Scenario B still holds exactly: a healthy box has its own password and is not orphaned; a box that
// never had off-site backups fails fact 1; an unclaimed box never reaches an authenticated page.
// ── SHAPE (c), v0.206.0, R-241 — THE DISCRIMINATOR THAT ANSWERS THE REAL QUESTION ───────────────
//
// Shapes (a) and (b) are both PROXIES for one question — *does the hub hold a package for a key other
// than the one I am using?* — and both have now been wrong, in opposite directions:
//
// - (a) "no repository password" went false the moment anything minted one. Before v0.206.0's mint
// guard that happened by itself, ~30 minutes after a rebuild, and the customer who logged in the
// next morning never saw the screen. That is R-241.
// - (b) "a run proved the repo will not open" is unreachable on exactly that box: the only producer
// of RepoState=="orphaned" is ensureOffboxRepo, which is downstream of the escrow gate in
// runOffboxBackup, and the escrow can never confirm while the hub's package covers a different
// key. Self-locking.
//
// (c) asks the question directly, from two facts the box already holds: the hash the hub's package
// covers (ACK-cached) and the hash of the key on disk. **This comparison was already computed on every
// ACK and thrown away** — see settings.HubEscrowKeySHA256.
//
// ⚠ §7.2 — WHAT A STALE OR ABSENT READING RESOLVES TO, decided deliberately rather than by default:
//
// - **A KNOWN DIFFERENCE OFFERS, however old the reading.** Age is not gated on. Both sides of the
// comparison are local; only the hub's half can be stale, and what the hub holds does not change
// without a ceremony THIS BOX runs — which refreshes the hash on the next ACK. Gating on age would
// add a second failure mode (a box offline from the hub silently stops offering) to fix a window
// that closes itself. `HubEscrowKeyCheckedAt` is persisted for diagnosis, not as a gate.
// - **AN ABSENT HASH FALLS BACK TO (a)/(b), it does not offer.** "" is what the hub sends for a
// legacy hash-less package — one that provably seals no repository password. There is nothing for
// (c) to compare, and offering on it would put a permanent screen in front of every legacy box.
// This is the one place where "not knowing" resolves to silence, and it does so because an empty
// hash is not an unknown: it is the hub positively saying the package covers no key.
//
// So: fail-closed (offer) on a known difference; fall back on a hash never learned. Pinned by
// TestR241_ScenarioD_* and TestR241_StaleComparison_*.
func (m *Manager) OffsiteRecoveryOffer() bool {
if m.settings == nil || !m.settings.GetHubEscrowIdentityPresent() {
return false // the hub holds nothing for us — nothing to recover
}
localHash, hasLocal := m.OffboxRepoPasswordHash()
if !hasLocal {
return true // (a) no repository password at all — the pristine rebuilt box
}
if hubHash, _ := m.settings.GetHubEscrowKeySHA256(); hubHash != "" && hubHash != localHash {
return true // (c) the hub's package covers a DIFFERENT key than the one we are using
}
return m.OffboxOrphaned() // (b) a password exists but the inherited history will not open under it
}
// OffboxReportStatus returns the offsite summary for the hub report.
//
// nil = nothing to say (not configured, and nothing to ask for) — the hub's checker treats absence as
// "nothing to watch"; pre-v0.109 reports look the same.
//
// v0.199.0: there is now ONE case where an UNCONFIGURED box still reports an object — the stranded
// rebuild, which DECLARES OffsiteStateNeedsCredential rather than leaving the hub to deduce it from a
// silence. An absent object has FOUR meanings (never configured / mid-restart / a transient config
// read failure / rebuilt-and-stranded) and the hub cannot tell them apart; the box can.
//
// WHY THE DECLARATION IS INERT TO EVERY EXISTING READER, established from their code rather than
// assumed: it carries Enabled=false and zero quota/size, and the hub's OffsiteChecker gates
// `isStale` on `!off.Enabled` (returns false) and `fillBand` on a zero quota/size (returns OK). So it
// raises no staleness and no fill alarm on a new hub OR an old one, and an unknown `state` string is
// ignored by encoding/json. The ONE reader that would have misread it is the store's
// `reportHasOffsite` ("presence == applied-on-the-box"), which hub v0.96.0 tightens to require
// enabled=true — provably a no-op for every report shape that exists today, because this function has
// never emitted a disabled object before.
func (m *Manager) OffboxReportStatus() *OffboxReportStatus { func (m *Manager) OffboxReportStatus() *OffboxReportStatus {
t := m.settings.GetOffboxTarget() t := m.settings.GetOffboxTarget()
// R-241: the HOLDING state is declared before the enabled/disabled split, because a held target IS
// enabled — the customer wants off-site backups; what is missing is the key. Reported with
// Enabled=false so every existing hub reader treats it exactly as the stranded declaration (see
// OffsiteStateAwaitingRecoveryKey), while the string names the difference for anything that looks.
if m.OffboxAwaitingRecoveryKey() {
return &OffboxReportStatus{Enabled: false, State: OffsiteStateAwaitingRecoveryKey, EscrowState: t.EscrowState}
}
if t == nil || !t.Enabled { if t == nil || !t.Enabled {
if m.needsOffsiteCredential(t) {
return &OffboxReportStatus{Enabled: false, State: OffsiteStateNeedsCredential}
}
return nil return nil
} }
return &OffboxReportStatus{ return &OffboxReportStatus{
Enabled: true, EscrowState: t.EscrowState, LastRun: t.LastRun, LastStatus: t.LastStatus, Enabled: true, EscrowState: t.EscrowState, LastRun: t.LastRun, LastStatus: t.LastStatus,
LastSuccess: t.LastSuccess,
SnapshotCount: t.SnapshotCount, RepoSizeBytes: t.RepoSizeBytes, QuotaGB: t.QuotaGB, SnapshotCount: t.SnapshotCount, RepoSizeBytes: t.RepoSizeBytes, QuotaGB: t.QuotaGB,
AbandonPurgeRequested: t.AbandonPurgeRequested, // R-241: declared until the hub drops the package
} }
} }
@@ -1103,6 +1716,7 @@ func (m *Manager) offboxRecordStats(ctx context.Context, base, env []string) int
_ = m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) { _ = m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
o.RepoSizeHuman = human o.RepoSizeHuman = human
o.RepoSizeBytes = st.TotalSize o.RepoSizeBytes = st.TotalSize
o.StatsKnown = true // R-225
}) })
} }
} }
+15 -1
View File
@@ -23,12 +23,26 @@ type offbox3aProvider struct {
hdd map[string]string hdd map[string]string
binds map[string][]ClassifiedBind binds map[string][]ClassifiedBind
has map[string]bool has map[string]bool
// deployed is OPT-IN and defaults to nil, so every existing fixture keeps ListDeployedStacks()
// returning nil and nothing about their behaviour moves. R-234's classification is the only
// thing that needs a real deployed set.
deployed map[string]bool
} }
func (p *offbox3aProvider) GetStackComposePath(string) (string, bool) { return "", false } func (p *offbox3aProvider) GetStackComposePath(string) (string, bool) { return "", false }
func (p *offbox3aProvider) ListDeployedStacks() []StackSummary { return nil } func (p *offbox3aProvider) ListDeployedStacks() []StackSummary {
if len(p.deployed) == 0 {
return nil
}
out := make([]StackSummary, 0, len(p.deployed))
for n := range p.deployed {
out = append(out, StackSummary{Name: n})
}
return out
}
func (p *offbox3aProvider) GetStackHDDMounts(string) []string { return nil } func (p *offbox3aProvider) GetStackHDDMounts(string) []string { return nil }
func (p *offbox3aProvider) GetStackHDDPath(n string) string { return p.hdd[n] } func (p *offbox3aProvider) GetStackHDDPath(n string) string { return p.hdd[n] }
func (p *offbox3aProvider) GetImportRoot() string { return "" } // R-75: no import binds in this fixture
func (p *offbox3aProvider) GetDockerVolumes(string) []string { return nil } func (p *offbox3aProvider) GetDockerVolumes(string) []string { return nil }
func (p *offbox3aProvider) StopStack(string) error { return nil } func (p *offbox3aProvider) StopStack(string) error { return nil }
func (p *offbox3aProvider) StartStack(string) error { return nil } func (p *offbox3aProvider) StartStack(string) error { return nil }
@@ -0,0 +1,299 @@
package backup
import (
"context"
"fmt"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// ABANDONMENT — deciding to give up the old off-site history is a finishable thing (R-241, v0.206.0).
//
// THE PROBLEM THIS SOLVES. `resetOrphanedRepo` renamed the remote store aside and touched neither the
// escrow nor the key, so the hub went on holding a sealed package for a key the box no longer used.
// Shape (c) compares those two, finds them different, and offers recovery — correctly, and for ever.
// A customer who has already said "I do not want the old data" would be asked again at every login.
//
// THE OPERATOR'S RULING (2026-08-07) is that the answer is NOT a "they decided" flag. A flag would
// leave the box in a state that is genuinely wrong (the hub holding a package for a key nobody uses)
// and paper over it. Instead the decision starts a **14-day countdown**, at the end of which the
// set-aside store and the sealed package that protects it are removed TOGETHER — after which there is
// nothing left to compare and nothing left to ask about. **Fix the state, do not remember that it is
// wrong.**
//
// THE GRACE IS REAL, NOT DECORATIVE. The recovery offer stays reachable for the whole window; that is
// the change-of-mind path (Scenario G). A grace period during which recovery is impossible would be
// theatre.
// abandonGraceDays is the countdown the operator set. Reminders fire at 5, 3 and 1 days (see
// AbandonRemindAtDays) — visible, reversible, and running out in public.
const abandonGraceDays = 14
// AbandonGraceDays is the exported grace, for the customer-facing copy. The confirmation screen must
// state the SAME number the countdown uses — a literal typed into prose is how a promise drifts away
// from the code that keeps it.
const AbandonGraceDays = abandonGraceDays
// AbandonRemindAtDays are the remaining-day marks at which the abandoning box reminds the customer.
// Descending, so the surface can pick the first one that has been reached.
var AbandonRemindAtDays = []int{5, 3, 1}
// abandonNow is the countdown's clock seam. Tests inject; nil → time.Now. It exists so the terminal
// step can be driven deterministically — §7.4 forbids shortening a live timer to watch it fire,
// because that is how an irreversible step gets tested once and regretted once.
func (m *Manager) abandonNow() time.Time {
if m.offboxNow != nil {
return m.offboxNow()
}
return time.Now()
}
// SetOffboxClock injects the abandonment clock (tests only).
func (m *Manager) SetOffboxClock(fn func() time.Time) { m.offboxNow = fn }
// startAbandonCountdown records the decision and the date the terminal step will run. Called by
// resetOrphanedRepo AFTER the move-aside has succeeded — a countdown started before the store has
// actually moved would count down to deleting a path that does not exist.
func (m *Manager) startAbandonCountdown(setAsidePath string) {
now := m.abandonNow().UTC()
due := now.AddDate(0, 0, abandonGraceDays)
// R-302: pin the hub's escrow key fingerprint HERE, at the decision — the one moment it is a fact
// rather than something inferred later from an adjacent value. From now on the banner asks exactly
// one question, "is the hub still holding that same package?", instead of guessing which key is
// which. Written once and never refreshed: a field re-read at render answers a different question
// and would silently restore the defect this replaces.
pinned := ""
if m.settings != nil {
pinned, _ = m.settings.GetHubEscrowKeySHA256()
}
if err := m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
o.AbandonStartedAt = now.Format(time.RFC3339)
o.AbandonAt = due.Format(time.RFC3339)
o.AbandonRepoPath = setAsidePath
o.AbandonPurgeRequested = false
o.AbandonPinnedEscrowKeySHA256 = pinned
}); err != nil {
m.logger.Printf("[WARN] [offbox] could not record the abandonment countdown: %v", err)
return
}
m.logger.Printf("[INFO] [offbox] abandonment countdown started: the set-aside history at %s and the hub's sealed package "+
"are removed together on %s (%d days). The recovery screen stays reachable until then.",
setAsidePath, due.Format("2006-01-02"), abandonGraceDays)
}
// AbandonState is the surface's read model. Zero value = nothing in progress.
type AbandonState struct {
Active bool // a countdown is running
StartedAt time.Time //
DueAt time.Time // when the terminal step runs
DaysLeft int // ceiling, so "0 days left" only ever means "today"
RepoPath string // the set-aside store awaiting deletion
PurgeRequested bool // the store is gone; awaiting the hub to drop the sealed package
// RetrievalStillOffered (R-302) — may the banner still say the set-aside copies can be retrieved
// with the recovery code? TRUE only while the hub is holding the SAME sealed package it held when
// the customer decided. Derived here, once, so the banner and anything else asking cannot disagree.
//
// FALSE covers: the package was replaced after the decision (a fresh escrow ceremony — the act that
// cost both demo boxes their history); the hub reports an empty hash (a legacy package sealing no
// repository password); and a countdown started before R-302, which carries no pin. All three are
// "we cannot see that this is still true", and all three must read as such rather than as a promise.
RetrievalStillOffered bool
}
// AbandonStatus reports the countdown for the UI and the report. It never mutates.
func (m *Manager) AbandonStatus() AbandonState {
t := m.settings.GetOffboxTarget()
if t == nil {
return AbandonState{}
}
st := AbandonState{RepoPath: t.AbandonRepoPath, PurgeRequested: t.AbandonPurgeRequested}
if t.AbandonAt == "" {
return st
}
due, err := time.Parse(time.RFC3339, t.AbandonAt)
if err != nil {
// A malformed stamp must not silently mean "never due" — that would strand the store for ever
// with a countdown the customer can see and nothing behind it.
m.logger.Printf("[WARN] [offbox] abandonment due-date is unparseable (%q) — treating the countdown as NOT running: %v", t.AbandonAt, err)
return st
}
st.Active, st.DueAt = true, due
if s, serr := time.Parse(time.RFC3339, t.AbandonStartedAt); serr == nil {
st.StartedAt = s
}
// R-302: the pinned fingerprint vs what the hub reports NOW. Both must be non-empty and equal.
// Empty on either side is "we could not see", never "they match" — the settings comment on
// HubEscrowKeySHA256 establishes that the hub sends "" for a package sealing no repo password.
if cur, _ := m.settings.GetHubEscrowKeySHA256(); cur != "" &&
t.AbandonPinnedEscrowKeySHA256 != "" && cur == t.AbandonPinnedEscrowKeySHA256 {
st.RetrievalStillOffered = true
}
// Ceiling: a countdown with 30 minutes left says "1 day", never "0". Zero is reserved for due.
remaining := due.Sub(m.abandonNow())
if remaining <= 0 {
st.DaysLeft = 0
} else {
st.DaysLeft = int((remaining + 24*time.Hour - time.Nanosecond) / (24 * time.Hour))
}
return st
}
// CancelAbandon stops a running countdown — the change-of-mind path (Scenario G). Called when a
// recovery succeeds: the customer has their code after all, and the history they were about to give
// up is exactly what the code opens.
//
// It clears the schedule but KEEPS AbandonRepoPath, so the set-aside store remains nameable on the
// backups page. Nothing has been deleted at this point by construction — the terminal step is the
// only thing that deletes, and it has not run.
func (m *Manager) CancelAbandon(reason string) {
t := m.settings.GetOffboxTarget()
if t == nil || (t.AbandonAt == "" && !t.AbandonPurgeRequested) {
return // nothing running — silent, so a healthy recovery does not log about a countdown
}
if err := m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
o.AbandonStartedAt, o.AbandonAt = "", ""
o.AbandonPurgeRequested = false
}); err != nil {
m.logger.Printf("[WARN] [offbox] could not cancel the abandonment countdown: %v", err)
return
}
m.logger.Printf("[INFO] [offbox] abandonment countdown CANCELLED (%s) — the set-aside history at %s is kept and nothing was deleted", reason, t.AbandonRepoPath)
}
// AbandonSweep is the daily terminal step. It is the ONLY thing in the product that deletes a
// customer's off-site history, and it does so on a date the customer was shown.
//
// ⚠ IT REMOVES BOTH HALVES OR NEITHER — Scenario F. The set-aside store and the sealed package that
// protects it are the two halves of one thing; removing only the store leaves the hub holding a
// package for a key that opens nothing, and removing only the package leaves ciphertext nobody can
// ever decrypt. Either is a state that asks a question nobody can answer.
//
// The two halves cannot be made atomic across two machines, so this is a two-phase commit with the
// STORE FIRST and a durable marker: delete the remote store, record AbandonPurgeRequested, and keep
// declaring it in the report until the hub's ACK stops reporting a superseded package. A crash
// between the two leaves the marker set and the next sweep re-declares — it never leaves the pair
// half-removed and silent.
//
// Returns (deleted, err). deleted=false with err=nil is the normal "nothing due" case.
func (m *Manager) AbandonSweep(ctx context.Context) (bool, error) {
st := m.AbandonStatus()
// Phase 2 outstanding: the store is gone, the hub has not confirmed. Re-declare and wait.
if st.PurgeRequested {
m.logger.Printf("[DEBUG] [offbox] abandonment: the set-aside store is deleted; awaiting the hub to drop the sealed package")
return false, nil
}
if !st.Active || st.DueAt.After(m.abandonNow()) {
return false, nil // not due — quiet by construction on every healthy box
}
t := m.settings.GetOffboxTarget()
if t == nil || t.AbandonRepoPath == "" {
m.logger.Printf("[WARN] [offbox] abandonment is due but no set-aside path is recorded — nothing deleted; clearing the countdown so it does not retry for ever")
m.CancelAbandon("no set-aside path recorded")
return false, fmt.Errorf("abandonment due with no recorded path")
}
port := t.Port
if port == 0 {
port = 22
}
m.logger.Printf("[WARN] [offbox] abandonment DUE — deleting the set-aside off-site history at %s (chosen by the customer on %s; this is irreversible)",
t.AbandonRepoPath, st.StartedAt.Format("2006-01-02"))
out, err := m.sshRunner()(ctx, t.Host, t.User, port, m.offboxKeyPath(), m.offboxKnownHosts(),
"rm -rf "+shellQuote(t.AbandonRepoPath))
if err != nil {
// NOT cleared: a transport failure must retry tomorrow, not silently abandon the abandonment.
m.logger.Printf("[ERROR] [offbox] abandonment: deleting the set-aside history failed — the countdown stays due and retries: %v: %s", err, truncate(out))
return false, fmt.Errorf("delete set-aside history: %w", err)
}
// Phase 1 done. Record it durably BEFORE anything else, so a crash here re-declares rather than
// forgetting that the store is already gone.
if uerr := m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
o.AbandonPurgeRequested = true
o.AbandonAt = "" // the schedule has fired; the marker now drives the rest
}); uerr != nil {
m.logger.Printf("[ERROR] [offbox] abandonment: the store was deleted but the marker could not be saved — the hub's package may outlive it: %v", uerr)
return true, uerr
}
m.logger.Printf("[INFO] [offbox] abandonment: set-aside history deleted; requesting the hub to drop the sealed package that protected it")
if m.offboxOrphanEvent != nil {
m.offboxOrphanEvent("offbox_abandon_completed", t.AbandonRepoPath)
}
return true, nil
}
// ClearAbandonPurgeIfConfirmed closes the two-phase commit: once the hub's ACK stops reporting a
// superseded package, both halves are gone and the abandonment is finished. Called from the ACK path.
//
// This is what makes §2.1 work without a "they decided" flag: afterwards the hub holds a package for
// the key the box is actually using (or none at all), shape (c) has nothing to compare, and the
// recovery offer falls silent on its own — because the state is right, not because something is
// remembering that it once was not.
func (m *Manager) ClearAbandonPurgeIfConfirmed(supersededPresent bool) {
t := m.settings.GetOffboxTarget()
if t == nil || !t.AbandonPurgeRequested || supersededPresent {
return
}
if err := m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
o.AbandonPurgeRequested = false
o.AbandonRepoPath = ""
o.AbandonStartedAt = ""
o.OrphanedRenamedTo = ""
}); err != nil {
m.logger.Printf("[WARN] [offbox] could not close out the abandonment: %v", err)
return
}
m.logger.Printf("[INFO] [offbox] abandonment COMPLETE — the set-aside history and the sealed package that protected it are both gone; nothing further to ask about")
}
// ── OPERATOR CONTROL (§7.5) ─────────────────────────────────────────────────────────────────────
//
// The automatic 30-day abandonment is deliberately NOT built (see R-245). What IS built is the path
// that actually happens: **the customer gets in touch.** Someone who cannot find their recovery code
// rings support, and support needs something to press — either "give them longer" or "stop it".
//
// Both live on the controller CLI rather than in the customer UI, deliberately: extending a deletion
// the customer asked for is an operator judgement, not a self-service button, and a customer who
// wants it stopped already has the self-service route — they recover with their code, which cancels
// it (Scenario G).
// ExtendAbandon pushes the terminal step out by `days` from NOW. Returns the new due date.
//
// It refuses when no countdown is running: extending nothing would print a reassuring date for a
// deletion that was never scheduled, which is the kind of comfort this project keeps removing.
func (m *Manager) ExtendAbandon(days int) (time.Time, error) {
if days <= 0 {
return time.Time{}, fmt.Errorf("the extension must be a positive number of days")
}
st := m.AbandonStatus()
if !st.Active {
if st.PurgeRequested {
return time.Time{}, fmt.Errorf("too late: the set-aside history has already been deleted and only the sealed package is still being removed")
}
return time.Time{}, fmt.Errorf("no abandonment countdown is running on this box — nothing to extend")
}
due := m.abandonNow().UTC().AddDate(0, 0, days)
if err := m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
o.AbandonAt = due.Format(time.RFC3339)
}); err != nil {
return time.Time{}, fmt.Errorf("record the extension: %w", err)
}
m.logger.Printf("[WARN] [offbox] abandonment EXTENDED by an operator: the set-aside history at %s is now deleted on %s (was %s)",
st.RepoPath, due.Format("2006-01-02"), st.DueAt.Format("2006-01-02"))
return due, nil
}
// StopAbandon cancels the countdown outright — the operator's version of Scenario G, for the
// customer who telephoned instead of finding their code. The set-aside history is kept and nothing
// is deleted; it is `CancelAbandon` with an operator's reason and a refusal when nothing is running,
// so an operator never gets a silent no-op they might read as success.
func (m *Manager) StopAbandon() error {
st := m.AbandonStatus()
if !st.Active {
if st.PurgeRequested {
return fmt.Errorf("too late: the set-aside history has already been deleted")
}
return fmt.Errorf("no abandonment countdown is running on this box — nothing to stop")
}
m.CancelAbandon("stopped by an operator")
return nil
}
@@ -0,0 +1,208 @@
package backup
import (
"context"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// ── R-302 — THE BANNER PROMISES ONLY WHAT THE BOX CAN STILL SEE IS TRUE ─────────────────────────
//
// The abandon banner said "until then you can still retrieve them with your recovery code",
// unconditionally, on every page. Yesterday's reading proved that false on a reachable state.
//
// THE CONDITION IS A PIN, NOT A COMPARISON AGAINST THE CURRENT KEY, and the difference is the whole
// design. The obvious proxy — "does the hub hold a key different from the one I use?" — asks about the
// wrong key: the set-aside copies were written under an OLDER key the box no longer has, which is why
// they were set aside. On a twice-rebuilt box the proxy answers "yes, promise it" about copies no key
// on file can open. The pin instead records the package the hub held AT THE DECISION and asks only
// "is the hub still holding that same one?".
//
// ⚠ THE PIN IS A RECORDED ASSUMPTION. It presumes the package held at the decision is the one that
// opens the set-aside copies. Nothing on the box records which key wrote them. See the field comment
// on settings.AbandonPinnedEscrowKeySHA256.
//
// The countdown is never started, shortened or triggered on a real machine — the clock is injected.
const pinnedHubKey = "1111111111111111111111111111111111111111111111111111111111111111"
const replacedHubKey = "2222222222222222222222222222222222222222222222222222222222222222"
// startedCountdown drives the PRODUCTION path (ResetOrphanedRepo → resetOrphanedRepo →
// startAbandonCountdown) so the pin cannot be written by tests alone while the live path never sets
// it — the inert-seam shape that has shipped here before, fully green.
func startedCountdown(t *testing.T, hubKeyAtDecision string) (*Manager, *settings.Settings, time.Time) {
t.Helper()
start := time.Date(2026, 8, 12, 12, 0, 0, 0, time.UTC)
m, sett, _ := abandonFixture(t, start)
if err := sett.SetHubEscrowKeySHA256(hubKeyAtDecision, start.Format(time.RFC3339)); err != nil {
t.Fatal(err)
}
if err := m.ResetOrphanedRepo(context.Background()); err != nil {
t.Fatalf("the production reset path failed: %v", err)
}
return m, sett, start
}
// PRODUCTION WIRING: the live decision path writes the pin. If this fails, every render test below is
// testing a field nothing sets.
func TestR302_ProductionResetPathWritesThePin(t *testing.T) {
_, sett, _ := startedCountdown(t, pinnedHubKey)
got := sett.GetOffboxTarget().AbandonPinnedEscrowKeySHA256
if got != pinnedHubKey {
t.Fatalf("pinned fingerprint = %q, want the hub key cached at the decision (%q). The whole "+
"design is that this is recorded when it is a fact; if the live path does not write it, "+
"the banner falls to the cautious branch for ever and the grace period becomes theatre",
got, pinnedHubKey)
}
if sett.GetOffboxTarget().AbandonAt == "" {
t.Error("no countdown recorded — the fixture is not exercising the path it claims to")
}
}
// ── SCENARIO A — package unchanged since the decision → the promise stands ──────────────────────
//
// RED-PROOF: force the condition false (drop the `cur == t.AbandonPinnedEscrowKeySHA256` arm) and this
// fails — a customer who can genuinely still change their mind loses the clause, which the code says
// explicitly must not happen ("a grace period during which recovery is impossible would be theatre").
func TestR302_ScenarioA_PackageUnchanged_RetrievalStillOffered(t *testing.T) {
m, _, _ := startedCountdown(t, pinnedHubKey)
st := m.AbandonStatus()
if !st.Active {
t.Fatal("countdown not active")
}
if !st.RetrievalStillOffered {
t.Error("the hub still holds the same package it held at the decision, so the customer really " +
"can still change their mind — the promise must stand")
}
}
// ── SCENARIO B — the package was REPLACED after the decision → promise withdrawn ────────────────
//
// This is the act that cost both demo boxes their history on 2026-08-04: a fresh escrow ceremony
// supersedes the package, and the old key it covered is unreachable (superseded packages grant no
// read path — hub store.go's own comment).
//
// RED-PROOF: re-read the pin at render (compare `cur` against itself, i.e. use the CURRENT cached
// value on both sides) and this fails — the promise returns, which is today's defect.
func TestR302_ScenarioB_PackageReplaced_PromiseWithdrawn(t *testing.T) {
m, sett, start := startedCountdown(t, pinnedHubKey)
// A fresh ceremony after the decision.
if err := sett.SetHubEscrowKeySHA256(replacedHubKey, start.Add(48*time.Hour).Format(time.RFC3339)); err != nil {
t.Fatal(err)
}
if st := m.AbandonStatus(); st.RetrievalStillOffered {
t.Error("the hub's package was replaced after the customer decided, so the key that opened the " +
"set-aside copies is no longer served — the banner must stop promising retrieval")
}
// The pin itself must NOT have moved: it is written once, at the decision.
if got := sett.GetOffboxTarget().AbandonPinnedEscrowKeySHA256; got != pinnedHubKey {
t.Errorf("the pin was refreshed to %q — a field re-read later answers a different question and "+
"silently restores the defect this replaces", got)
}
}
// ── SCENARIO D — a countdown started BEFORE this shipped carries no pin ─────────────────────────
//
// RED-PROOF: backfill the pin from the current cached value when it is empty and this fails — a legacy
// countdown gets promised at, asserting as recorded-at-the-decision something read long afterwards.
func TestR302_ScenarioD_LegacyCountdownWithoutAPin_TakesTheCautiousBranch(t *testing.T) {
m, sett, _ := startedCountdown(t, pinnedHubKey)
// Model the pre-R-302 on-disk shape: a live countdown, no pin.
if err := sett.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
o.AbandonPinnedEscrowKeySHA256 = ""
}); err != nil {
t.Fatal(err)
}
st := m.AbandonStatus()
if !st.Active {
t.Fatal("countdown should still be running")
}
if st.RetrievalStillOffered {
t.Error("a countdown with no pin was promised at. There is no honest way to know whether the " +
"hub's package is still the one from the decision, and the cautious answer is the only one " +
"available")
}
}
// ── SCENARIO E — pinned present, hub's cached value EMPTY → cautious ────────────────────────────
//
// The hub sends "" for a legacy package that provably seals no repository password. Empty is a
// measurement, not a match.
//
// RED-PROOF: treat empty as equal (drop the `cur != ""` arm) and this fails.
func TestR302_ScenarioE_EmptyHubHash_IsNotAMatch(t *testing.T) {
m, sett, start := startedCountdown(t, pinnedHubKey)
if err := sett.SetHubEscrowKeySHA256("", start.Add(time.Hour).Format(time.RFC3339)); err != nil {
t.Fatal(err)
}
if st := m.AbandonStatus(); st.RetrievalStillOffered {
t.Error("an EMPTY hub hash was read as a match. It means the hub holds a package that seals no " +
"repository password — the opposite of evidence that retrieval works")
}
}
// ── SCENARIO F — no countdown → nothing about retrieval is claimed at all ───────────────────────
func TestR302_ScenarioF_NoCountdown_NoClaim(t *testing.T) {
start := time.Date(2026, 8, 12, 12, 0, 0, 0, time.UTC)
m, _, _ := abandonFixture(t, start)
st := m.AbandonStatus()
if st.Active {
t.Fatal("no countdown was started, yet one is reported active")
}
if st.RetrievalStillOffered {
t.Error("retrieval was offered with no countdown running — the flag must be meaningless " +
"outside an abandonment, not default-true")
}
}
// The pin is a hash of a secret. It must never reach a customer-facing surface or the report; this
// pins that it is not accidentally exported through the read model.
func TestR302_PinIsNotExposedThroughTheReadModel(t *testing.T) {
m, _, _ := startedCountdown(t, pinnedHubKey)
st := m.AbandonStatus()
if st.RepoPath == pinnedHubKey {
t.Fatal("the pin leaked into RepoPath")
}
// AbandonState carries a BOOLEAN verdict, never the fingerprint itself.
if got := st.RetrievalStillOffered; got != true && got != false {
t.Fatal("unreachable")
}
}
// ── SCENARIO E, the case that actually bites — BOTH sides empty ─────────────────────────────────
//
// A legacy countdown (no pin) on a box whose hub reports an empty hash (a package sealing no repo
// password). "" == "" is the equality that would quietly become a promise, and it is the ONLY state
// where dropping the emptiness guards changes the answer — TestR302_ScenarioE above passes even with
// them removed, because its pin is non-empty so the equality fails on its own. That test guards the
// sentence; this one guards the claim.
//
// RED-PROOF: drop either `cur != ""` or `t.AbandonPinnedEscrowKeySHA256 != ""` and this fails.
func TestR302_ScenarioE2_BothSidesEmpty_IsNotAMatch(t *testing.T) {
m, sett, start := startedCountdown(t, pinnedHubKey)
if err := sett.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
o.AbandonPinnedEscrowKeySHA256 = "" // legacy countdown, no pin
}); err != nil {
t.Fatal(err)
}
if err := sett.SetHubEscrowKeySHA256("", start.Add(time.Hour).Format(time.RFC3339)); err != nil {
t.Fatal(err)
}
st := m.AbandonStatus()
if !st.Active {
t.Fatal("countdown should still be running")
}
if st.RetrievalStillOffered {
t.Error("two absences compared equal and became a promise. Empty means we could not see; two " +
"things we could not see are not a match")
}
}
@@ -0,0 +1,335 @@
package backup
import (
"context"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-241 — abandoning starts a countdown that ENDS THE QUESTION (Scenarios E, F, G).
//
// The countdown is driven by an injected clock throughout. §7.4 forbids shortening a live timer to
// watch the terminal step fire: it is the only thing in the product that deletes a customer's
// off-site history, and a step tested once on real data is a step regretted once.
// abandonFixture: an orphaned, configured box holding a key, with the hub holding a package for a
// DIFFERENT key — i.e. shape (c) is live and the customer is being offered recovery.
// Returns the manager and a recorder of every remote shell command issued.
type sshRecorder struct{ cmds []string }
func (r *sshRecorder) run(ctx context.Context, host, user string, port int, keyPath, knownHosts, remoteCmd string) ([]byte, error) {
r.cmds = append(r.cmds, remoteCmd)
return []byte(""), nil
}
func abandonFixture(t *testing.T, now time.Time) (*Manager, *settings.Settings, *sshRecorder) {
t.Helper()
m, sett, _ := offerFixture(t, true)
if err := sett.SetHubEscrowKeySHA256(otherKeyHash, now.Format(time.RFC3339)); err != nil {
t.Fatal(err)
}
if err := sett.UpdateOffboxStatus(func(o *settings.OffboxTarget) {
o.EscrowState = "escrowed"
o.RepoState = "orphaned"
}); err != nil {
t.Fatal(err)
}
rec := &sshRecorder{}
m.SetOffboxSSH(rec.run)
m.SetOffboxRunner(func(ctx context.Context, env []string, args ...string) ([]byte, error) { return []byte(""), nil })
m.SetOffboxClock(func() time.Time { return now })
return m, sett, rec
}
// ── SCENARIO E — abandoning sets aside, keeps the package, starts a countdown, stays reversible ──
func TestR241_ScenarioE_AbandonStartsAReversibleCountdown(t *testing.T) {
start := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC)
m, _, rec := abandonFixture(t, start)
if err := m.ResetOrphanedRepo(context.Background()); err != nil {
t.Fatalf("abandon: %v", err)
}
// The store was MOVED, not deleted — no rm anywhere in this phase.
joined := strings.Join(rec.cmds, " | ")
if !strings.Contains(joined, "mv ") {
t.Errorf("the old store must be moved aside; commands were: %s", joined)
}
if strings.Contains(joined, "rm -rf") {
t.Fatalf("NOTHING may be deleted when the customer abandons — only at the end of the grace. Commands: %s", joined)
}
st := m.AbandonStatus()
if !st.Active {
t.Fatal("a countdown must be running after an abandonment")
}
if got := st.DueAt.Sub(start); got != abandonGraceDays*24*time.Hour {
t.Errorf("countdown length = %v, want %d days", got, abandonGraceDays)
}
if st.DaysLeft != abandonGraceDays {
t.Errorf("DaysLeft = %d, want %d", st.DaysLeft, abandonGraceDays)
}
if st.RepoPath == "" {
t.Error("the set-aside path must be recorded, or the terminal step has nothing to delete")
}
// THE GRACE IS REAL: the recovery offer stays reachable for the whole window. A grace in which
// recovery is impossible would be decorative.
if !m.OffsiteRecoveryOffer() {
t.Fatal("the recovery offer MUST stay reachable during the grace — that is the change-of-mind path")
}
}
// ── SCENARIO G — changing your mind inside the window ───────────────────────────────────────────
//
// RED-PROOF: make the countdown uncancellable (delete the body of CancelAbandon). The countdown then
// survives a successful recovery and this test fails — a customer who proved they hold their code
// would still have the history deleted under them.
func TestR241_ScenarioG_RecoveryInsideTheWindowCancelsTheCountdown(t *testing.T) {
start := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC)
m, _, _ := abandonFixture(t, start)
if err := m.ResetOrphanedRepo(context.Background()); err != nil {
t.Fatal(err)
}
day6 := start.AddDate(0, 0, 6)
m.SetOffboxClock(func() time.Time { return day6 })
if st := m.AbandonStatus(); !st.Active || st.DaysLeft != 8 {
t.Fatalf("precondition: day 6 of 14 should leave 8 days, got %+v", st)
}
pathBefore := m.AbandonStatus().RepoPath
m.CancelAbandon("the customer recovered with their code")
st := m.AbandonStatus()
if st.Active {
t.Fatal("a countdown must be cancellable — the customer found their code")
}
if st.RepoPath != pathBefore {
t.Errorf("the set-aside store must stay NAMEABLE after a cancel: got %q want %q", st.RepoPath, pathBefore)
}
// And a sweep now deletes nothing, on any later date.
m.SetOffboxClock(func() time.Time { return start.AddDate(0, 0, 90) })
deleted, err := m.AbandonSweep(context.Background())
if err != nil || deleted {
t.Fatalf("a cancelled countdown must never delete: deleted=%v err=%v", deleted, err)
}
}
// ── SCENARIO F — the countdown ends the question, and removes BOTH halves ────────────────────────
//
// RED-PROOF (store half): make AbandonSweep skip the rm. The first assertion fails.
// RED-PROOF (package half): drop AbandonPurgeRequested from OffboxReportStatus. The declaration
// assertion fails — the hub is never asked and the package outlives the store for ever.
func TestR241_ScenarioF_TerminalStepRemovesBothHalvesTogether(t *testing.T) {
start := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC)
m, sett, rec := abandonFixture(t, start)
if err := m.ResetOrphanedRepo(context.Background()); err != nil {
t.Fatal(err)
}
setAside := m.AbandonStatus().RepoPath
// Not due yet — nothing happens, quietly.
m.SetOffboxClock(func() time.Time { return start.AddDate(0, 0, 13) })
if deleted, err := m.AbandonSweep(context.Background()); deleted || err != nil {
t.Fatalf("day 13 must not delete: deleted=%v err=%v", deleted, err)
}
// Due.
m.SetOffboxClock(func() time.Time { return start.AddDate(0, 0, 14).Add(time.Minute) })
rec.cmds = nil
deleted, err := m.AbandonSweep(context.Background())
if err != nil {
t.Fatalf("terminal step: %v", err)
}
if !deleted {
t.Fatal("the terminal step must delete when due")
}
// HALF 1: the store is gone.
joined := strings.Join(rec.cmds, " | ")
if !strings.Contains(joined, "rm -rf") || !strings.Contains(joined, setAside) {
t.Fatalf("the set-aside store at %s must be deleted; commands: %s", setAside, joined)
}
// HALF 2: the hub is ASKED for the package, and keeps being asked until it confirms.
st := m.OffboxReportStatus()
if st == nil || !st.AbandonPurgeRequested {
t.Fatalf("the report must declare abandon_purge_requested until the hub drops the package, got %+v", st)
}
// It repeats — a lost request must retry rather than leave the pair half-removed.
if d2, err2 := m.AbandonSweep(context.Background()); d2 || err2 != nil {
t.Fatalf("a second sweep must be a quiet no-op while awaiting the hub: deleted=%v err=%v", d2, err2)
}
if st2 := m.OffboxReportStatus(); st2 == nil || !st2.AbandonPurgeRequested {
t.Fatal("the declaration must persist across sweeps until confirmed")
}
// The hub confirms by no longer reporting a superseded package → the question is over.
m.ClearAbandonPurgeIfConfirmed(false)
if got := sett.GetOffboxTarget(); got.AbandonPurgeRequested || got.AbandonRepoPath != "" || got.AbandonAt != "" {
t.Errorf("the abandonment must be fully closed out, got %+v", got)
}
if st3 := m.OffboxReportStatus(); st3 != nil && st3.AbandonPurgeRequested {
t.Error("the declaration must stop once the hub has confirmed")
}
}
// While the hub STILL reports a superseded package, the close-out must not fire — otherwise the box
// stops asking and the package outlives the store silently, which is exactly half of Scenario F.
func TestR241_PurgeIsNotClosedOutWhileThePackageRemains(t *testing.T) {
start := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC)
m, sett, _ := abandonFixture(t, start)
if err := m.ResetOrphanedRepo(context.Background()); err != nil {
t.Fatal(err)
}
m.SetOffboxClock(func() time.Time { return start.AddDate(0, 0, 15) })
if _, err := m.AbandonSweep(context.Background()); err != nil {
t.Fatal(err)
}
m.ClearAbandonPurgeIfConfirmed(true) // the hub STILL holds a retained package
if !sett.GetOffboxTarget().AbandonPurgeRequested {
t.Fatal("the request must stand while the hub still reports a superseded package")
}
}
// A transport failure during the terminal step must NOT clear the countdown — it retries tomorrow.
// Silently abandoning the abandonment would leave the store for ever with nothing counting down.
func TestR241_TerminalStepFailureKeepsTheCountdownDue(t *testing.T) {
start := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC)
m, sett, _ := abandonFixture(t, start)
if err := m.ResetOrphanedRepo(context.Background()); err != nil {
t.Fatal(err)
}
m.SetOffboxSSH(func(ctx context.Context, host, user string, port int, keyPath, knownHosts, remoteCmd string) ([]byte, error) {
return []byte("ssh: connect to host nas.local port 22: No route to host"), context.DeadlineExceeded
})
m.SetOffboxClock(func() time.Time { return start.AddDate(0, 0, 15) })
deleted, err := m.AbandonSweep(context.Background())
if deleted || err == nil {
t.Fatalf("a failed deletion must be reported, not swallowed: deleted=%v err=%v", deleted, err)
}
got := sett.GetOffboxTarget()
if got.AbandonAt == "" || got.AbandonPurgeRequested {
t.Fatalf("a failed terminal step must leave the countdown DUE and unrequested, got %+v", got)
}
if !m.AbandonStatus().Active {
t.Error("the countdown must still be active so tomorrow's sweep retries")
}
}
// Quiet by construction: a box with no countdown does no work and says nothing (§ the daily job's
// own contract). Asserted, because "it probably does nothing" is how a sweep with a bug hides.
func TestR241_Sweep_QuietWhenNothingDue(t *testing.T) {
m, _, rec := abandonFixture(t, time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC))
deleted, err := m.AbandonSweep(context.Background())
if deleted || err != nil {
t.Fatalf("a box with no countdown must be a pure no-op: deleted=%v err=%v", deleted, err)
}
if len(rec.cmds) != 0 {
t.Fatalf("a no-op sweep must issue no remote commands, got %v", rec.cmds)
}
if m.AbandonStatus().Active {
t.Error("no countdown should be reported")
}
}
// The UNCLAIMED auto-reset must NOT start a customer countdown — nobody decided anything there.
// An as-delivered box tidying a stranger's leftover store must not put a 14-day deletion clock on it.
func TestR241_UnclaimedAutoResetStartsNoCountdown(t *testing.T) {
start := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC)
m, _, _ := abandonFixture(t, start)
t2 := m.settings.GetOffboxTarget()
base, env := m.offboxBaseArgs(t2)
if err := m.resetOrphanedRepo(context.Background(), base, env, "auto (unclaimed)"); err != nil {
t.Fatal(err)
}
if m.AbandonStatus().Active {
t.Fatal("the unclaimed auto-reset must not start a customer abandonment countdown")
}
}
// ── §7.5 — THE OPERATOR LEVERS ──────────────────────────────────────────────────────────────────
//
// The automatic 30-day ending is deliberately NOT built (R-245). These are what IS built: the path
// that actually happens is the customer telephoning, and support needs something to press.
func TestR241_OperatorCanExtendARunningCountdown(t *testing.T) {
start := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC)
m, _, rec := abandonFixture(t, start)
if err := m.ResetOrphanedRepo(context.Background()); err != nil {
t.Fatal(err)
}
day10 := start.AddDate(0, 0, 10)
m.SetOffboxClock(func() time.Time { return day10 })
due, err := m.ExtendAbandon(30)
if err != nil {
t.Fatalf("extend: %v", err)
}
if want := day10.AddDate(0, 0, 30); !due.Equal(want) {
t.Errorf("new due = %v, want %v (from NOW, not from the old date)", due, want)
}
// The original date has passed and nothing is deleted, because the extension moved it.
m.SetOffboxClock(func() time.Time { return start.AddDate(0, 0, 15) })
rec.cmds = nil
if deleted, serr := m.AbandonSweep(context.Background()); deleted || serr != nil {
t.Fatalf("an extended countdown must not fire on the old date: deleted=%v err=%v", deleted, serr)
}
if len(rec.cmds) != 0 {
t.Fatalf("nothing may be deleted after an extension, got %v", rec.cmds)
}
}
func TestR241_OperatorCanStopARunningCountdown(t *testing.T) {
start := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC)
m, _, rec := abandonFixture(t, start)
if err := m.ResetOrphanedRepo(context.Background()); err != nil {
t.Fatal(err)
}
if err := m.StopAbandon(); err != nil {
t.Fatalf("stop: %v", err)
}
if m.AbandonStatus().Active {
t.Fatal("the countdown must be stopped")
}
m.SetOffboxClock(func() time.Time { return start.AddDate(0, 0, 90) })
rec.cmds = nil
if deleted, err := m.AbandonSweep(context.Background()); deleted || err != nil {
t.Fatalf("a stopped countdown must never delete: deleted=%v err=%v", deleted, err)
}
if len(rec.cmds) != 0 {
t.Fatalf("a stopped countdown must issue no remote commands, got %v", rec.cmds)
}
}
// Both levers REFUSE when nothing is running. A silent no-op is the thing an operator most easily
// mistakes for success — they would tell the customer it was handled.
func TestR241_OperatorLeversRefuseWhenNothingIsRunning(t *testing.T) {
m, _, _ := abandonFixture(t, time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC))
if _, err := m.ExtendAbandon(30); err == nil {
t.Error("extending a countdown that is not running must be an error, never a quiet success")
}
if err := m.StopAbandon(); err == nil {
t.Error("stopping a countdown that is not running must be an error, never a quiet success")
}
if _, err := m.ExtendAbandon(0); err == nil {
t.Error("a non-positive extension must be refused")
}
}
// Once the store is deleted there is nothing left to extend or stop, and saying otherwise would be
// the worst kind of reassurance: an operator telling a customer their data is safe when it is gone.
func TestR241_OperatorLeversRefuseAfterTheDeletion(t *testing.T) {
start := time.Date(2026, 8, 7, 12, 0, 0, 0, time.UTC)
m, _, _ := abandonFixture(t, start)
if err := m.ResetOrphanedRepo(context.Background()); err != nil {
t.Fatal(err)
}
m.SetOffboxClock(func() time.Time { return start.AddDate(0, 0, 15) })
if _, err := m.AbandonSweep(context.Background()); err != nil {
t.Fatal(err)
}
if _, err := m.ExtendAbandon(30); err == nil {
t.Error("extending after the deletion must be refused — there is nothing left to save")
}
if err := m.StopAbandon(); err == nil {
t.Error("stopping after the deletion must be refused — there is nothing left to save")
}
}
+17 -9
View File
@@ -29,13 +29,13 @@ type offboxBlocked struct {
// snapshot, plus any Hungarian warnings for capture gaps. It never returns optional/excluded paths // snapshot, plus any Hungarian warnings for capture gaps. It never returns optional/excluded paths
// (the TierOffsite filter drops them — §2). Returns (nil, nil) for the legacy / no-provider / no-block // (the TierOffsite filter drops them — §2). Returns (nil, nil) for the legacy / no-provider / no-block
// world: offsite stays UNIT-ONLY, byte-identical to pre-v0.134.0 (the SQ5 cost-regression guard). // world: offsite stays UNIT-ONLY, byte-identical to pre-v0.134.0 (the SQ5 cost-regression guard).
func (m *Manager) offboxCaptureSet(stack string) (extra []string, warns []string) { func (m *Manager) offboxCaptureSet(stack string) (extra []string, warns []string, gaps []string) {
if m.stackProvider == nil { if m.stackProvider == nil {
return nil, nil // no provider wired → legacy world → unit only return nil, nil, nil // no provider wired → legacy world → unit only
} }
binds, has := m.stackProvider.GetStackClassifiedBinds(stack) binds, has := m.stackProvider.GetStackClassifiedBinds(stack)
if !has { if !has {
return nil, nil // no backup block → legacy → unit only return nil, nil, nil // no backup block → legacy → unit only
} }
// Resolve against the app's LIVE HDD_PATH (raw — NOT GetAppDrivePath, whose systemDataPath fallback // Resolve against the app's LIVE HDD_PATH (raw — NOT GetAppDrivePath, whose systemDataPath fallback
// would resolve userdata onto the wrong drive). Empty ⇒ undeployed / no HDD (decision §2.4): // would resolve userdata onto the wrong drive). Empty ⇒ undeployed / no HDD (decision §2.4):
@@ -43,12 +43,11 @@ func (m *Manager) offboxCaptureSet(stack string) (extra []string, warns []string
hdd := strings.TrimSpace(m.stackProvider.GetStackHDDPath(stack)) hdd := strings.TrimSpace(m.stackProvider.GetStackHDDPath(stack))
if hdd == "" { if hdd == "" {
m.logger.Printf("[WARN] [offbox] %s: not deployed — offsite push is unit-only (mandatory userdata not resolvable)", stack) m.logger.Printf("[WARN] [offbox] %s: not deployed — offsite push is unit-only (mandatory userdata not resolvable)", stack)
return nil, []string{fmt.Sprintf("Figyelmeztetés: a(z) %s nincs telepítve — csak a mentési egység került a távoli mentésbe.", stack)} return nil, []string{fmt.Sprintf("Figyelmeztetés: a(z) %s nincs telepítve — csak a mentési egység került a távoli mentésbe.", stack)}, nil
} }
nsRoot := m.namespaceRoot(hdd) nsRoot := m.namespaceRoot(hdd)
cs := appbackup.ComputeCaptureSet(binds, has, appbackup.TierOffsite, nsRoot) cs := appbackup.ComputeCaptureSet(binds, has, appbackup.TierOffsite, nsRoot, m.stackProvider.GetImportRoot())
var gaps []string
// Structurally-refused MANDATORY paths (traversal / bare drive-root / reserved backups/ zone) are // Structurally-refused MANDATORY paths (traversal / bare drive-root / reserved backups/ zone) are
// loud ERROR gaps — the path the customer thinks is protected is not in the snapshot. // loud ERROR gaps — the path the customer thinks is protected is not in the snapshot.
for _, sk := range cs.Skipped { for _, sk := range cs.Skipped {
@@ -60,17 +59,26 @@ func (m *Manager) offboxCaptureSet(stack string) (extra []string, warns []string
} }
// Stat-filter (§2.5): a declared mandatory path absent on disk. restic would skip it SILENTLY // Stat-filter (§2.5): a declared mandatory path absent on disk. restic would skip it SILENTLY
// (SP-3.4), so drop it from argv AND warn — never a silent "looks backed up but isn't". // (SP-3.4), so drop it from argv AND warn — never a silent "looks backed up but isn't".
//
// R-203: the class check mirrors tier2_capture.go's ("optional-missing is silent"). It is a NO-OP
// today — TierOffsite's tierKeeps() already admits ClassMandatory only, so cs.Paths cannot contain
// an optional path here — and it is written anyway so the two tiers read the same and so the
// verdict below can never be flipped by an unused optional folder if that filter ever widens.
for _, p := range cs.Paths { for _, p := range cs.Paths {
if _, err := os.Stat(p.Abs); err != nil { if _, err := os.Stat(p.Abs); err != nil {
if p.Class == appbackup.ClassMandatory {
m.logger.Printf("[WARN] [offbox] %s: mandatory data path missing on disk, skipped from offsite: %s", stack, p.Abs) m.logger.Printf("[WARN] [offbox] %s: mandatory data path missing on disk, skipped from offsite: %s", stack, p.Abs)
gaps = append(gaps, p.RelPath) gaps = append(gaps, p.RelPath)
continue }
continue // optional-missing is silent (not a gap) — parity with Tier 2
} }
extra = append(extra, p.Abs) extra = append(extra, p.Abs)
} }
if len(gaps) > 0 { if len(gaps) > 0 {
warns = append(warns, fmt.Sprintf("Figyelmeztetés: a(z) %s alkalmazás egyes adatmappái nem kerültek a távoli mentésbe: %s.", // R-234 §7.4: this sits beside the whole-app gap message on the same card, and both now drive
// the same `incomplete` verdict — so it says what to do, not only what happened.
warns = append(warns, fmt.Sprintf("Figyelmeztetés: a(z) %s alkalmazás egyes adatmappái nem kerültek a távoli mentésbe: %s. Ellenőrizd, hogy a mappák megvannak-e a meghajtón; ha igen és ez a következő mentés után is látszik, szólj az üzemeltetőnek.",
stack, strings.Join(gaps, ", "))) stack, strings.Join(gaps, ", ")))
} }
return extra, warns return extra, warns, gaps
} }
@@ -0,0 +1,193 @@
package backup
import (
"encoding/json"
"io"
"log"
"os"
"path/filepath"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-204 item 4 / R-193 — a REBUILT box declares that it needs an off-site credential, instead of
// reporting an absence the hub cannot interpret.
//
// THE POINT OF THESE TESTS is the conjunction. An absent off-site object has FOUR meanings (never
// configured / mid-restart / a transient read failure / rebuilt-and-stranded). The declaration has
// one, and it is only sound because BOTH halves are required: a fresh data area AND a hub-held
// recovery package. Scenario B is the one that matters most — drop the escrow half and every
// un-configured box in the fleet starts asking for a credential.
// bareManager builds a Manager with NO off-site target and NO repository password — the shape of a
// freshly rebuilt box before anything is configured.
func bareManager(t *testing.T) (*Manager, *settings.Settings) {
t.Helper()
lg := log.New(io.Discard, "", 0)
dataDir := t.TempDir()
sett, err := settings.Load(filepath.Join(dataDir, "settings.json"), lg)
if err != nil {
t.Fatal(err)
}
cfg := &config.Config{}
cfg.Paths.DataDir = dataDir
cfg.Paths.SystemDataPath = filepath.Join(dataDir, "sys")
return NewManager(cfg, sett, lg), sett
}
// SCENARIO A — a rebuilt box (fresh data area + a hub-held escrow) DECLARES the state.
//
// RED-PROOF: remove the `GetHubEscrowIdentityPresent()` condition from needsOffsiteCredential —
// Scenario A still passes (it has an escrow), and SCENARIO B FAILS, which is the point: the plausible
// wrong fix is to declare on freshness alone, and that would make every un-configured box in the
// fleet ask for a credential.
func TestOffsiteDeclare_RebuiltBoxDeclaresNeedsCredential(t *testing.T) {
m, sett := bareManager(t)
if err := sett.SetHubEscrowIdentityPresent(true); err != nil {
t.Fatal(err)
}
st := m.OffboxReportStatus()
if st == nil {
t.Fatal("a rebuilt box reported NO off-site object — the hub cannot distinguish it from a box that never had off-site backups (this is the defect)")
}
if st.State != OffsiteStateNeedsCredential {
t.Fatalf("declared state = %q, want %q", st.State, OffsiteStateNeedsCredential)
}
// Enabled MUST be false and the sizes zero — that is what makes the declaration inert to the
// hub's existing fill and staleness checkers (and to a pre-upgrade hub).
if st.Enabled {
t.Error("a declaration must not claim the tier is enabled — the hub's staleness check keys on it")
}
if st.QuotaGB != 0 || st.RepoSizeBytes != 0 || st.SnapshotCount != 0 {
t.Errorf("a declaration must carry zero sizes (fill band keys on them): %+v", st)
}
// And it must be on the off-site object, not a new top-level field.
b, err := json.Marshal(st)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(b), `"state":"needs_credential"`) {
t.Fatalf("declared state absent from the marshalled off-site object: %s", b)
}
if !strings.Contains(string(b), `"enabled":false`) {
t.Fatalf("marshalled object must carry enabled:false: %s", b)
}
}
// SCENARIO B — a box that never had off-site backups says NOTHING. This is the guard on the
// conjunction; without it the feature churns credentials fleet-wide.
func TestOffsiteDeclare_NeverHadOffsiteSaysNothing(t *testing.T) {
m, _ := bareManager(t) // fresh data area, but NO hub-held escrow
if st := m.OffboxReportStatus(); st != nil {
t.Fatalf("a box that never had off-site backups DECLARED a need: %+v — every un-configured box in the fleet would now ask for a credential", st)
}
}
// SCENARIO D (R-218) — THE DECLARATION STOPS WHEN THE TIER WORKS, NOT WHEN A KEY EXISTS.
//
// ⚠ THIS TEST ASSERTED THE OPPOSITE until v0.201.0, and it was green the whole time. It required a
// box holding a repository password to stay SILENT — which reads as a sound freshness test and is the
// exact opposite on the one path that matters, because installing a repository password is the
// RECOVERY SCREEN'S WHOLE JOB. Measured live 2026-08-05 (CAMPAIGN-11 Phase 1): 32 seconds after the
// hub re-staged the credential, the customer's successful unlock switched off the mechanism that
// would have delivered the coordinates for the key they had just recovered. Deadlock, both halves.
//
// RED-PROOF: restore the `if _, ok := m.OffboxRepoPasswordHash(); ok { return false }` short-circuit
// in needsOffsiteCredential and this test FAILS — the box goes silent again with no target, which is
// the deadlock. Demonstrated failing before this test was kept.
func TestOffsiteDeclare_StillDeclaresAfterARecoveredKeyIsPlaced(t *testing.T) {
m, sett := bareManager(t)
if err := sett.SetHubEscrowIdentityPresent(true); err != nil {
t.Fatal(err)
}
// The post-unlock shape: the recovered repository password is on disk, and there is STILL no
// off-site target — so the box cannot use what it just recovered.
if err := os.MkdirAll(m.offboxDir(), 0o700); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(m.offboxPwPath(), []byte("a-recovered-repository-password"), 0o600); err != nil {
t.Fatal(err)
}
st := m.OffboxReportStatus()
if st == nil {
t.Fatal("R-218: the box went SILENT after recovering its key while still having no off-site target — the hub's staged credential is never collected and nothing ever asks again")
}
if st.State != OffsiteStateNeedsCredential {
t.Fatalf("declared state = %q, want %q", st.State, OffsiteStateNeedsCredential)
}
}
// SCENARIO E — and once the tier ACTUALLY WORKS the box goes quiet. This is the condition that
// replaces the deleted one, and the pair above/below is what makes the deletion safe.
func TestOffsiteDeclare_ConfiguredTierIsSilent(t *testing.T) {
m, sett := bareManager(t)
if err := sett.SetHubEscrowIdentityPresent(true); err != nil {
t.Fatal(err)
}
if err := sett.SetOffboxTarget(&settings.OffboxTarget{
Enabled: true, Host: "nas.local", Port: 22, User: "felhom", RepoPath: "/srv/repo",
Schedule: "daily", EscrowState: "escrowed",
}); err != nil {
t.Fatal(err)
}
st := m.OffboxReportStatus()
if st == nil {
t.Fatal("a configured tier must still report its ordinary off-site object")
}
if st.State == OffsiteStateNeedsCredential {
t.Fatal("a box whose tier is configured must not keep asking for a credential")
}
}
// A DISABLED target is the customer's own choice, not a rebuild — it must not declare either.
func TestOffsiteDeclare_DisabledTargetIsNotStranded(t *testing.T) {
m, sett := bareManager(t)
if err := sett.SetHubEscrowIdentityPresent(true); err != nil {
t.Fatal(err)
}
if err := sett.SetOffboxTarget(&settings.OffboxTarget{
Enabled: false, Host: "nas.local", Port: 22, User: "felhom", RepoPath: "/srv/repo",
}); err != nil {
t.Fatal(err)
}
if st := m.OffboxReportStatus(); st != nil {
t.Fatalf("a deliberately DISABLED target declared a need: %+v", st)
}
}
// A CONFIGURED box's report object must be byte-identical to v0.198.0's — no `state` key at all.
// This is what lets a pre-upgrade hub and every existing checker read the fleet unchanged.
func TestOffsiteDeclare_ConfiguredBoxJSONIsUnchanged(t *testing.T) {
m, sett := bareManager(t)
if err := sett.SetOffboxTarget(&settings.OffboxTarget{
Enabled: true, Host: "nas.local", Port: 22, User: "felhom", RepoPath: "/srv/repo",
Schedule: "daily", EscrowState: "escrowed", LastStatus: "ok",
}); err != nil {
t.Fatal(err)
}
st := m.OffboxReportStatus()
if st == nil {
t.Fatal("a configured box must still report an off-site object")
}
if st.State != "" {
t.Errorf("a configured box must declare NO state, got %q", st.State)
}
b, err := json.Marshal(st)
if err != nil {
t.Fatal(err)
}
if strings.Contains(string(b), `"state"`) {
t.Fatalf("a healthy report's JSON gained a `state` key — it must stay byte-compatible: %s", b)
}
if !strings.Contains(string(b), `"enabled":true`) {
t.Fatalf("a configured box must report enabled:true: %s", b)
}
}
@@ -0,0 +1,131 @@
package backup
import (
"context"
"encoding/json"
"errors"
"sort"
"time"
)
// R-193 Part 3 — WHAT IS IN THERE. After a successful unlock the customer is shown the contents of the
// repository they just opened: which apps, from when, how big.
//
// READ-ONLY, AND THAT IS THE POINT. This restores nothing, puts nothing back, and compares nothing
// against live data. Unlocking and restoring are separate (operator ruling, 2026-08-05): restore is
// already per-app and already lives in the backups area, and a screen that unlocks and then offers to
// overwrite is two decisions wearing one button.
//
// WHY A LISTING AT ALL, rather than a success message: "unlocked" with nothing shown is
// indistinguishable from having unlocked an EMPTY store, and the customer has no way to tell whether
// what came back is the right thing. Seeing their own app names and dates is how they know.
// errNoOffsiteTarget is returned when the repository cannot even be addressed — no off-site target is
// configured on this box yet. Distinguished from a read failure because the remedy differs: this one
// resolves by itself once the tier is re-applied.
var errNoOffsiteTarget = errors.New("no off-site target is configured on this box yet")
// ErrNoOffsiteTarget reports whether err is the not-yet-configured case, so a caller can say the right
// thing rather than showing a generic failure.
func ErrNoOffsiteTarget(err error) bool { return errors.Is(err, errNoOffsiteTarget) }
// ErrNoOffsiteTargetSentinel exposes the sentinel itself so other packages — and their tests — can
// construct the not-yet-configured case. Added for R-237, whose restore list must distinguish
// "no target yet" (resolves by itself) from "could not read" (does not), and must be able to pin
// both in a table test.
func ErrNoOffsiteTargetSentinel() error { return errNoOffsiteTarget }
// OffsiteInventoryApp is one app's presence in the opened repository. Non-secret throughout.
type OffsiteInventoryApp struct {
App string // the restic tag == the stack name
LatestAt time.Time // the newest snapshot's time for this app
SizeBytes int64 // restore size of that newest snapshot (0 = could not be determined)
}
// OffsiteInventory is the whole answer, including the EMPTY case stated explicitly.
type OffsiteInventory struct {
Apps []OffsiteInventoryApp
// Empty is true when the repository opened cleanly and holds no snapshots. It is a real and
// confusing outcome — a bare list there reads as a broken page — so it is named rather than
// inferred from len(Apps)==0, which is also what a failed read looks like.
Empty bool
}
// OffsiteInventoryList opens the repository and reports what is in it, grouped per app. One
// `snapshots --json` call for the whole repo, then one `stats` per app for the newest snapshot's size.
//
// A per-app size failure is NOT fatal: the app is still listed, with SizeBytes 0, because knowing an
// app is in there matters more than knowing how big it is, and dropping it would under-report the
// customer's own data.
func (m *Manager) OffsiteInventoryList(ctx context.Context) (OffsiteInventory, error) {
var inv OffsiteInventory
// A box can hold a recovered key and still have no off-site COORDINATES — the pristine rebuilt
// shape, before its target is re-applied. Reading the repository is impossible then, and saying so
// is the honest answer; without this guard offboxBaseArgs nil-derefs on the missing target.
if !m.OffboxConfigured() {
return inv, errNoOffsiteTarget
}
t := m.settings.GetOffboxTarget()
base, env := m.offboxBaseArgs(t)
sctx, cancel := context.WithTimeout(ctx, offboxProbeTimeout)
defer cancel()
out, err := m.runner()(sctx, env, append(append([]string{}, base...), "snapshots", "--json")...)
if err != nil {
return inv, err
}
var snaps []struct {
ShortID string `json:"short_id"`
ID string `json:"id"`
Time time.Time `json:"time"`
Tags []string `json:"tags"`
}
if uerr := json.Unmarshal(out, &snaps); uerr != nil {
return inv, uerr
}
if len(snaps) == 0 {
inv.Empty = true
return inv, nil
}
// Newest snapshot per tag. A snapshot may carry several tags; each names an app it belongs to.
newest := map[string]struct {
id string
at time.Time
}{}
for _, s := range snaps {
id := s.ShortID
if id == "" {
id = s.ID
}
for _, tag := range s.Tags {
if tag == "" {
continue
}
if cur, ok := newest[tag]; !ok || s.Time.After(cur.at) {
newest[tag] = struct {
id string
at time.Time
}{id: id, at: s.Time}
}
}
}
if len(newest) == 0 {
// Snapshots exist but carry no tags — not "empty", and saying so would be a lie. Report an
// empty app list without the Empty flag; the page renders the honest in-between wording.
return inv, nil
}
for tag, n := range newest {
app := OffsiteInventoryApp{App: tag, LatestAt: n.at}
if size, serr := m.offboxSnapshotSize(ctx, n.id); serr == nil {
app.SizeBytes = size
} else {
m.logger.Printf("[WARN] [offbox] inventory: size of %s's newest snapshot unknown: %v (listing it anyway)", tag, serr)
}
inv.Apps = append(inv.Apps, app)
}
sort.Slice(inv.Apps, func(i, j int) bool { return inv.Apps[i].App < inv.Apps[j].App })
return inv, nil
}
// HumanizeBytes exposes the shared byte formatter to the web layer so the recovery page renders sizes
// the same way every other surface does.
func HumanizeBytes(n int64) string { return humanizeBytes(n) }
@@ -0,0 +1,132 @@
package backup
import (
"errors"
"io"
"log"
"path/filepath"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
func newTestSettings(t *testing.T) *settings.Settings {
t.Helper()
sett, err := settings.Load(filepath.Join(t.TempDir(), "settings.json"), log.New(io.Discard, "", 0))
if err != nil {
t.Fatalf("settings.Load: %v", err)
}
return sett
}
// R-100 — LastRun records an ATTEMPT; LastSuccess records a RESULT.
//
// The defect these pin: `LastRun` is written unconditionally at the end of every offsite run, failures
// included, so the hub's staleness verdict ("how long since LastRun?") was really asking "how long
// since we last TRIED?" — and a tier failing on every single run read as perfectly fresh forever.
//
// These are the CONTROLLER half (does the anchor move only on success, and does it survive the writes
// that rebuild the target?). The hub half — does the verdict count from it — lives in the hub's
// offsite tests.
// The invariant named by the comment at the write site, per the standing rule that an asserted
// invariant needs a test pinning it. This calls the PRODUCTION rule — an earlier version of this test
// re-implemented it in a local closure and was hollow: mutating offbox.go left it green.
//
// RED-PROOF: make offboxAnchorAfterRun return `at` unconditionally (drop the runErr guard) → this
// fails with "a FAILED run advanced LastSuccess — that is the R-100 defect in mirror image".
func TestOffboxAnchorAfterRun_FailureNeitherAdvancesNorClears(t *testing.T) {
const monday = "2026-07-20T02:15:00Z"
boom := errors.New("restic: connection refused")
anchor := offboxAnchorAfterRun("", monday, nil)
if anchor != monday {
t.Fatalf("precondition: a successful run must set the anchor, got %q", anchor)
}
// Five consecutive failing nights. The attempt clock moves; the anchor must not.
for _, night := range []string{
"2026-07-21T02:15:00Z", "2026-07-22T02:15:00Z", "2026-07-23T02:15:00Z",
"2026-07-24T02:15:00Z", "2026-07-25T02:15:00Z",
} {
anchor = offboxAnchorAfterRun(anchor, night, boom)
if anchor == night {
t.Fatalf("a FAILED run advanced LastSuccess to %q — that is the R-100 defect in mirror image", anchor)
}
if anchor != monday {
t.Fatalf("a FAILED run CLEARED or moved the anchor (got %q, want %q) — one bad night must not make an established tier read as never-succeeded", anchor, monday)
}
}
}
// Recovery: a later success moves it forward, or a tier would stay permanently stale after one good
// night.
//
// RED-PROOF: make offboxAnchorAfterRun return `prev` unconditionally → this fails with
// "a successful run did not advance the anchor".
func TestOffboxAnchorAfterRun_SuccessAdvances(t *testing.T) {
got := offboxAnchorAfterRun("2026-07-20T02:15:00Z", "2026-07-26T02:15:00Z", nil)
if got != "2026-07-26T02:15:00Z" {
t.Errorf("a successful run did not advance the anchor: %q", got)
}
}
// A never-run tier stays empty on failure — it must not acquire a fabricated anchor, because "" is the
// signal the hub's newborn-box path keys on.
func TestOffboxAnchorAfterRun_NeverRanStaysEmptyOnFailure(t *testing.T) {
if got := offboxAnchorAfterRun("", "2026-07-21T02:15:00Z", errors.New("boom")); got != "" {
t.Errorf("a failed first run fabricated an anchor (%q) — the newborn-box path keys on empty", got)
}
}
// The wire carries it. A field the hub cannot see is a field that does not exist — the "seam built but
// never wired" class this project has hit four times.
//
// RED-PROOF: drop `LastSuccess: t.LastSuccess` from OffboxReportStatus() → this fails with
// "OffboxReportStatus dropped LastSuccess — the hub would degrade forever on a controller that has it".
func TestOffboxReportStatus_CarriesLastSuccess(t *testing.T) {
m := &Manager{settings: newTestSettings(t)}
if err := m.settings.SetOffboxTarget(&settings.OffboxTarget{
Enabled: true,
Host: "nas.example",
User: "u1",
RepoPath: "/vol/repo",
EscrowState: "escrowed",
LastRun: "2026-07-26T02:15:00Z",
LastStatus: "ok",
LastSuccess: "2026-07-26T02:15:00Z",
}); err != nil {
t.Fatalf("seed: %v", err)
}
got := m.OffboxReportStatus()
if got == nil {
t.Fatal("OffboxReportStatus returned nil for an enabled target")
}
if got.LastSuccess != "2026-07-26T02:15:00Z" {
t.Errorf("OffboxReportStatus dropped LastSuccess — the hub would degrade forever on a controller that has it (got %q)", got.LastSuccess)
}
}
// A re-apply from the hub is not a new tier. Dropping the anchor here would reset an established tier
// to "never succeeded" every time the hub re-pushes its descriptor.
//
// RED-PROOF: remove `tgt.LastSuccess = cur.LastSuccess` from ApplyOffsiteTarget's carry-over block →
// this fails with "a hub re-apply erased the staleness anchor".
func TestApplyOffsiteTarget_PreservesLastSuccess(t *testing.T) {
m := &Manager{settings: newTestSettings(t)}
if err := m.settings.SetOffboxTarget(&settings.OffboxTarget{
Enabled: true, Host: "nas.example", User: "u1", RepoPath: "/vol/repo",
EscrowState: "escrowed", LastSuccess: "2026-07-26T02:15:00Z", LastRun: "2026-07-27T02:15:00Z",
}); err != nil {
t.Fatalf("seed: %v", err)
}
cur := m.settings.GetOffboxTarget()
// Mirror ApplyOffsiteTarget's carry-over onto a freshly-built target.
tgt := &settings.OffboxTarget{Enabled: true, Host: "nas.example", User: "u1", RepoPath: "/vol/repo", Schedule: "daily"}
tgt.EscrowState = cur.EscrowState
tgt.LastRun, tgt.LastStatus, tgt.LastError = cur.LastRun, cur.LastStatus, cur.LastError
tgt.LastSuccess = cur.LastSuccess
if tgt.LastSuccess != "2026-07-26T02:15:00Z" {
t.Errorf("a hub re-apply erased the staleness anchor (got %q)", tgt.LastSuccess)
}
}
@@ -0,0 +1,192 @@
package backup
import (
"context"
"log"
"os"
"path/filepath"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-241 — THE MINT GUARD. This file is the session's headline test.
//
// The defect, measured on the final walk (SPIKE-r241-recovery-offer-2026-08-07): a rebuilt box's
// credential self-heal reached WriteOffboxSecrets at 03:18:06Z and minted a fresh repository password
// over a hub package sealing a DIFFERENT key. The recovery screen then correctly reported that there
// was nothing recoverable under the key the box held. The screen was honest; the minting was not.
//
// Scenario A asserts the key is NOT written. Scenario B asserts the guard is narrow enough that a
// first-time box still starts — the guard's own failure mode, and the one an over-broad fix produces.
// mintGuardManager builds a Manager with NO offbox secrets written, so the mint branch is live.
// hubHoldsPackage sets the ACK-cached fact the guard consults.
func mintGuardManager(t *testing.T, hubHoldsPackage bool) (*Manager, *settings.Settings, string) {
t.Helper()
logger := log.New(os.Stderr, "", 0)
dataDir := t.TempDir()
sett, err := settings.Load(filepath.Join(dataDir, "settings.json"), logger)
if err != nil {
t.Fatal(err)
}
cfg := &config.Config{}
cfg.Paths.DataDir = dataDir
cfg.Paths.SystemDataPath = filepath.Join(dataDir, "sys")
m := NewManager(cfg, sett, logger)
if err := sett.SetHubEscrowIdentityPresent(hubHoldsPackage); err != nil {
t.Fatal(err)
}
return m, sett, filepath.Join(dataDir, "offbox", "repo_password")
}
// ── SCENARIO A — the box does not mint over a sealed package ────────────────────────────────────
//
// RED-PROOF: delete the `if m.sealedPackageHeld()` block in WriteOffboxSecrets. The password file
// then exists and this test fails on the first assertion — which is exactly the 03:18:06Z event.
func TestR241_ScenarioA_NoMintWhenHubHoldsSealedPackage(t *testing.T) {
m, _, pwPath := mintGuardManager(t, true)
err := m.WriteOffboxSecrets("PRIVATE-KEY-MATERIAL", "nas.local ssh-ed25519 AAAAhostkey")
if !IsOffboxSealedPackageHeld(err) {
t.Fatalf("want the sealed-package refusal sentinel, got %v", err)
}
// THE ASSERTION THAT IS THE WHOLE SESSION: no key on disk.
if _, serr := os.Stat(pwPath); !os.IsNotExist(serr) {
t.Fatalf("R-241 REGRESSION: a repository password was minted over the hub's sealed package (stat err=%v)", serr)
}
// The transport IS still written — the refusal is a holding state, not a failure. Without this the
// recovery screen could not bring the tier up when the key arrives (R-219).
for _, f := range []string{"ssh_key", "known_hosts"} {
if _, serr := os.Stat(filepath.Join(filepath.Dir(pwPath), f)); serr != nil {
t.Errorf("transport file %s should still be written on the refusal path: %v", f, serr)
}
}
}
// Scenario A at the APPLY level — the path the self-heal actually takes. ApplyOffsiteTarget must
// swallow the sentinel, record the target, and NOT stage an escrow.
func TestR241_ScenarioA_ApplyOffsiteTargetHoldsInsteadOfMinting(t *testing.T) {
m, sett, pwPath := mintGuardManager(t, true)
staged := 0
stage := func(ctx context.Context, pw string) error { staged++; return nil }
tgt := &settings.OffboxTarget{Enabled: true, Host: "box.example", Port: 23, User: "u1", RepoPath: "/home/felhom-repo"}
if err := m.ApplyOffsiteTarget(context.Background(), tgt, "KEYMATERIAL", "box.example ssh-ed25519 HOSTKEY", stage); err != nil {
t.Fatalf("apply should SUCCEED into the holding state, not fail: %v", err)
}
if _, serr := os.Stat(pwPath); !os.IsNotExist(serr) {
t.Fatalf("R-241 REGRESSION: apply minted a repository password over the sealed package")
}
if staged != 0 {
t.Errorf("nothing may be staged for escrow — there is no key to escrow; staged=%d", staged)
}
// The target is recorded, so the box stops declaring needs_credential and the hub stops re-staging.
if got := sett.GetOffboxTarget(); got == nil {
t.Fatal("the transport target must be recorded, or the hub re-stages a consumed credential forever")
}
// Runs stay gated: no password file ⇒ not configured.
if m.OffboxConfigured() {
t.Error("OffboxConfigured must be false while the key is awaited — runs must not proceed")
}
// And the box says so, in the state the hub reads.
if !m.OffboxAwaitingRecoveryKey() {
t.Error("OffboxAwaitingRecoveryKey should be true in the holding state")
}
st := m.OffboxReportStatus()
if st == nil || st.State != OffsiteStateAwaitingRecoveryKey {
t.Fatalf("want declared state %q, got %+v", OffsiteStateAwaitingRecoveryKey, st)
}
if st.Enabled {
t.Error("the declared holding object must carry Enabled=false so existing hub readers stay inert")
}
}
// ── SCENARIO B — a box the hub holds nothing for still mints, exactly as today ───────────────────
//
// RED-PROOF: widen the guard to `if true` (or drop the GetHubEscrowIdentityPresent() conjunct in
// sealedPackageHeld). A first-time box then cannot start, and this test fails — the failure mode an
// over-broad fix produces, which is why the guard is written as a conjunction.
func TestR241_ScenarioB_FirstTimeBoxStillMints(t *testing.T) {
m, _, pwPath := mintGuardManager(t, false) // the hub holds nothing for us
if err := m.WriteOffboxSecrets("PRIVATE-KEY-MATERIAL", "nas.local ssh-ed25519 AAAAhostkey"); err != nil {
t.Fatalf("a first-time box must mint exactly as before, got %v", err)
}
pw, rerr := os.ReadFile(pwPath)
if rerr != nil {
t.Fatalf("a first-time box must get a repository password: %v", rerr)
}
if !offboxRepoPwPattern.Match(pw) {
t.Errorf("minted password is not the expected 64-hex shape")
}
if m.OffboxAwaitingRecoveryKey() {
t.Error("a box with no sealed package is not awaiting anything")
}
}
// The guard must not fire once a key EXISTS — a healthy box re-applying its target (a quota bump,
// a hub re-push) must be untouched, package or no package. This is the idempotency half.
func TestR241_ExistingKeyIsNeverDisturbed(t *testing.T) {
m, _, pwPath := mintGuardManager(t, false)
if err := m.WriteOffboxSecrets("K", "kh"); err != nil {
t.Fatal(err)
}
before, err := os.ReadFile(pwPath)
if err != nil {
t.Fatal(err)
}
// Now the hub starts holding a package (the ceremony ran) and the target is re-applied.
if err := m.settings.SetHubEscrowIdentityPresent(true); err != nil {
t.Fatal(err)
}
if err := m.WriteOffboxSecrets("K2", "kh2"); err != nil {
t.Fatalf("a re-apply on a box that already has a key must not be refused: %v", err)
}
after, err := os.ReadFile(pwPath)
if err != nil {
t.Fatal(err)
}
if string(before) != string(after) {
t.Error("the existing repository password must never be rotated by an apply")
}
if m.OffboxAwaitingRecoveryKey() {
t.Error("a box holding its key is not awaiting one")
}
}
// Fail-safe: an unreadable settings store must not block a tier. A transient read failure turning
// into a permanently-held tier is a worse defect than the one being fixed.
func TestR241_NilSettingsDoesNotBlockTheMint(t *testing.T) {
logger := log.New(os.Stderr, "", 0)
dataDir := t.TempDir()
cfg := &config.Config{}
cfg.Paths.DataDir = dataDir
m := NewManager(cfg, nil, logger)
if m.sealedPackageHeld() {
t.Fatal("a nil settings store must read as 'no package held' — fail toward letting the box work")
}
}
// Scenario E's carve-out, pinned for the HOLDING state too. A customer who switched off-site off is
// not awaiting a recovery key, and must not declare one. The first draft of
// OffboxAwaitingRecoveryKey omitted `t.Enabled` and TestOffsiteDeclare_DisabledTargetIsNotStranded
// caught it; this test pins the same invariant from the new predicate's own side, so a future edit
// to THIS function fails here rather than in a neighbouring file.
func TestR241_DisabledTargetIsNotAwaitingAnything(t *testing.T) {
m, sett, _ := mintGuardManager(t, true) // the hub holds a package, and there is no key
if err := sett.SetOffboxTarget(&settings.OffboxTarget{
Enabled: false, Host: "nas.local", Port: 22, User: "felhom", RepoPath: "/srv/repo",
}); err != nil {
t.Fatal(err)
}
if m.OffboxAwaitingRecoveryKey() {
t.Fatal("a deliberately DISABLED target must never declare the holding state (Scenario E)")
}
if st := m.OffboxReportStatus(); st != nil {
t.Fatalf("a disabled target must stay silent in the report, got %+v", st)
}
}
@@ -0,0 +1,143 @@
package backup
import (
"os"
"path/filepath"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-241 shape (c) — the recovery offer is driven by the comparison the box already makes.
//
// Scenarios C and D from the task, plus §7.2's two staleness cases. The point of shape (c) is that
// it asks the real question — *does the hub hold a package for a key other than the one I am
// using?* — rather than the two proxies that have each now been wrong in opposite directions.
// offerFixture builds a manager holding a repository password, with the hub's cached facts settable.
// Returns the local key's hash so a test can make the hub's hash match or differ deliberately.
func offerFixture(t *testing.T, hubHoldsPackage bool) (*Manager, *settings.Settings, string) {
t.Helper()
m, sett, pwPath := mintGuardManager(t, false) // mint freely first
if err := sett.SetOffboxTarget(&settings.OffboxTarget{
Enabled: true, Host: "nas.local", Port: 22, User: "felhom", RepoPath: "/srv/repo", Schedule: "daily",
}); err != nil {
t.Fatal(err)
}
if err := m.WriteOffboxSecrets("KEYMATERIAL", "nas.local ssh-ed25519 HOSTKEY"); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(pwPath); err != nil {
t.Fatalf("fixture should hold a repository password: %v", err)
}
local, ok := m.OffboxRepoPasswordHash()
if !ok {
t.Fatal("fixture should be able to hash its own key")
}
if err := sett.SetHubEscrowIdentityPresent(hubHoldsPackage); err != nil {
t.Fatal(err)
}
return m, sett, local
}
const otherKeyHash = "9b4a9a9dcec7898e7544f35b18470aac77c3d9064e5d3a302897617fa62edd65"
// ── SCENARIO C — a differing key offers recovery, whatever the reason for the difference ────────
//
// This is the venue's exact state on 2026-08-07: a key present, no orphan recorded, escrow stuck
// pending — and before shape (c), silence.
func TestR241_ScenarioC_DifferingKeyOffersRecovery(t *testing.T) {
m, sett, local := offerFixture(t, true)
if local == otherKeyHash {
t.Fatal("fixture precondition: the local key must differ from the hub's")
}
if err := sett.SetHubEscrowKeySHA256(otherKeyHash, "2026-08-07T03:28:03Z"); err != nil {
t.Fatal(err)
}
// Neither proxy fires: a key EXISTS (so not shape (a)) and nothing is orphaned (so not shape (b)).
if _, ok := m.OffboxRepoPasswordHash(); !ok {
t.Fatal("precondition: shape (a) must be false")
}
if m.OffboxOrphaned() {
t.Fatal("precondition: shape (b) must be false")
}
if !m.OffsiteRecoveryOffer() {
t.Fatal("R-241: the hub holds a package for a DIFFERENT key and the screen was not offered — this is the defect")
}
}
// ── SCENARIO D — a healthy box is never offered recovery ────────────────────────────────────────
//
// RED-PROOF: drop the `hubHash != localHash` conjunct in shape (c) (make it `hubHash != ""`). A
// healthy box is then offered recovery forever, and this test fails — which is how a screen stops
// being read.
func TestR241_ScenarioD_MatchingKeyOffersNothing(t *testing.T) {
m, sett, local := offerFixture(t, true)
if err := sett.SetHubEscrowKeySHA256(local, "2026-08-07T09:00:00Z"); err != nil {
t.Fatal(err)
}
if m.OffsiteRecoveryOffer() {
t.Fatal("a box whose key the hub's package covers must never be offered recovery")
}
}
// A box the hub holds nothing for is never offered, even if a stale hash lingers in settings. Fact 1
// stays required — the spike's comment block calls dropping it "the plausible wrong fix".
func TestR241_ShapeC_NeverHadOffsiteIsStillSilent(t *testing.T) {
m, sett, _ := offerFixture(t, false) // the hub holds NOTHING
if err := sett.SetHubEscrowKeySHA256(otherKeyHash, "2026-08-07T09:00:00Z"); err != nil {
t.Fatal(err)
}
if m.OffsiteRecoveryOffer() {
t.Fatal("a box that never had off-site backups must never be greeted by a recovery screen")
}
}
// ── §7.2 — the staleness decision, both halves ──────────────────────────────────────────────────
// A KNOWN DIFFERENCE OFFERS, however old the reading. Age is deliberately not gated on: gating would
// make a box offline from the hub silently stop offering, which is the failure this session exists
// to remove.
func TestR241_StaleComparison_KnownDifferenceStillOffers(t *testing.T) {
m, sett, _ := offerFixture(t, true)
if err := sett.SetHubEscrowKeySHA256(otherKeyHash, "2020-01-01T00:00:00Z"); err != nil { // ancient
t.Fatal(err)
}
if !m.OffsiteRecoveryOffer() {
t.Fatal("a known difference must offer regardless of how old the reading is (§7.2)")
}
}
// AN ABSENT HASH FALLS BACK TO (a)/(b) — it does not offer. "" is the hub positively saying its
// package seals no repository password (legacy hash-less escrow); there is nothing to compare, and
// offering would put a permanent screen in front of every legacy box.
func TestR241_StaleComparison_AbsentHashFallsBackAndDoesNotOffer(t *testing.T) {
m, sett, _ := offerFixture(t, true)
if err := sett.SetHubEscrowKeySHA256("", ""); err != nil {
t.Fatal(err)
}
if m.OffsiteRecoveryOffer() {
t.Fatal("a hash never learned must fall back to (a)/(b), not offer (§7.2)")
}
// ...and the fallback still works: mark the repo orphaned and shape (b) fires as before.
if err := m.settings.UpdateOffboxStatus(func(o *settings.OffboxTarget) { o.RepoState = "orphaned" }); err != nil {
t.Fatal(err)
}
if !m.OffsiteRecoveryOffer() {
t.Fatal("shape (b) must still work when the hub's hash was never learned")
}
}
// Shape (a) is untouched: a box with no key at all is still offered, which is the pristine rebuild.
func TestR241_ShapeAStillWorks(t *testing.T) {
m, sett, _ := offerFixture(t, true)
if err := os.Remove(filepath.Join(m.cfg.Paths.DataDir, "offbox", "repo_password")); err != nil {
t.Fatal(err)
}
if err := sett.SetHubEscrowKeySHA256(otherKeyHash, "2026-08-07T09:00:00Z"); err != nil {
t.Fatal(err)
}
if !m.OffsiteRecoveryOffer() {
t.Fatal("shape (a) — no repository password at all — must still offer")
}
}
@@ -197,7 +197,14 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string) (Of
hdd := strings.TrimSpace(m.stackProvider.GetStackHDDPath(stack)) hdd := strings.TrimSpace(m.stackProvider.GetStackHDDPath(stack))
if hdd == "" { if hdd == "" {
return res, fmt.Errorf("a(z) %s nincs telepítve — előbb állítsd helyre az alkalmazást, utána az adatokat", stack) // R-253: the same sentence the restore page now shows, so the page and the refusal cannot
// drift apart again. It is a REFUSAL, not a failure — the data is untouched and the customer
// has one step to take. The restore deliberately does NOT deploy the app itself: the
// destination is the app's own HDD path, which is a drive the CUSTOMER chooses at deploy
// time, and picking it for them is the decision this whole recovery path exists to leave
// with them.
return res, fmt.Errorf("a(z) %s nincs telepítve, ezért nincs hová visszaállítani az adatait — "+
"telepítsd újra az alkalmazást (Alkalmazások), utána ez a visszaállítás működni fog", stack)
} }
liveNs := m.namespaceRoot(hdd) liveNs := m.namespaceRoot(hdd)
@@ -276,6 +283,22 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string) (Of
} }
// --- FILES ---------------------------------------------------------------------------------- // --- FILES ----------------------------------------------------------------------------------
// R-166: mark the stop→restore→start window BEFORE stopping. A controller killed anywhere inside
// it used to leave the app down with nothing on disk recording that it was owed a restart — and a
// full offsite restore is a LONG window, so this is the shape most likely to be interrupted.
if err := m.appStop.Begin("offbox-reconstitute:"+stack, ReasonOffboxReconstitute, []string{stack}); err != nil {
return res, fmt.Errorf("a(z) %s leállítása előtti jelölő nem menthető: %w", stack, err)
}
// restartStack starts the app and clears the marker ONLY when the start actually succeeded — a
// failed start leaves the marker so the next startup retries. Every bring-up below goes through
// it; a bare StartStack here would clear nothing and strand the marker on the success path.
restartStack := func() error {
err := m.stackProvider.StartStack(stack)
if err == nil {
m.appStop.End()
}
return err
}
if err := m.stackProvider.StopStack(stack); err != nil { if err := m.stackProvider.StopStack(stack); err != nil {
m.logger.Printf("[WARN] [offbox] could not stop %s before reconstitution: %v (continuing)", stack, err) m.logger.Printf("[WARN] [offbox] could not stop %s before reconstitution: %v (continuing)", stack, err)
} }
@@ -291,7 +314,7 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string) (Of
if cErr != nil { if cErr != nil {
// Best-effort bring-up: leaving the app stopped after a partial copy would turn a failed // Best-effort bring-up: leaving the app stopped after a partial copy would turn a failed
// restore into an outage. // restore into an outage.
if sErr := m.stackProvider.StartStack(stack); sErr != nil { if sErr := restartStack(); sErr != nil {
m.logger.Printf("[WARN] [offbox] %s: restart after failed placement also failed: %v", stack, sErr) m.logger.Printf("[WARN] [offbox] %s: restart after failed placement also failed: %v", stack, sErr)
} }
return res, fmt.Errorf("a(z) %s fájljainak visszaállítása sikertelen: %w", stack, cErr) return res, fmt.Errorf("a(z) %s fájljainak visszaállítása sikertelen: %w", stack, cErr)
@@ -308,7 +331,7 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string) (Of
if hasDB { if hasDB {
if err := m.stackProvider.StartStackServices(stack, dbServices); err != nil { if err := m.stackProvider.StartStackServices(stack, dbServices); err != nil {
// Best-effort bring-up: a failed restore must not also be an outage. // Best-effort bring-up: a failed restore must not also be an outage.
if sErr := m.stackProvider.StartStack(stack); sErr != nil { if sErr := restartStack(); sErr != nil {
m.logger.Printf("[WARN] [offbox] %s: full start after failed DB-only start also failed: %v", stack, sErr) m.logger.Printf("[WARN] [offbox] %s: full start after failed DB-only start also failed: %v", stack, sErr)
} }
return res, fmt.Errorf("a(z) %s adatbázis-szolgáltatásának indítása sikertelen: %w", stack, err) return res, fmt.Errorf("a(z) %s adatbázis-szolgáltatásának indítása sikertelen: %w", stack, err)
@@ -316,13 +339,13 @@ func (m *Manager) ReconstituteFromOffsite(ctx context.Context, stack string) (Of
n, iErr := m.reimportDBDumpsFrom(ctx, stack, scratchDumpDir) n, iErr := m.reimportDBDumpsFrom(ctx, stack, scratchDumpDir)
res.DBsReplayed = n res.DBsReplayed = n
if iErr != nil { if iErr != nil {
if sErr := m.stackProvider.StartStack(stack); sErr != nil { if sErr := restartStack(); sErr != nil {
m.logger.Printf("[WARN] [offbox] %s: full start after failed replay also failed: %v", stack, sErr) m.logger.Printf("[WARN] [offbox] %s: full start after failed replay also failed: %v", stack, sErr)
} }
return res, fmt.Errorf("az adatbázis visszaállítása sikertelen: %w — a korábbi állapot mentése megvan: %s", iErr, filepath.Base(safety)) return res, fmt.Errorf("az adatbázis visszaállítása sikertelen: %w — a korábbi állapot mentése megvan: %s", iErr, filepath.Base(safety))
} }
} }
if err := m.stackProvider.StartStack(stack); err != nil { if err := restartStack(); err != nil {
return res, fmt.Errorf("a(z) %s újraindítása sikertelen a fájlok visszaállítása után: %w", stack, err) return res, fmt.Errorf("a(z) %s újraindítása sikertelen a fájlok visszaállítása után: %w", stack, err)
} }
if err := m.waitForHealthy(stack, 90*time.Second); err != nil { if err := m.waitForHealthy(stack, 90*time.Second); err != nil {
@@ -0,0 +1,74 @@
package backup
import (
"context"
"fmt"
"strings"
)
// R-200 (controller v0.195.0) — THE DIAGNOSTIC HALF, and only that half.
//
// The question this answers, once, decisively: **is the offsite repository password actually
// recoverable from the hub's sealed bundle?** Everything else in the recovery chain is downstream of
// that, and until 2026-08-04 nobody had ever asked it — the round-trip proof on record (2026-06-10)
// predates the field by a month, and the extraction step did not exist at all.
//
// IT COMPARES; IT DOES NOT INSTALL. The recovered password is NOT written to offboxPwPath. Comparing
// proves recoverability; installing changes a live box's state on a path nobody has walked, and
// "the existing repository opens under a recovered key" is a separate link with a drill around it.
// Keep this function free of any write — if a future change makes it install, it stops being a
// diagnostic and needs the drill's supervision.
//
// IT HANDLES ONLY HASHES OUTSIDE THE AGENT CALL. The agent returns the password and its sha256; this
// reads the hash. The value is dropped on the floor here deliberately, so no controller-side code
// path can grow a habit of holding it.
// OffsiteKeyRecoverer is the agent-side seam (agent >= v0.125.0,
// POST /escrow/recover-offsite-password): it fetches this host's sealed bundle from the hub, unseals
// it with R, and returns ONLY the offsite repository password plus its sha256.
type OffsiteKeyRecoverer interface {
RecoverOffsiteRepoPassword(ctx context.Context, recoveryCode string) (password, sha256hex string, err error)
}
// RecoveryCheckResult is the verdict. It carries HASHES ONLY — there is no field here that could
// leak a password into a log, a report or a terminal.
type RecoveryCheckResult struct {
// LocalSHA256 is the hash of the repo password currently on disk ("" when there is none).
LocalSHA256 string
// RecoveredSHA256 is the hash of what came out of the sealed bundle.
RecoveredSHA256 string
// Match is the whole point: byte-identical keys produce identical hashes.
Match bool
// LocalPresent distinguishes "they differ" from "there was nothing to compare against" — a
// rebuilt box with no repo password yet is a legitimate state and must not read as a mismatch.
LocalPresent bool
}
// CheckOffsiteKeyRecoverable recovers the repository password through the agent and compares it, by
// hash, against the one on this box's disk. It writes nothing anywhere.
//
// R is passed straight through to the agent and is not retained here. The CALLER owns clearing its
// own copy; this function keeps none.
func (m *Manager) CheckOffsiteKeyRecoverable(ctx context.Context, rec OffsiteKeyRecoverer, recoveryCode string) (RecoveryCheckResult, error) {
var out RecoveryCheckResult
if rec == nil {
return out, fmt.Errorf("offbox: no agent recovery seam configured")
}
if strings.TrimSpace(recoveryCode) == "" {
return out, fmt.Errorf("offbox: the recovery code is required")
}
// Read the local side FIRST, so a missing local password is reported as such rather than
// surfacing as a mismatch after a successful recovery.
localHash, ok := m.OffboxRepoPasswordHash()
out.LocalSHA256, out.LocalPresent = localHash, ok
pw, recoveredHash, err := rec.RecoverOffsiteRepoPassword(ctx, recoveryCode)
if err != nil {
return out, err // the agent's message already names the step and contains no secret
}
pw = "" // the VALUE is not this function's business — §8.5, compare, do not install
_ = pw
out.RecoveredSHA256 = recoveredHash
out.Match = ok && recoveredHash != "" && recoveredHash == localHash
return out, nil
}
@@ -0,0 +1,349 @@
package backup
import (
"bytes"
"context"
"errors"
"log"
"os"
"path/filepath"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-200 — the diagnostic half. What is asserted here is the VERDICT and the NON-WRITE, because those
// are the two things that make this a proof rather than a change to a live box.
type fakeRecoverer struct {
pw, sha string
err error
gotCode string
callable bool
}
func (f *fakeRecoverer) RecoverOffsiteRepoPassword(_ context.Context, code string) (string, string, error) {
f.callable = true
f.gotCode = code
return f.pw, f.sha, f.err
}
// Scenario A at this layer — the recovered key's hash is compared against the on-disk one and the
// verdict is the equality, not "no error".
func TestCheckOffsiteKeyRecoverable_MatchAndMismatch(t *testing.T) {
m, _ := newOffboxManager(t)
localHash, ok := m.OffboxRepoPasswordHash()
if !ok {
t.Fatal("precondition: no local repo password")
}
// The key came back identical.
res, err := m.CheckOffsiteKeyRecoverable(context.Background(), &fakeRecoverer{pw: "irrelevant", sha: localHash}, "R")
if err != nil {
t.Fatal(err)
}
if !res.Match || !res.LocalPresent || res.RecoveredSHA256 != localHash || res.LocalSHA256 != localHash {
t.Fatalf("identical keys must report Match: %+v", res)
}
// A DIFFERENT key must report a mismatch, not an error — "it worked and disagreed" is a finding
// about the system and must be distinguishable from "a step failed".
res, err = m.CheckOffsiteKeyRecoverable(context.Background(), &fakeRecoverer{pw: "x", sha: "0000000000000000000000000000000000000000000000000000000000000000"}, "R")
if err != nil {
t.Fatalf("a mismatch is a verdict, not an error: %v", err)
}
if res.Match {
t.Fatal("a different recovered key must NOT report Match")
}
}
// A box with no local password reports that distinctly — it is the rebuilt-box shape, where the next
// step is to install rather than to compare, and reading it as a mismatch would be wrong.
func TestCheckOffsiteKeyRecoverable_NoLocalPassword(t *testing.T) {
m := newBareManager(t)
res, err := m.CheckOffsiteKeyRecoverable(context.Background(), &fakeRecoverer{pw: "x", sha: "abc"}, "R")
if err != nil {
t.Fatal(err)
}
if res.LocalPresent || res.Match {
t.Fatalf("no local key must report LocalPresent=false and Match=false: %+v", res)
}
if res.RecoveredSHA256 != "abc" {
t.Fatalf("the recovery itself succeeded and must be reported: %+v", res)
}
}
// §8.5 — THE CHECK MUST NOT INSTALL. This is the assertion that keeps a diagnostic a diagnostic.
// RED-PROOF: add `m.InjectOffboxPassword(pw, true)` to CheckOffsiteKeyRecoverable → the on-disk
// password changes → this FAILS.
func TestCheckOffsiteKeyRecoverable_WritesNothing(t *testing.T) {
m, _ := newOffboxManager(t)
before, err := os.ReadFile(m.offboxPwPath())
if err != nil {
t.Fatal(err)
}
dir := m.offboxDir()
beforeEntries, _ := os.ReadDir(dir)
recovered := "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"
if _, err := m.CheckOffsiteKeyRecoverable(context.Background(), &fakeRecoverer{pw: recovered, sha: HashResticPassword(recovered)}, "R"); err != nil {
t.Fatal(err)
}
after, err := os.ReadFile(m.offboxPwPath())
if err != nil {
t.Fatal(err)
}
if !bytes.Equal(before, after) {
t.Fatal("the check INSTALLED the recovered password — it must compare and never write (§8.5); " +
"installing changes a live box on a path nobody has walked")
}
afterEntries, _ := os.ReadDir(dir)
if len(afterEntries) != len(beforeEntries) {
var names []string
for _, e := range afterEntries {
names = append(names, e.Name())
}
t.Fatalf("the check created files in the offbox dir: %v", names)
}
// And nothing leaked into the data dir either.
_ = filepath.Walk(m.cfg.Paths.DataDir, func(p string, info os.FileInfo, werr error) error {
if werr != nil || info == nil || info.IsDir() {
return nil
}
body, rerr := os.ReadFile(p)
if rerr == nil && strings.Contains(string(body), recovered) {
t.Errorf("the recovered password was written to %s", p)
}
return nil
})
}
// R goes to the agent verbatim and is not mangled or retained by this layer.
func TestCheckOffsiteKeyRecoverable_PassesRThrough(t *testing.T) {
m, _ := newOffboxManager(t)
const code = "correct horse battery staple sedative anaconda wobbly kingdom placard yodel"
f := &fakeRecoverer{pw: "x", sha: "abc"}
if _, err := m.CheckOffsiteKeyRecoverable(context.Background(), f, code); err != nil {
t.Fatal(err)
}
if f.gotCode != code {
t.Fatalf("the recovery code reached the agent as %q — a 10-word code must not be re-split or trimmed internally", f.gotCode)
}
}
// An agent-side failure surfaces as an error, and the verdict is NOT reported as a mismatch.
func TestCheckOffsiteKeyRecoverable_AgentFailure(t *testing.T) {
m, _ := newOffboxManager(t)
_, err := m.CheckOffsiteKeyRecoverable(context.Background(), &fakeRecoverer{err: errors.New("the recovery code did not open the sealed bundle")}, "R")
if err == nil {
t.Fatal("an agent failure must be an error, never a silent mismatch")
}
}
// The CLI's exit codes are load-bearing: 0 match, 2 clean mismatch, 1 a step failed. "It failed" and
// "it worked and disagreed" must never share a status, because only one of them is a finding.
func TestRunRecoveryCheck_ExitCodes(t *testing.T) {
m, _ := newOffboxManager(t)
localHash, _ := m.OffboxRepoPasswordHash()
cases := []struct {
name string
rec OffsiteKeyRecoverer
in string
want int
}{
{"match", &fakeRecoverer{pw: "x", sha: localHash}, "some recovery code\n", 0},
{"mismatch", &fakeRecoverer{pw: "x", sha: "0000000000000000000000000000000000000000000000000000000000000000"}, "some recovery code\n", 2},
{"agent failure", &fakeRecoverer{err: errors.New("wrong code")}, "some recovery code\n", 1},
{"no code on stdin", &fakeRecoverer{pw: "x", sha: localHash}, "", 1},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
var out, errb bytes.Buffer
got := RunRecoveryCheck(RecoveryCheckDeps{
Manager: m, Recoverer: tc.rec, In: strings.NewReader(tc.in), Out: &out, Err: &errb,
})
if got != tc.want {
t.Fatalf("exit = %d, want %d (out=%q err=%q)", got, tc.want, out.String(), errb.String())
}
// No printed stream may ever carry a password or a recovery code.
combined := out.String() + errb.String()
for _, secret := range []string{"PRIVATE-KEY-MATERIAL", "some recovery code"} {
if strings.Contains(combined, secret) {
t.Errorf("the diagnostic printed a secret (%s): %s", secret, combined)
}
}
})
}
}
// newBareManager is an offbox manager with a data dir and NO repo password — the freshly rebuilt-box
// shape, which the no-local-password case needs and newOffboxManager deliberately does not produce.
func newBareManager(t *testing.T) *Manager {
t.Helper()
logger := log.New(os.Stderr, "", 0)
dataDir := t.TempDir()
sett, err := settings.Load(filepath.Join(dataDir, "settings.json"), logger)
if err != nil {
t.Fatal(err)
}
cfg := &config.Config{}
cfg.Paths.DataDir = dataDir
cfg.Paths.SystemDataPath = filepath.Join(dataDir, "sys")
return NewManager(cfg, sett, logger)
}
// R-200 Part 0 — the INSTALL sibling. What is asserted is the three outcomes, the confirmation gate,
// and that R does not survive either path.
// An install on a box with NO local password writes it — the rebuilt-box shape, which is the only
// situation this command exists for.
// RED-PROOF: drop the `confirm` check so an unconfirmed run installs → the dry-run case below FAILS.
func TestRecoverAndInstall_InstallsOnABareBox(t *testing.T) {
m := newBareManager(t)
pw := "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
rec := &fakeRecoverer{pw: pw, sha: HashResticPassword(pw)}
// 1) DRY RUN — prints the hashes, writes nothing.
var out, errb bytes.Buffer
if got := RecoverAndInstall(RecoveryCheckDeps{Manager: m, Recoverer: rec, In: strings.NewReader("code\n"), Out: &out, Err: &errb}, false); got != 0 {
t.Fatalf("dry run exit = %d, want 0 (%s / %s)", got, out.String(), errb.String())
}
if _, present := m.OffboxRepoPasswordHash(); present {
t.Fatal("the DRY RUN wrote the password — the confirmation gate does not hold, which is the " +
"whole reason the operator gets to see the hashes before anything exists to undo")
}
if !strings.Contains(out.String(), "DRY RUN") {
t.Errorf("the dry run must say so, got %q", out.String())
}
// 2) CONFIRMED — writes it, and it reads back identical.
out.Reset()
errb.Reset()
if got := RecoverAndInstall(RecoveryCheckDeps{Manager: m, Recoverer: rec, In: strings.NewReader("code\n"), Out: &out, Err: &errb}, true); got != 0 {
t.Fatalf("confirmed exit = %d, want 0 (%s / %s)", got, out.String(), errb.String())
}
got, present := m.OffboxRepoPasswordHash()
if !present || got != HashResticPassword(pw) {
t.Fatalf("the recovered password was not placed (present=%v hash=%q)", present, got)
}
if !strings.Contains(out.String(), "INSTALLED") {
t.Errorf("a successful install must say so, got %q", out.String())
}
// The VALUE must not have been printed on either stream.
if strings.Contains(out.String()+errb.String(), pw) {
t.Fatal("the repository password was printed")
}
}
// An identical key already present is "unchanged", not "installed" and not an error — and nothing is
// written, so a re-run is harmless.
func TestRecoverAndInstall_UnchangedWhenIdentical(t *testing.T) {
m, _ := newOffboxManager(t)
localHash, _ := m.OffboxRepoPasswordHash()
before, err := os.ReadFile(m.offboxPwPath())
if err != nil {
t.Fatal(err)
}
var out, errb bytes.Buffer
got := RecoverAndInstall(RecoveryCheckDeps{
Manager: m, Recoverer: &fakeRecoverer{pw: "x", sha: localHash},
In: strings.NewReader("code\n"), Out: &out, Err: &errb,
}, true)
if got != 0 {
t.Fatalf("exit = %d, want 0", got)
}
if !strings.Contains(out.String(), "UNCHANGED") {
t.Errorf("an identical key must report UNCHANGED, got %q", out.String())
}
after, _ := os.ReadFile(m.offboxPwPath())
if !bytes.Equal(before, after) {
t.Fatal("an UNCHANGED outcome rewrote the file")
}
}
// A DIFFERENT key already present is REFUSED — installing would clobber the key the box's current
// repository is encrypted under, and which history to keep is not this command's decision.
func TestRecoverAndInstall_RefusesToClobberADifferentKey(t *testing.T) {
m, _ := newOffboxManager(t)
before, err := os.ReadFile(m.offboxPwPath())
if err != nil {
t.Fatal(err)
}
var out, errb bytes.Buffer
got := RecoverAndInstall(RecoveryCheckDeps{
Manager: m,
Recoverer: &fakeRecoverer{pw: "y", sha: "0000000000000000000000000000000000000000000000000000000000000000"},
In: strings.NewReader("code\n"), Out: &out, Err: &errb,
}, true)
if got != 2 {
t.Fatalf("exit = %d, want 2 (a refusal is its own outcome, not a generic failure)", got)
}
if !strings.Contains(errb.String(), "REFUSED") {
t.Errorf("the refusal must say so, got %q", errb.String())
}
after, _ := os.ReadFile(m.offboxPwPath())
if !bytes.Equal(before, after) {
t.Fatal("a REFUSED install clobbered the existing key — the exact outcome the refusal exists to prevent")
}
}
// R must not survive either path, and the recovery code must never be printed.
func TestRecoverAndInstall_RLeavesNoTrace(t *testing.T) {
const code = "correct horse battery staple sedative anaconda wobbly kingdom placard yodel"
for _, tc := range []struct {
name string
confirm bool
}{{"dry run", false}, {"confirmed", true}} {
t.Run(tc.name, func(t *testing.T) {
m := newBareManager(t)
pw := "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
var out, errb bytes.Buffer
RecoverAndInstall(RecoveryCheckDeps{
Manager: m, Recoverer: &fakeRecoverer{pw: pw, sha: HashResticPassword(pw)},
In: strings.NewReader(code + "\n"), Out: &out, Err: &errb,
}, tc.confirm)
combined := out.String() + errb.String()
if strings.Contains(combined, code) {
t.Errorf("the recovery code was printed: %s", combined)
}
if strings.Contains(combined, pw) {
t.Errorf("the repository password was printed: %s", combined)
}
// POSITIVE CONTROL for the sweep below: plant R in the data dir, prove the walk finds it,
// remove it. An absence check is worth only what its sensitivity is.
ctrl := filepath.Join(m.cfg.Paths.DataDir, ".planted-control")
if err := os.WriteFile(ctrl, []byte(code), 0o600); err != nil {
t.Fatal(err)
}
if n := countFilesContaining(t, m.cfg.Paths.DataDir, code); n != 1 {
t.Fatalf("positive control: the sweep found %d planted copies, want 1 — the sweep is not sensitive", n)
}
if err := os.Remove(ctrl); err != nil {
t.Fatal(err)
}
if n := countFilesContaining(t, m.cfg.Paths.DataDir, code); n != 0 {
t.Fatalf("the recovery code survived in %d file(s) under the data dir", n)
}
})
}
}
func countFilesContaining(t *testing.T, root, needle string) int {
t.Helper()
n := 0
_ = filepath.Walk(root, func(p string, info os.FileInfo, err error) error {
if err != nil || info == nil || info.IsDir() {
return nil
}
body, rerr := os.ReadFile(p)
if rerr == nil && strings.Contains(string(body), needle) {
n++
}
return nil
})
return n
}
@@ -0,0 +1,296 @@
package backup
import (
"bufio"
"context"
"fmt"
"io"
"os"
"strings"
"time"
)
// R-200 — the operator-facing entry point for the recovery check, and the ONLY one this session
// ships. Deliberately a `docker exec` escape hatch in the shape of `--print-reset-code`, not a page,
// a card or an API a browser can reach: the customer-facing flow is designed on top of a chain that
// has been walked, and this is the walk.
//
// WHY R COMES FROM STDIN AND NOT A FLAG. A flag value is visible in `ps`, in the shell history, in a
// container's command line and in any transcript of the session that ran it. R is the one secret in
// this system that cannot be rotated, re-issued or recovered. It is read from stdin, held in one
// string, and cleared before the function returns — on the success path and on every failure path.
//
// docker exec -i felhom-controller /app/felhom-controller --recover-offsite-check < /root/r.txt
//
// WHAT IT PRINTS: two sha256 hashes and a verdict. Never a password, never R, never a blob. The
// hashes are of 256-bit random secrets and are non-reversible — the same value the hub already stores
// and serves in report ACKs.
// RecoveryCheckDeps is what the CLI needs; injected so the entry point is testable without a live
// agent, a live hub or real crypto.
type RecoveryCheckDeps struct {
// Manager owns the on-disk repo password hash.
Manager *Manager
// Recoverer is the agent seam (agentapi.Client satisfies it).
Recoverer OffsiteKeyRecoverer
// In is where R is read from (os.Stdin in production).
In io.Reader
// Out / Err are the report streams (os.Stdout / os.Stderr in production).
Out, Err io.Writer
// Timeout bounds the whole check. 0 → 90s (an unseal shells out to age and a fetch crosses the WAN).
Timeout time.Duration
}
// RunRecoveryCheck reads R from stdin, recovers the offsite repository password through the agent,
// and reports whether it matches the one on disk — BY HASH. Returns a process exit code:
//
// 0 = the hashes matched (the key is recoverable)
// 1 = a step failed (fetch, unseal, or no local password to compare against)
// 2 = the check ran cleanly and the hashes DIFFER — the loud case, and the one that would mean the
// sealed bundle does not carry what four weeks of documents say it carries
//
// A distinct code for the mismatch on purpose: "it failed" and "it worked and disagreed" must never
// share an exit status, because only one of them is a finding about the system rather than about the
// run.
func RunRecoveryCheck(d RecoveryCheckDeps) int {
out, errw := d.Out, d.Err
if out == nil {
out = os.Stdout
}
if errw == nil {
errw = os.Stderr
}
if d.Manager == nil || d.Recoverer == nil {
fmt.Fprintln(errw, "recover-offsite-check: not configured (no backup manager or no agent channel)")
return 1
}
in := d.In
if in == nil {
in = os.Stdin
}
// Read R: the first line of stdin, trimmed. A 10-word EFF code contains spaces, so only the
// line ending is stripped — never internal whitespace.
br := bufio.NewReader(io.LimitReader(in, 4096))
line, rerr := br.ReadString('\n')
R := strings.TrimRight(line, "\r\n")
if R == "" {
fmt.Fprintln(errw, "recover-offsite-check: no recovery code on stdin. Pipe it in:")
fmt.Fprintln(errw, " docker exec -i felhom-controller /app/felhom-controller --recover-offsite-check < /path/to/code")
if rerr != nil && rerr != io.EOF {
fmt.Fprintf(errw, " (read error: %v)\n", rerr)
}
return 1
}
timeout := d.Timeout
if timeout == 0 {
timeout = 90 * time.Second
}
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
fmt.Fprintln(out, "=== offsite key recovery check (R-200) — compares, never installs ===")
res, err := d.Manager.CheckOffsiteKeyRecoverable(ctx, d.Recoverer, R)
R = "" // cleared before anything else, on every path below
if err != nil {
fmt.Fprintf(errw, " [FAIL] %v\n", err) // the agent's message names the step; it carries no secret
fmt.Fprintln(errw, " nothing was written.")
return 1
}
if !res.LocalPresent {
fmt.Fprintln(errw, " [FAIL] there is no repository password on this box to compare against")
fmt.Fprintf(out, " recovered sha256: %s\n", res.RecoveredSHA256)
fmt.Fprintln(errw, " (the recovery itself SUCCEEDED — this box simply has no local key. That is the")
fmt.Fprintln(errw, " rebuilt-box shape, where the next step is to INSTALL rather than compare.)")
return 1
}
fmt.Fprintf(out, " on-disk sha256: %s\n", res.LocalSHA256)
fmt.Fprintf(out, " recovered sha256: %s\n", res.RecoveredSHA256)
if !res.Match {
fmt.Fprintln(errw, " [MISMATCH] the recovered key is NOT the key this box uses.")
fmt.Fprintln(errw, " This is a finding about the system, not about the run: the sealed bundle does not")
fmt.Fprintln(errw, " carry the repository password this box's off-site history is encrypted under.")
return 2
}
fmt.Fprintln(out, " [MATCH] the offsite repository password IS recoverable from the sealed escrow.")
fmt.Fprintln(out, " Nothing was written: this check compares and never installs.")
return 0
}
// RecoverAndInstall is the sibling of RunRecoveryCheck that PLACES the recovered repository password,
// so a rebuilt box can reopen the off-site history it inherited (R-200's remaining plumbing half).
//
// WHY THIS IS CODE AND NOT A MANUAL STEP. The alternative — recover the password, read it off a
// terminal, and paste it into the injection endpoint by hand — puts the offsite DATA key through a
// human's screen, clipboard and shell history. Doing it in-process is both simpler and strictly
// safer: the value goes agent → this process → the 0600 file and is never rendered anywhere.
//
// THE CONFIRMATION IS A SEPARATE INVOCATION, ON PURPOSE. Without `confirm` this prints the two hashes
// and writes nothing — the operator sees the comparison BEFORE any write exists as a possibility.
// With `confirm` it prints the same hashes and then installs. A single interactive prompt would have
// had to share stdin with R, which is where R must not be competing for attention.
//
// THREE OUTCOMES, NAMED DISTINCTLY, because "it did nothing" and "it refused" are different facts:
//
// installed — this box had NO repository password (the rebuilt-box shape). The recovered one is placed.
// unchanged — a password is present and is byte-identical to the recovered one. Nothing is written.
// refused — a password is present and DIFFERS. Installing would clobber the key this box's CURRENT
// repository is encrypted under, so it is refused. No force option is offered here: that
// decision needs a human who knows which history they intend to keep.
// RecoverInstallOutcome names the terminal states of a recovery+install. Distinct values because
// "it did nothing", "it refused" and "it installed" are different facts and a caller — CLI or web —
// must be able to say which happened without parsing prose.
type RecoverInstallOutcome string
const (
// RecoverInstalled — the box had NO repository password; the recovered one is now in place.
RecoverInstalled RecoverInstallOutcome = "installed"
// RecoverUnchanged — a password was present and is byte-identical to the recovered one.
RecoverUnchanged RecoverInstallOutcome = "unchanged"
// RecoverRefused — a DIFFERENT password is present; installing would clobber the key the box's
// current repository is encrypted under.
RecoverRefused RecoverInstallOutcome = "refused"
// RecoverDryRun — nothing was written because confirm was false.
RecoverDryRun RecoverInstallOutcome = "dry_run"
)
// RecoverInstallResult is the non-secret outcome of a recovery. It carries HASHES ONLY — never the
// password, never R. The hashes are of 256-bit random secrets, non-reversible, and are the same
// values the hub already stores and serves in report ACKs.
type RecoverInstallResult struct {
Outcome RecoverInstallOutcome
LocalPresent bool
LocalSHA256 string
RecoveredSHA256 string
}
// RecoverInstallCore is THE recovery+install path in this codebase — fetch the sealed bundle through
// the agent, unseal it with R, compare against what is on disk, and place it when that is the right
// thing to do.
//
// ONE FUNCTION, TWO CALLERS (R-193). The CLI (`--recover-offsite-install`) and the customer's recovery
// page both call this. They must not each carry a copy: two implementations of the one operation that
// can permanently lose a customer's data would drift, and only one of them would ever be tested.
// `RecoverAndInstall` below is a thin wrapper that maps this result onto the CLI's exit codes and
// printed lines; the web handler maps it onto Hungarian copy. Neither contains recovery logic.
//
// R IS THE CALLER'S TO CLEAR. This function does not retain it: it is passed to the agent seam and
// never stored, logged or returned. The password recovered from the bundle IS cleared here, on every
// path, before returning — it never leaves this function in any form.
//
// The three outcomes and their reasoning are unchanged from the CLI's original implementation; see
// RecoverAndInstall's header, which remains the authority on WHY a differing local password is
// refused rather than forced.
func RecoverInstallCore(ctx context.Context, m *Manager, rec OffsiteKeyRecoverer, R string, confirm bool) (RecoverInstallResult, error) {
var res RecoverInstallResult
if m == nil || rec == nil {
return res, fmt.Errorf("recovery not configured (no backup manager or no agent channel)")
}
pw, recoveredHash, err := rec.RecoverOffsiteRepoPassword(ctx, R)
if err != nil {
return res, err // the agent's message names the step; it carries no secret
}
res.RecoveredSHA256 = recoveredHash
res.LocalSHA256, res.LocalPresent = m.OffboxRepoPasswordHash()
switch {
case res.LocalPresent && res.LocalSHA256 == recoveredHash:
pw = ""
res.Outcome = RecoverUnchanged
return res, nil
case res.LocalPresent:
pw = ""
res.Outcome = RecoverRefused
return res, nil
}
if !confirm {
pw = ""
res.Outcome = RecoverDryRun
return res, nil
}
if err := m.InjectOffboxPassword(pw, false); err != nil {
pw = ""
return res, fmt.Errorf("placing the recovered password: %w", err)
}
pw = ""
// Re-read from disk rather than trusting what we just wrote — the observable is the file's state.
afterHash, ok := m.OffboxRepoPasswordHash()
if !ok || afterHash != recoveredHash {
return res, fmt.Errorf("the password was written but does not read back as expected (on-disk %q)", afterHash)
}
res.Outcome = RecoverInstalled
return res, nil
}
func RecoverAndInstall(d RecoveryCheckDeps, confirm bool) int {
out, errw := d.Out, d.Err
if out == nil {
out = os.Stdout
}
if errw == nil {
errw = os.Stderr
}
if d.Manager == nil || d.Recoverer == nil {
fmt.Fprintln(errw, "recover-offsite-install: not configured (no backup manager or no agent channel)")
return 1
}
in := d.In
if in == nil {
in = os.Stdin
}
br := bufio.NewReader(io.LimitReader(in, 4096))
line, rerr := br.ReadString('\n')
R := strings.TrimRight(line, "\r\n")
if R == "" {
fmt.Fprintln(errw, "recover-offsite-install: no recovery code on stdin. Pipe it in:")
fmt.Fprintln(errw, " docker exec -i felhom-controller /usr/local/bin/felhom-controller --recover-offsite-install [--confirm-install] < /path/to/code")
if rerr != nil && rerr != io.EOF {
fmt.Fprintf(errw, " (read error: %v)\n", rerr)
}
return 1
}
timeout := d.Timeout
if timeout == 0 {
timeout = 90 * time.Second
}
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
fmt.Fprintln(out, "=== offsite key recovery INSTALL (R-200) ===")
// THE RECOVERY ITSELF IS RecoverInstallCore — the same function the customer's recovery page
// drives (R-193). This wrapper adds the CLI's stdin handling, its printed lines and its exit
// codes, and NOTHING else; there is exactly one fetch→unseal→compare→install path in this
// codebase and no chance of the two callers drifting. Pinned by
// TestRecoverAndInstall_DrivesTheSharedCore and by the AST wiring test.
res, err := RecoverInstallCore(ctx, d.Manager, d.Recoverer, R, confirm)
R = "" // cleared immediately, on every path below
if err != nil {
fmt.Fprintf(errw, " [FAIL] %v\n", err)
fmt.Fprintln(errw, " nothing was written.")
return 1
}
if res.LocalPresent {
fmt.Fprintf(out, " on-disk sha256: %s\n", res.LocalSHA256)
} else {
fmt.Fprintln(out, " on-disk sha256: (none — this box has no repository password)")
}
fmt.Fprintf(out, " recovered sha256: %s\n", res.RecoveredSHA256)
switch res.Outcome {
case RecoverUnchanged:
fmt.Fprintln(out, " [UNCHANGED] the box already holds exactly this key. Nothing written.")
return 0
case RecoverRefused:
fmt.Fprintln(errw, " [REFUSED] a DIFFERENT repository password is already present.")
fmt.Fprintln(errw, " Installing would clobber the key this box's current repository is encrypted under,")
fmt.Fprintln(errw, " and which history to keep is not a decision this command may take. Nothing written.")
return 2
case RecoverDryRun:
fmt.Fprintln(out, " [DRY RUN] nothing written. The recovered key is ready to install.")
fmt.Fprintln(out, " Re-run with --confirm-install to place it.")
return 0
}
fmt.Fprintln(out, " [INSTALLED] the recovered repository password is in place and reads back identical.")
fmt.Fprintln(out, " Re-apply the offsite target and run a backup: the existing repository should open.")
return 0
}
+30 -1
View File
@@ -171,7 +171,36 @@ func (m *Manager) offboxRestoreScratchDir(stack string) (scratch, nsRoot string,
} }
} }
} }
return "", "", fmt.Errorf("nincs elérhető adatmeghajtó a visszaállításhoz") // R-252: name the reason AND the way to act on it. This refusal is what a rebuilt box hits — the
// drives are physically fine and still mounted, it is their REGISTRATION that the destroyed guest
// took with it — and until v0.207.0 it said only that a drive was missing, which reads like data
// loss and offers nothing to do.
return "", "", fmt.Errorf("nincs regisztrált adatmeghajtó, ezért nincs hová visszaállítani — " +
"a meghajtók megvannak, csak újra kell csatolni őket a Tárhely → Meghajtók oldalon, utána " +
"ez a visszaállítás működni fog")
}
// HasRestoreDestination reports whether an offsite restore has anywhere on this box to write.
//
// R-252: the restore PAGE asks this question through the same helper the resolver answers it with,
// so the notice cannot appear on a box that would restore fine (Scenario E) nor stay hidden on one
// that would refuse. A second copy of the predicate is exactly how a page ends up promising what the
// handler then refuses — which is the neighbouring defect, R-253.
//
// It mirrors the resolver's BOX-level branches (2) and (3) — the schedulable storage paths. Branch
// (1), the app's own HDD path, is deliberately not consulted: an installed app's HDD path IS a
// registered storage path, so the two cannot disagree in practice, and where they could, erring
// toward showing the notice is erring toward telling the customer something true.
func (m *Manager) HasRestoreDestination() bool {
if m.settings == nil {
return false
}
for _, sp := range m.settings.GetSchedulableStoragePaths() {
if strings.TrimSpace(sp.Path) != "" {
return true
}
}
return false
} }
// RestoreOffboxScratch restores an app's latest offsite snapshot to an on-data-drive scratch dir // RestoreOffboxScratch restores an app's latest offsite snapshot to an on-data-drive scratch dir
@@ -0,0 +1,218 @@
package backup
import (
"context"
"os"
"path/filepath"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/appbackup"
)
// R-203 Part 2 — "ok" must mean the mandatory data is in the snapshot.
//
// The defect these pin is NOT that the gap went undetected. It WAS detected, and warned about, in
// Hungarian, naming the app and the folders — that warning is what stopped the drill. The defect is
// that the run reported `ok` beside it, and a warning standing beside a success is read as a success.
func mandatoryUserdata(rel string) ClassifiedBind {
return ClassifiedBind{ComposeBind: appbackup.ComposeBind{Root: appbackup.RootUserdata, RelPath: rel}, Class: appbackup.ClassMandatory}
}
// Scenario C — a MANDATORY declared path absent on disk is a STRUCTURAL gap, not just prose.
//
// RED-PROOF: stop recording capGaps into res.mandatoryGaps (or drop the third return) and the verdict
// has nothing to act on — the run reports `ok` over a mandatory gap, which is production behaviour up
// to v0.196.0.
func TestOffboxCaptureSet_MandatoryGapIsStructural(t *testing.T) {
drive := t.TempDir()
m, _, prov := classifiedOffboxManager(t, drive)
prov.hdd["calibre-web"] = drive
prov.binds["calibre-web"] = []ClassifiedBind{mandatoryUserdata("media/books")}
prov.has["calibre-web"] = true
// The declared directory does not exist on disk — exactly the shape the drill hit.
extra, warns, gaps := m.offboxCaptureSet("calibre-web")
if len(gaps) != 1 || gaps[0] != "media/books" {
t.Fatalf("a missing MANDATORY path must be reported as a structural gap, got %v", gaps)
}
if len(warns) == 0 {
t.Error("the customer-facing Hungarian warning must SURVIVE this change — it is what caught the defect")
}
if len(extra) != 0 {
t.Errorf("a missing path must not be handed to restic, got %v", extra)
}
// Create it: no gap, no warning, and the path IS captured.
nsRoot := appbackup.NamespaceRootFor(drive, m.systemDataPath)
if err := os.MkdirAll(filepath.Join(appbackup.UserdataDir(nsRoot), "media", "books"), 0o755); err != nil {
t.Fatal(err)
}
extra2, warns2, gaps2 := m.offboxCaptureSet("calibre-web")
if len(gaps2) != 0 || len(warns2) != 0 {
t.Fatalf("a PRESENT mandatory path must be silent, got gaps %v warns %v", gaps2, warns2)
}
if len(extra2) != 1 {
t.Fatalf("a present mandatory path must be handed to restic, got %v", extra2)
}
}
// Scenario D — an OPTIONAL declared path absent on disk changes nothing.
//
// RED-PROOF: remove the `p.Class == ClassMandatory` check in the stat-filter → an optional gap starts
// being reported, and together with the verdict would flip every app with an unused optional folder
// to not-ok, which is how a status stops being read.
//
// STATED BECAUSE IT CHANGES WHAT THIS PROVES: TierOffsite's tierKeeps() already admits ClassMandatory
// only, so an optional path cannot reach the stat-filter today. The class check is therefore a NO-OP
// and NO customer-visible warning disappears with it. It is written for parity with Tier 2 and so the
// verdict can never be flipped by an optional folder if that tier filter ever widens.
func TestOffboxCaptureSet_OptionalGapIsSilent(t *testing.T) {
drive := t.TempDir()
m, _, prov := classifiedOffboxManager(t, drive)
prov.hdd["komga"] = drive
prov.binds["komga"] = []ClassifiedBind{optionalUserdata("media/comics")}
prov.has["komga"] = true
extra, warns, gaps := m.offboxCaptureSet("komga")
if len(gaps) != 0 {
t.Fatalf("an absent OPTIONAL path must be silent, got gaps %v", gaps)
}
if len(warns) != 0 {
t.Fatalf("an absent OPTIONAL path must raise no customer warning, got %v", warns)
}
if len(extra) != 0 {
t.Fatalf("an absent path must not be captured, got %v", extra)
}
}
// A mandatory path that IS present alongside an absent optional one: still silent, still captured.
func TestOffboxCaptureSet_MixedClassesOnlyMandatoryCounts(t *testing.T) {
drive := t.TempDir()
m, _, prov := classifiedOffboxManager(t, drive)
prov.hdd["mixed"] = drive
prov.binds["mixed"] = []ClassifiedBind{mandatoryUserdata("docs"), optionalUserdata("cache")}
prov.has["mixed"] = true
nsRoot := appbackup.NamespaceRootFor(drive, m.systemDataPath)
if err := os.MkdirAll(filepath.Join(appbackup.UserdataDir(nsRoot), "docs"), 0o755); err != nil {
t.Fatal(err)
}
extra, warns, gaps := m.offboxCaptureSet("mixed")
if len(gaps) != 0 || len(warns) != 0 {
t.Fatalf("a present mandatory + absent optional must be silent, got gaps %v warns %v", gaps, warns)
}
if len(extra) != 1 {
t.Fatalf("the mandatory path must be captured, got %v", extra)
}
}
// The verdict rule itself, over its inputs. The surrounding run needs a live restic, so the decision
// is asserted where it is made rather than through a fake repository.
func TestMandatoryGapsDecideTheVerdict(t *testing.T) {
verdict := func(gaps map[string][]string) string {
if len(gaps) > 0 {
return "incomplete"
}
return "ok"
}
cases := []struct {
name string
gaps map[string][]string
want string
}{
{"no gaps", nil, "ok"},
{"empty map", map[string][]string{}, "ok"},
{"one app one folder", map[string][]string{"calibre-web": {"media/books"}}, "incomplete"},
{"two apps", map[string][]string{"a": {"x"}, "b": {"y"}}, "incomplete"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := verdict(tc.gaps)
if got != tc.want {
t.Fatalf("gaps %v → %q, want %q", tc.gaps, got, tc.want)
}
// "incomplete" must be distinct from every value that already existed, so a checker or a
// template matching on those cannot silently treat a coverage gap as one of them.
if got == "ok" && tc.want == "incomplete" {
t.Fatal("a coverage gap must never read as ok")
}
})
}
}
// Scenario C, THROUGH THE RUN — the verdict itself, not just the capture set.
//
// The first version of this file tested offboxCaptureSet alone, and its "red-proof" PASSED: the
// mutation (dropping the gap recording) lives in runOffboxInternal, which that test never reaches.
// A mutation that the test cannot observe is not a red-proof, and the fix is the test, not the code.
//
// RED-PROOF (now real): make the gap recording unreachable (`if false && len(capGaps) > 0`) or
// restore `o.LastStatus = "ok"` unconditionally → this FAILS with the run reporting ok over a
// mandatory gap, which is production behaviour up to v0.196.0.
func TestOffboxRun_MandatoryGapMakesTheRunIncomplete(t *testing.T) {
drive := t.TempDir()
m, sett, prov := classifiedOffboxManager(t, drive)
mkUnit(t, drive, "calibre-web")
prov.hdd["calibre-web"] = drive
prov.has["calibre-web"] = true
// Declared MANDATORY and deliberately ABSENT on disk — the drill's shape.
prov.binds["calibre-web"] = []ClassifiedBind{mandatoryUserdata("media/books")}
_ = sett.SetAppOffbox("calibre-web", true)
var gapNotified map[string][]string
m.SetOffboxGapNotify(func(g map[string][]string) { gapNotified = g })
cap := &backupCapture{}
m.SetOffboxRunner(cap.runner())
if err := m.RunOffboxBackup(context.Background()); err != nil {
t.Fatalf("the run itself must SUCCEED — a coverage gap is not a failed run: %v", err)
}
got := sett.GetOffboxTarget()
if got.LastStatus != "incomplete" {
t.Fatalf("LastStatus = %q, want \"incomplete\" — a run that dropped a MANDATORY directory is "+
"not a successful run, and reporting ok beside a warning is how this defect hid", got.LastStatus)
}
// What WAS captured is still recorded — half a backup is not no backup.
if got.LastSuccess == "" {
t.Error("LastSuccess must still record what was captured (§8.5) — suppressing it would be its own lie")
}
if cap.backups != 1 {
t.Errorf("the unit must still be pushed, got %d backup calls", cap.backups)
}
// And the OPERATOR is told, not only the log.
if len(gapNotified) != 1 || len(gapNotified["calibre-web"]) != 1 || gapNotified["calibre-web"][0] != "media/books" {
t.Fatalf("the operator gap signal did not fire with the app and folder, got %v", gapNotified)
}
}
// The companion: no gap → ok, and no operator signal. Without this, "incomplete" everywhere would
// also pass the test above.
func TestOffboxRun_NoGapStaysOk(t *testing.T) {
drive := t.TempDir()
m, sett, prov := classifiedOffboxManager(t, drive)
mkUnit(t, drive, "calibre-web")
nsRoot := appbackup.NamespaceRootFor(drive, m.systemDataPath)
if err := os.MkdirAll(filepath.Join(appbackup.UserdataDir(nsRoot), "media", "books"), 0o755); err != nil {
t.Fatal(err)
}
prov.hdd["calibre-web"] = drive
prov.has["calibre-web"] = true
prov.binds["calibre-web"] = []ClassifiedBind{mandatoryUserdata("media/books")}
_ = sett.SetAppOffbox("calibre-web", true)
fired := false
m.SetOffboxGapNotify(func(map[string][]string) { fired = true })
cap := &backupCapture{}
m.SetOffboxRunner(cap.runner())
if err := m.RunOffboxBackup(context.Background()); err != nil {
t.Fatalf("run: %v", err)
}
if got := sett.GetOffboxTarget(); got.LastStatus != "ok" {
t.Fatalf("LastStatus = %q, want ok — a complete run must not be downgraded", got.LastStatus)
}
if fired {
t.Error("the operator gap signal must NOT fire when nothing was missed")
}
}
@@ -0,0 +1,165 @@
package backup
import (
"context"
"strings"
"testing"
)
// R-234 — a run that SKIPPED an app the customer selected is not a successful run.
//
// The same paragraph the R-203 verdict block already carries — "a warning beside a success is read
// as a success" — was applied to one of the two shapes it describes. An app missing a declared
// mandatory FOLDER made the run `incomplete`; an app skipped ENTIRELY, with nothing of it in the
// snapshot at all, still reported `ok`. The smaller gap moved the verdict and the bigger one did not.
//
// Run-level on purpose: the classification and the verdict are both inside the run, and the sibling
// test file records what happened when its first version asserted the capture helper alone — its
// red-proof passed while the defect was untouched.
// Scenario A — a selected, DEPLOYED app with no recovery unit makes the run incomplete, names itself,
// and does not suppress what was captured.
//
// RED-PROOF: drop `unprotected` from the verdict condition (leave only mandatoryGaps) → this FAILS
// with the run reporting ok over a skipped app, which is production behaviour up to v0.204.0.
func TestOffboxRun_SkippedSelectedAppIsIncomplete(t *testing.T) {
drive := t.TempDir()
m, sett, prov := classifiedOffboxManager(t, drive)
// `kept` has a unit and is pushed; `dropped` is selected and deployed but has NO unit, so the
// per-app loop skips it. backedUp>0 is what made the existing no-silent-success guard stay quiet.
mkUnit(t, drive, "kept")
prov.hdd["kept"] = drive
prov.has["kept"] = true
prov.hdd["dropped"] = drive
prov.has["dropped"] = true
prov.deployed = map[string]bool{"kept": true, "dropped": true}
_ = sett.SetAppOffbox("kept", true)
_ = sett.SetAppOffbox("dropped", true)
var gapNotified map[string][]string
m.SetOffboxGapNotify(func(g map[string][]string) { gapNotified = g })
cap := &backupCapture{}
m.SetOffboxRunner(cap.runner())
if err := m.RunOffboxBackup(context.Background()); err != nil {
t.Fatalf("the run itself must SUCCEED — a skipped app is a coverage gap, not a failed run: %v", err)
}
got := sett.GetOffboxTarget()
if got.LastStatus != "incomplete" {
t.Fatalf("LastStatus = %q, want \"incomplete\" — the customer selected an app and the run did not "+
"carry it; on 2026-08-06 this reported „✓ Rendben” and the restore refused minutes later", got.LastStatus)
}
// Scenario A: the counters and the anchor still record what WAS captured.
if got.LastSuccess == "" {
t.Error("LastSuccess must still record what was captured — half a backup is not no backup")
}
if cap.backups != 1 {
t.Errorf("the app that HAD a unit must still be pushed, got %d backup calls", cap.backups)
}
// Scenario E: which app, and why.
if !strings.Contains(got.LastWarning, "dropped") {
t.Errorf("the warning must NAME the skipped app, got %q", got.LastWarning)
}
if !strings.Contains(got.LastWarning, "nincs helyi ment") {
t.Errorf("the warning must say WHY it was skipped, got %q", got.LastWarning)
}
if !strings.Contains(got.LastWarning, "következő ment") {
t.Errorf("the warning must say WHEN it will be protected, got %q", got.LastWarning)
}
// Scenario B: the operator hears about it, in the same vocabulary as a folder gap.
if len(gapNotified["dropped"]) == 0 {
t.Fatalf("the operator signal must carry the skipped app, got %v", gapNotified)
}
}
// Scenario C — a healthy run is untouched. Without this, "always incomplete" would also pass above,
// and a status that is never green is a status that stops being read.
//
// RED-PROOF: count EVERY skip (drop the classification switch and use len(res.missing)) → a healthy
// run goes amber and this FAILS.
func TestOffboxRun_HealthyRunStaysOk(t *testing.T) {
drive := t.TempDir()
m, sett, prov := classifiedOffboxManager(t, drive)
mkUnit(t, drive, "kept")
prov.hdd["kept"] = drive
prov.has["kept"] = true
_ = sett.SetAppOffbox("kept", true)
fired := false
m.SetOffboxGapNotify(func(map[string][]string) { fired = true })
cap := &backupCapture{}
m.SetOffboxRunner(cap.runner())
if err := m.RunOffboxBackup(context.Background()); err != nil {
t.Fatalf("run: %v", err)
}
got := sett.GetOffboxTarget()
if got.LastStatus != "ok" {
t.Fatalf("LastStatus = %q, want ok — every selected app was carried", got.LastStatus)
}
if fired {
t.Error("the operator signal must NOT fire when nothing was missed")
}
if strings.Contains(got.LastWarning, "NEM kerültek be") {
t.Errorf("a healthy run must carry no skip warning, got %q", got.LastWarning)
}
}
// Scenario D — a box with NOTHING selected keeps today's behaviour: ok, with the existing
// zero-selection notice. An unconfigured box reporting incomplete forever is its own defect.
//
// RED-PROOF: count the empty selection as a gap → this box goes permanently amber and this FAILS.
func TestOffboxRun_NothingSelectedIsNotAGap(t *testing.T) {
drive := t.TempDir()
m, sett, _ := classifiedOffboxManager(t, drive)
cap := &backupCapture{}
m.SetOffboxRunner(cap.runner())
if err := m.RunOffboxBackup(context.Background()); err != nil {
t.Fatalf("run: %v", err)
}
got := sett.GetOffboxTarget()
if got.LastStatus != "ok" {
t.Fatalf("LastStatus = %q, want ok — nothing was selected, so nothing was skipped", got.LastStatus)
}
if !strings.Contains(got.LastWarning, "nincs mentésre jelölt alkalmazás") {
t.Errorf("the existing zero-selection notice must survive, got %q", got.LastWarning)
}
}
// Scenario F — a selected app that is NOT deployed. Decided deliberately: it is NAMED with what to do
// about it, and it does NOT move the verdict, because a box left amber forever by an app somebody
// removed is a status nobody reads.
func TestOffboxRun_SelectedButUndeployedIsNamedNotCounted(t *testing.T) {
drive := t.TempDir()
m, sett, prov := classifiedOffboxManager(t, drive)
mkUnit(t, drive, "kept")
prov.hdd["kept"] = drive
prov.has["kept"] = true
prov.deployed = map[string]bool{"kept": true} // "removed-app" deliberately absent
_ = sett.SetAppOffbox("kept", true)
// selected, no unit, and NOT in the deployed set
_ = sett.SetAppOffbox("removed-app", true)
fired := false
m.SetOffboxGapNotify(func(map[string][]string) { fired = true })
cap := &backupCapture{}
m.SetOffboxRunner(cap.runner())
if err := m.RunOffboxBackup(context.Background()); err != nil {
t.Fatalf("run: %v", err)
}
got := sett.GetOffboxTarget()
if got.LastStatus != "ok" {
t.Fatalf("LastStatus = %q, want ok — an app that is not installed cannot be protected, and must "+
"not hold the box amber forever", got.LastStatus)
}
if !strings.Contains(got.LastWarning, "removed-app") {
t.Errorf("the undeployed selection must still be NAMED, got %q", got.LastWarning)
}
if !strings.Contains(got.LastWarning, "vedd ki a kijelöl") {
t.Errorf("it must say what to do about it, got %q", got.LastWarning)
}
if fired {
t.Error("an undeployed app must not raise the operator gap signal")
}
}
@@ -0,0 +1,121 @@
package backup
import (
"fmt"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// the real demo-hp target shape — the values the sanitiser must remove literally
func diagTarget() *settings.OffboxTarget {
return &settings.OffboxTarget{
Host: "u629488-sub3.your-storagebox.de", User: "u629488-sub3",
RepoPath: "/home/felhom-repo", Port: 23,
}
}
// F-DIAG — four causes collapsed into one string, and that string was a RAW error passthrough.
//
// Two separate defects in one line of code:
// - an operator could not tell a full quota from a dead network without reading logs;
// - `err.Error()` from restic/ssh carries the repo reference `sftp:<user>@<host>:<path>`, so the
// notification carried a customer-identifying location (and potentially a credential) off the box,
// breaking the keys-not-values rule at the one place the text leaves the machine.
func TestClassifyOffsiteFailure_EachCauseIsDistinct(t *testing.T) {
cases := []struct {
name string
err error
want OffsiteFailureClass
}{
{"quota gate", fmt.Errorf("A távoli mentés túllépte a tárhelykeretet (51/50 GB) — törölj régi mentéseket vagy kérj nagyobb keretet."), OffsiteFailQuota},
{"orphaned repo", fmt.Errorf("probe: %w", ErrOffboxOrphaned), OffsiteFailOrphaned},
{"no repo", fmt.Errorf("restic: unable to open config file: Stat: file does not exist\nIs there a repository at the following location?"), OffsiteFailNoRepo},
{"no units", fmt.Errorf("off-box backup produced no snapshots: 3 app(s) toggled but no recovery unit was found on any connected drive (missing: a, b, c)"), OffsiteFailNoUnits},
{"transport refused", fmt.Errorf("dial tcp 1.2.3.4:23: connect: connection refused"), OffsiteFailTransport},
{"transport timeout", fmt.Errorf("ssh: handshake failed: i/o timeout"), OffsiteFailTransport},
{"transport auth", fmt.Errorf("ssh: permission denied (publickey)"), OffsiteFailTransport},
{"unclassified", fmt.Errorf("restic: some future error nobody has seen"), OffsiteFailUnknown},
}
seen := map[OffsiteFailureClass]bool{}
for _, c := range cases {
got := ClassifyOffsiteFailure(c.err)
if got != c.want {
t.Errorf("%s: class = %q, want %q", c.name, got, c.want)
}
seen[got] = true
}
// The whole point of F-DIAG: the causes must not collapse.
if len(seen) < 5 {
t.Errorf("only %d distinct classes across %d causes — the causes are still collapsing", len(seen), len(cases))
}
}
// An unclassifiable error must say so rather than being folded into a neighbour. Inventing a precision
// the code does not have is how a confident-but-wrong diagnosis ships.
func TestClassifyOffsiteFailure_UnknownIsHonest(t *testing.T) {
if got := ClassifyOffsiteFailure(fmt.Errorf("something entirely new")); got != OffsiteFailUnknown {
t.Errorf("an unclassifiable error was folded into %q instead of being reported as unknown", got)
}
msg := offsiteFailureMessage(diagTarget(), fmt.Errorf("something entirely new"), time.Minute)
if !strings.Contains(msg, "ismeretlen okból") {
t.Errorf("the unknown case does not admit it is unknown: %q", msg)
}
}
// THE SECRETS TEST. The repo reference must never survive into a message.
//
// RED-PROOF: make sanitiseOffsiteError return err.Error() unchanged → this fails with
// "the repo reference reached the message".
func TestOffsiteFailureMessage_NeverCarriesTheRepoReference(t *testing.T) {
leaky := []error{
fmt.Errorf(`Fatal: unable to open repository at sftp:u629488-sub3@u629488-sub3.your-storagebox.de:/home/felhom-repo: connection refused`),
fmt.Errorf(`ssh: connect to host u629488-sub3.your-storagebox.de port 23: Connection refused`),
fmt.Errorf(`restic: repo "sftp:u629488-sub3@u629488-sub3.your-storagebox.de:/home/felhom-repo" locked`),
}
for _, e := range leaky {
msg := offsiteFailureMessage(diagTarget(), e, 42*time.Second)
for _, forbidden := range []string{
"sftp:",
"your-storagebox.de",
"u629488-sub3",
"/home/felhom-repo",
} {
if strings.Contains(msg, forbidden) {
t.Errorf("the repo reference reached the message (%q leaked):\n %s", forbidden, msg)
}
}
if !strings.Contains(msg, "<repo>") {
t.Errorf("the redaction placeholder is absent — the detail may have been dropped silently instead of sanitised:\n %s", msg)
}
}
}
// The message must still be ACTIONABLE. Sanitising must not reduce it to a shrug — an operator needs
// the cause line plus enough residual detail to act.
func TestOffsiteFailureMessage_StaysActionable(t *testing.T) {
msg := offsiteFailureMessage(diagTarget(), fmt.Errorf("dial tcp: connect: connection refused"), 90*time.Second)
if !strings.Contains(msg, "nem érhető el") {
t.Errorf("the transport cause is not named: %q", msg)
}
if !strings.Contains(msg, "connection refused") {
t.Errorf("all actionable detail was stripped along with the secret: %q", msg)
}
if !strings.Contains(msg, "1m30s") {
t.Errorf("the duration was lost: %q", msg)
}
}
// A very long error must be bounded — an unbounded restic dump in an email is its own problem.
func TestSanitiseOffsiteError_IsBounded(t *testing.T) {
long := fmt.Errorf("%s", strings.Repeat("x", 5000))
if got := sanitiseOffsiteErrorFor(diagTarget(), long); len(got) > 320 {
t.Errorf("sanitised error is %d chars — unbounded", len(got))
}
if sanitiseOffsiteErrorFor(diagTarget(), nil) != "" {
t.Error("a nil error produced text")
}
}
+237 -28
View File
@@ -4,6 +4,7 @@ import (
"crypto/sha256" "crypto/sha256"
"encoding/hex" "encoding/hex"
"encoding/json" "encoding/json"
"errors"
"fmt" "fmt"
"io" "io"
"os" "os"
@@ -12,22 +13,32 @@ import (
"strings" "strings"
"time" "time"
"gitea.dooplex.hu/admin/felhom-controller/internal/system"
"gopkg.in/yaml.v3" "gopkg.in/yaml.v3"
) )
// RecoveryManifest describes an app's self-contained, SECRET-FREE recovery unit (Phase 2). // RecoveryManifest describes an app's self-contained recovery unit.
// //
// The unit on a drive is `<nsRoot>/backups/primary/<app>/` and contains: // The unit on a drive is `<nsRoot>/backups/primary/<app>/` and contains:
// compose/ docker-compose.yml + .felhom.yml + a SECRET-STRIPPED app.yaml //
// compose/ docker-compose.yml + .felhom.yml + app.yaml (0600; carries the PORTABLE secrets)
// db-dumps/ app-consistent DB dump(s) (written by the dump flow) // db-dumps/ app-consistent DB dump(s) (written by the dump flow)
// volume-dumps/ named-volume tars (written by the dump flow) // volume-dumps/ named-volume tars (written by the dump flow)
// manifest.json this file // manifest.json this file
// //
// The unit holds NO secret values, NO data-encrypting keys, and NOT the Docker image — only the // D5 (schema 2) changed what the unit holds. Before it held NO secret at all, which made
// pinned image tag(s) (re-pulled on restore) and the NAMES of the secret/data-key env vars. The // "restore from the drive alone" false: the fast, local, customer-doable Tier-1/2 restore secretly
// secret values are recovered at restore time from the guest's own app.yaml (live on the rootfs, // depended on the slow, operator-driven whole-guest restore, because a data-encrypting key or a DB
// or via the PBS whole-guest snapshot) — see Restore. "Restore from the unit alone" is therefore // password absent from the guest cannot be regenerated without rendering the restored data
// honestly "unit + the guest's app.yaml"; SecretSource records that dependency explicitly. // unreachable. The unit now carries the PORTABLE secret class (stacks.PortableSecretEnvVars) in its
// 0600 app.yaml, and Tier-1/2 needs the DRIVE AND NOTHING ELSE.
//
// It still holds NO `type: password` admin login (those are internet-reachable, so their blast radius
// is not bounded by the drive — they stay in the guest and are regenerated on restore) and NOT the
// Docker image, only the pinned tag(s), re-pulled on restore. SecretSource records the split.
//
// A schema-1 unit carries no secrets: the restore degrades to the pre-D5 guest-only behaviour rather
// than failing, and the next capture rewrites it (the app.yaml checksum changes).
type RecoveryManifest struct { type RecoveryManifest struct {
SchemaVersion int `json:"schema_version"` SchemaVersion int `json:"schema_version"`
AppName string `json:"app_name"` AppName string `json:"app_name"`
@@ -37,13 +48,17 @@ type RecoveryManifest struct {
Drive string `json:"drive"` // HDD_PATH (in-guest mount) Drive string `json:"drive"` // HDD_PATH (in-guest mount)
NamespaceRoot string `json:"namespace_root"` // resolved felhom-data namespace root NamespaceRoot string `json:"namespace_root"` // resolved felhom-data namespace root
ImagePins []string `json:"image_pins"` // image NOT stored — re-pulled on restore ImagePins []string `json:"image_pins"` // image NOT stored — re-pulled on restore
SecretEnvVars []string `json:"secret_env_vars"` // NAMES only — recovered from guest/PBS SecretEnvVars []string `json:"secret_env_vars"` // NAMES of every secret/password field
DataKeyEnvVars []string `json:"data_key_env_vars"` // fail-closed gate on restore DataKeyEnvVars []string `json:"data_key_env_vars"` // fail-closed gate on restore
SecretSource string `json:"secret_source"` // human note: where secrets come from SecretSource string `json:"secret_source"` // human note: where secrets come from
ConfigFiles []string `json:"config_files"` // captured into compose/ ConfigFiles []string `json:"config_files"` // captured into compose/
DBDumps []string `json:"db_dumps"` DBDumps []string `json:"db_dumps"`
VolumeDumps []string `json:"volume_dumps"` VolumeDumps []string `json:"volume_dumps"`
Checksums map[string]string `json:"checksums"` // sha256 of captured compose/ files Checksums map[string]string `json:"checksums"` // sha256 of captured compose/ files
// PortableSecretEnvVars (D5) are the NAMES of the secrets this unit's app.yaml CARRIES. Names only
// — the manifest is 0644 and never holds a value. The restore reads it to know which app.yaml env
// entries are secrets rather than plain config; absent (schema 1) ⇒ the unit carries none.
PortableSecretEnvVars []string `json:"portable_secret_env_vars,omitempty"`
// R-43/R-44 (v0.148.0): the coherence stamp. An offsite run refreshes the dumps FIRST and then // R-43/R-44 (v0.148.0): the coherence stamp. An offsite run refreshes the dumps FIRST and then
// captures the unit, so a manifest carrying an OffsiteRunID asserts "the db-dumps/ in this unit // captures the unit, so a manifest carrying an OffsiteRunID asserts "the db-dumps/ in this unit
// were taken by that run" — i.e. the snapshot is an internally coherent {DB@T, files@T} pair. // were taken by that run" — i.e. the snapshot is an internally coherent {DB@T, files@T} pair.
@@ -67,9 +82,11 @@ func (m *Manager) SetTier2Notifier(fn func(stackName, destLabel string, dur time
m.tier2Notify = fn m.tier2Notify = fn
} }
// CaptureRecoveryUnit writes/refreshes an app's secret-free recovery unit: it captures the // CaptureRecoveryUnit writes/refreshes an app's recovery unit: it captures the compose + metadata +
// compose + metadata + a secret-stripped app.yaml into compose/, enumerates the DB/volume dumps // an app.yaml carrying the PORTABLE secret class (D5) into compose/, enumerates the DB/volume dumps
// already present, and writes manifest.json. It NEVER writes a secret value or the Docker image. // already present, and writes manifest.json. It never writes the Docker image (only the pinned tag),
// and never writes a WITHHELD secret — the split is decided in buildUnitAppYaml, pinned by
// TestCaptureRecoveryUnitCarriesPortableSecretsOnly.
// //
// Idempotent: it builds the captured content in memory first and SKIPS all writes when the unit is // Idempotent: it builds the captured content in memory first and SKIPS all writes when the unit is
// already current (same config checksums, same dump set, same controller version) — so it can run on // already current (same config checksums, same dump set, same controller version) — so it can run on
@@ -106,7 +123,7 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error {
checksums[fname] = sha256Hex(data) checksums[fname] = sha256Hex(data)
configFiles = append(configFiles, fname) configFiles = append(configFiles, fname)
} }
appYaml := buildStrippedAppYaml(info) appYaml := buildUnitAppYaml(info)
files = append(files, capFile{"app.yaml", appYaml, 0600}) files = append(files, capFile{"app.yaml", appYaml, 0600})
checksums["app.yaml"] = sha256Hex(appYaml) checksums["app.yaml"] = sha256Hex(appYaml)
configFiles = append(configFiles, "app.yaml") configFiles = append(configFiles, "app.yaml")
@@ -150,7 +167,7 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error {
} }
manifest := &RecoveryManifest{ manifest := &RecoveryManifest{
SchemaVersion: 1, SchemaVersion: 2, // D5: compose/app.yaml carries the portable secret class
AppName: stackName, AppName: stackName,
DisplayName: info.DisplayName, DisplayName: info.DisplayName,
ControllerVer: version, ControllerVer: version,
@@ -160,7 +177,8 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error {
ImagePins: info.ImagePins, ImagePins: info.ImagePins,
SecretEnvVars: info.SecretEnvVars, SecretEnvVars: info.SecretEnvVars,
DataKeyEnvVars: info.DataKeyEnvVars, DataKeyEnvVars: info.DataKeyEnvVars,
SecretSource: "guest app.yaml (live rootfs) or PBS whole-guest snapshot — never stored in this unit", PortableSecretEnvVars: info.PortableSecretEnvVars,
SecretSource: "portable secrets (data keys, DB passwords, internal signing secrets) are IN this unit's compose/app.yaml (0600); internet-reachable admin logins are NOT, and come from the guest's app.yaml or are regenerated on restore",
ConfigFiles: configFiles, ConfigFiles: configFiles,
DBDumps: dbDumps, DBDumps: dbDumps,
VolumeDumps: volDumps, VolumeDumps: volDumps,
@@ -172,13 +190,158 @@ func (m *Manager) CaptureRecoveryUnit(stackName string) error {
return fmt.Errorf("writing manifest: %w", err) return fmt.Errorf("writing manifest: %w", err)
} }
m.logger.Printf("[INFO] [backup] Recovery unit captured for %s → %s (images=%d, secrets-referenced=%d, data_keys=%d)", // Counts and NAMES only — never a value (D5 puts more secrets through this path than before).
stackName, RecoveryUnitPath(nsRoot, stackName), len(info.ImagePins), len(info.SecretEnvVars), len(info.DataKeyEnvVars)) m.logger.Printf("[INFO] [backup] Recovery unit captured for %s → %s (images=%d, secrets-referenced=%d, data_keys=%d, portable-carried=%d/%d, withheld=%d)",
stackName, RecoveryUnitPath(nsRoot, stackName), len(info.ImagePins), len(info.SecretEnvVars),
len(info.DataKeyEnvVars), len(info.PortableSecrets), len(info.PortableSecretEnvVars),
len(withheldSecretNames(info)))
return nil return nil
} }
// UnitSpace is the target filesystem's occupancy at the moment a capture failed — the numbers that
// answer "why" without an operator logging in. Nil when the filesystem could not be read at all
// (system.GetDiskUsage returns nil on error), which is reported as unknown rather than as full.
type UnitSpace struct {
Path string
UsedGB float64
AvailGB float64
TotalGB float64
UsedPercent float64
}
// String renders the space figures for an operator, or says plainly that they are unknown. An absent
// reading must never render as zeros — "0 GB free" and "we could not look" are opposite diagnoses.
func (u *UnitSpace) String() string {
if u == nil {
return "target filesystem usage unavailable"
}
return fmt.Sprintf("%s: %.1f/%.1f GB used (%.0f%%), %.1f GB free",
u.Path, u.UsedGB, u.TotalGB, u.UsedPercent, u.AvailGB)
}
// SetUnitNotify wires the per-app recovery-unit capture failure alert (R-158 / R-167). INIT-ONLY —
// call once at startup, in main.go, alongside SetOffboxNotify. Nil-safe: an unwired seam is silently
// the pre-v0.191.0 behaviour, which is a `[WARN]` line and nothing else.
func (m *Manager) SetUnitNotify(fn func(stackName string, err error, usage *UnitSpace)) {
m.unitNotify = fn
}
// unitTargetSpace reads the occupancy of the filesystem a unit for `stackName` would be written to.
// Nil on an unreadable path — never a fabricated zero (§8.4: an unreadable filesystem is not a full
// one, and the drive gate already owns the absent-drive case).
func (m *Manager) unitTargetSpace(stackName string) *UnitSpace {
path := m.GetAppDrivePath(stackName)
if path == "" {
return nil
}
di := system.GetDiskUsage(path)
if di == nil {
return nil
}
return &UnitSpace{
Path: path, UsedGB: di.UsedGB, AvailGB: di.AvailGB,
TotalGB: di.TotalGB, UsedPercent: di.UsedPercent,
}
}
// ── The capture floor (R-165 / decision B2) ──────────────────────────────────────────────────────
//
// WHAT IT REPLACES. Until the `mp1`→`mp0` merge, the 20 G backup partition was a BULKHEAD as well as
// a ceiling: an app whose unit outgrew it was refused per app, its last good unit preserved
// byte-identical, and the overflow **could not reach `/var/lib/docker`** because that was a different
// filesystem. After the merge it can, and a full Docker data-root is a stopped box, not a slow one.
// This floor is that bulkhead, done deliberately instead of by accident.
//
// IT IS ABOUT THE FILESYSTEM'S HEADROOM, NEVER THE UNIT'S SIZE. A per-unit size cap would be R-163
// rebuilt inside one volume — the wall moved rather than removed — so a large unit on a filesystem
// with ample room is captured, whatever its size.
//
// IT REFUSES; IT NEVER DELETES. Nothing on this filesystem is generational: a unit is ONE fixed path
// per app (`backups/primary/<app>`) refreshed in place, and a DB dump is `<stack>-<dbtype>.sql`, also
// fixed. So "prune the oldest" could only mean deleting a DIFFERENT app's only local recovery unit to
// make room for this one, and that is not a trade this system makes. `pruneStalePrimaryDirs` is NOT a
// retention policy — it removes ORPHANED directories left when an app moves drives, and has no notion
// of age — so it must never be repurposed here.
const (
// FloorUsedPercent / FloorFreeGiB — the reserve. Two terms, whichever binds first, the same shape
// as `internal/fillwatch` (proven live on 2026-08-02: the critical alert fired on the free-byte
// term at 91% used, where a percent-only rule stayed silent).
//
// THEY SIT DELIBERATELY BEYOND fillwatch's CRITICAL BAND (95% / 2 GiB), so the customer is ALWAYS
// warned before a refusal can happen. A floor that fires before its own warning is a silent
// failure wearing a threshold; `TestFloorSitsBelowTheCriticalWarningBand` pins the ordering.
//
// 1 GiB is the reserve, not a working budget: §7.5 measures a DB-backed app's unit at up to ~2× its
// data, so no fixed number can guarantee a capture fits. What this guarantees is different and is
// the bulkhead's actual job — that a capture cannot consume the last of the space the container
// runtime needs to keep running.
FloorUsedPercent = 97.0
FloorFreeGiB = 1.0
)
// ErrCaptureFloor marks an app's backup refused for headroom. It is a REFUSAL, not a failure of the
// backup machinery — the distinction matters to a reader of the alert, which is why the message names
// the reserve rather than reporting an I/O error.
//
// R-181 widened what it covers: it now refuses the app's DB dump, volume dump and capture together
// (see admission.go), not the capture alone. The sentinel keeps its name because callers match it and
// "capture" still reads correctly for "capturing this app's backup"; the MESSAGE is what changed, and
// the message is what an operator sees.
var ErrCaptureFloor = errors.New("refused: backing up this app would leave the filesystem below the reserve")
// floorVerdict is the PURE predicate: given a reading and this app's estimated write, does the floor
// refuse, and on which term? Separated so the thresholds are unit-testable without a filesystem, a
// stack provider or a clock.
//
// TWO QUESTIONS, NOT ONE (R-181). "Is the filesystem already below the reserve?" is the headroom term
// and was all B2 asked. "Would THIS app's write take it below?" is the size term, and its absence is
// how an app was admitted at 96% used and then allowed to write 2 GB. Both terms are evaluated
// against BOTH thresholds — a large write can cross the percentage bound on a small volume and the
// free-byte bound on a large one, which is the same reason the reserve has two terms at all.
//
// §8.4 — A NIL READING NEITHER REFUSES NOR WARNS. An unreadable filesystem is the drive gate's
// business and has its own alert; refusing on it would block every backup on a box whose drive merely
// blipped, and warning on it would be a false alarm with a misleading cause.
//
// estGiB == 0 (no previous dump to estimate from) degrades to the headroom term alone, deliberately:
// refusing an app that has never been backed up would make the FIRST backup the one that can never
// happen (Scenario E).
func (m *Manager) floorVerdict(u *UnitSpace, estGiB float64) (*UnitSpace, floorReason) {
if u == nil {
return nil, floorAdmit
}
if u.UsedPercent >= FloorUsedPercent || u.AvailGB < FloorFreeGiB {
return u, floorHeadroom
}
if estGiB > 0 {
availAfter := u.AvailGB - estGiB
usedAfter := u.UsedPercent
if u.TotalGB > 0 {
usedAfter = (u.UsedGB + estGiB) / u.TotalGB * 100
}
if availAfter < FloorFreeGiB || usedAfter >= FloorUsedPercent {
return u, floorSize
}
}
return u, floorAdmit
}
// readUnitSpace goes through the seam when one is injected, so a test can state the filesystem's
// occupancy as an input instead of manufacturing it on a real disk. Nil seam → the real statfs.
func (m *Manager) readUnitSpace(stackName string) *UnitSpace {
if m.unitSpaceFn != nil {
return m.unitSpaceFn(stackName)
}
return m.unitTargetSpace(stackName)
}
// captureAllRecoveryUnits refreshes the recovery unit for every deployed stack. Best-effort: // captureAllRecoveryUnits refreshes the recovery unit for every deployed stack. Best-effort:
// a per-app failure is logged and does not abort the others. // a per-app failure is logged, NOTIFIED (R-158), and does not abort the others.
//
// R-181: the reserve is consulted through `admitApp`, which is the SAME verdict the DB-dump and
// volume-dump legs of this run already consulted for this app. When a run is in flight the answer
// here is a memo lookup — an app refused before its first write is refused here too, silently,
// because it was already alerted once. Outside a run (the periodic status refresh) it decides fresh.
func (m *Manager) captureAllRecoveryUnits() { func (m *Manager) captureAllRecoveryUnits() {
if m.stackProvider == nil { if m.stackProvider == nil {
return return
@@ -188,8 +351,20 @@ func (m *Manager) captureAllRecoveryUnits() {
if m.settings != nil && (m.settings.IsDisconnected(drivePath) || m.settings.IsDecommissioned(drivePath)) { if m.settings != nil && (m.settings.IsDisconnected(drivePath) || m.settings.IsDecommissioned(drivePath)) {
continue // drive not writable — skip, the existing unit stays as-is continue // drive not writable — skip, the existing unit stays as-is
} }
m.noteAttempted(stack.Name)
// The reserve, checked BEFORE anything is written. Per app, and the loop continues.
if !m.admitApp(stack.Name) {
continue
}
if err := m.CaptureRecoveryUnit(stack.Name); err != nil { if err := m.CaptureRecoveryUnit(stack.Name); err != nil {
m.noteFailure(stack.Name, "recovery-unit capture", err.Error())
m.logger.Printf("[WARN] [backup] Recovery unit capture failed for %s: %v", stack.Name, err) m.logger.Printf("[WARN] [backup] Recovery unit capture failed for %s: %v", stack.Name, err)
// R-158: per app, and the loop CONTINUES — one app's failure must not silence the
// others, and it must not abort their captures either. The space figures are read at
// the moment of failure, because the point is to answer "why" (usually: no room).
if m.unitNotify != nil {
m.unitNotify(stack.Name, err, m.unitTargetSpace(stack.Name))
}
} }
} }
} }
@@ -200,29 +375,63 @@ func (m *Manager) versionLocked() string {
return m.version return m.version
} }
// strippedAppYaml is the on-disk shape of the secret-free app.yaml captured into the unit. // strippedAppYaml is the on-disk shape of the app.yaml captured into the unit. The name is historical:
// since D5 the `env` map carries the PORTABLE secrets alongside the plain config (see buildUnitAppYaml).
type strippedAppYaml struct { type strippedAppYaml struct {
Deployed bool `yaml:"deployed"` Deployed bool `yaml:"deployed"`
Env map[string]string `yaml:"env"` Env map[string]string `yaml:"env"`
} }
// buildStrippedAppYaml renders a secret-free app.yaml (non-secret env only) as bytes. Deterministic: // buildUnitAppYaml renders the unit's app.yaml as bytes: the non-secret env PLUS the portable secret
// yaml.v3 sorts map keys and the secret-name list comes in stable metadata order, so identical input // values (D5). Deterministic: yaml.v3 sorts map keys and the name lists come in stable metadata order,
// yields identical bytes (needed for the checksum-skip guard). // so identical input yields identical bytes (needed for the checksum-skip guard).
func buildStrippedAppYaml(info RecoveryInfo) []byte { //
body, err := yaml.Marshal(strippedAppYaml{Deployed: true, Env: info.NonSecretEnv}) // This is the ONE place the capture side decides what does and does not reach the drive — there is no
// second path that writes a unit app.yaml. The caller writes the result 0600.
func buildUnitAppYaml(info RecoveryInfo) []byte {
env := make(map[string]string, len(info.NonSecretEnv)+len(info.PortableSecrets))
for k, v := range info.NonSecretEnv {
env[k] = v
}
// Portable secrets last: NonSecretEnv is disjoint from the secret set by construction
// (GetStackRecoveryInfo), so this cannot shadow a plain config value.
for k, v := range info.PortableSecrets {
env[k] = v
}
body, err := yaml.Marshal(strippedAppYaml{Deployed: true, Env: env})
if err != nil { if err != nil {
body = []byte("deployed: true\nenv: {}\n") body = []byte("deployed: true\nenv: {}\n")
} }
header := "# Captured by felhom-controller recovery unit — SECRET-FREE.\n" + header := "# Captured by felhom-controller recovery unit.\n" +
"# Secret/data-key values are intentionally omitted; recover them at restore from the\n" + "# This file CARRIES SECRETS (D5) so a Tier-1/2 restore needs the drive and nothing else:\n" +
"# guest's own app.yaml (live rootfs, or the PBS whole-guest snapshot). Stripped names:\n" "# data-encrypting keys, database passwords and internal signing secrets. Mode 0600.\n"
if len(info.SecretEnvVars) > 0 { if len(info.PortableSecretEnvVars) > 0 {
header += "# " + strings.Join(info.SecretEnvVars, ", ") + "\n" header += "# Carried: " + strings.Join(info.PortableSecretEnvVars, ", ") + "\n"
}
// The withheld class is named, not valued — an operator reading the unit must be able to see WHY a
// credential is missing rather than suspecting a capture bug.
if withheld := withheldSecretNames(info); len(withheld) > 0 {
header += "# WITHHELD (internet-reachable logins — stay in the guest, regenerated on restore): " +
strings.Join(withheld, ", ") + "\n"
} }
return []byte(header + string(body)) return []byte(header + string(body))
} }
// withheldSecretNames returns the secret names deliberately NOT carried by the unit, in stable order.
func withheldSecretNames(info RecoveryInfo) []string {
portable := make(map[string]bool, len(info.PortableSecretEnvVars))
for _, n := range info.PortableSecretEnvVars {
portable[n] = true
}
var out []string
for _, n := range info.SecretEnvVars {
if !portable[n] {
out = append(out, n)
}
}
return out
}
// writeManifest writes the manifest JSON atomically. // writeManifest writes the manifest JSON atomically.
func writeManifest(dst string, manifest *RecoveryManifest) error { func writeManifest(dst string, manifest *RecoveryManifest) error {
data, err := json.MarshalIndent(manifest, "", " ") data, err := json.MarshalIndent(manifest, "", " ")
@@ -0,0 +1,182 @@
package backup
import (
"io"
"log"
"path/filepath"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/appbackup"
)
// R-158 / R-167 (D-c, operator half) — a per-app Tier-1 recovery-unit capture failure must reach a
// hub channel.
//
// THE GAP THESE CLOSE. `captureAllRecoveryUnits` logged `[WARN] Recovery unit capture failed for %s`
// and stopped there. The manager carried three notify seams — tier2Notify, offboxNotify,
// offboxEnlargeBlockedNotify — and none for the unit capture, so the one page a person opens to ask
// whether ONE app is backed up (`/backups/apps`) was the one page that never said. Fifth instance in
// this project of a mechanism built and left disconnected.
// unitFailProvider lists a fixed set of stacks and refuses GetStackRecoveryInfo for the named ones,
// which is the earliest real failure inside CaptureRecoveryUnit ("stack %q not found").
type unitFailProvider struct {
stacks []string
fail map[string]bool
dir string
}
func (p *unitFailProvider) GetStackComposePath(string) (string, bool) { return "", false }
func (p *unitFailProvider) ListDeployedStacks() []StackSummary {
out := make([]StackSummary, 0, len(p.stacks))
for _, s := range p.stacks {
out = append(out, StackSummary{Name: s})
}
return out
}
func (p *unitFailProvider) GetStackHDDMounts(string) []string { return nil }
func (p *unitFailProvider) GetStackHDDPath(string) string { return "" }
func (p *unitFailProvider) GetImportRoot() string { return "" }
func (p *unitFailProvider) GetDockerVolumes(string) []string { return nil }
func (p *unitFailProvider) StopStack(string) error { return nil }
func (p *unitFailProvider) StartStack(string) error { return nil }
func (p *unitFailProvider) RefreshAndIsRunning(string) bool { return true }
func (p *unitFailProvider) GetStackRecoveryInfo(name string) (RecoveryInfo, bool) {
if p.fail[name] {
return RecoveryInfo{}, false
}
return RecoveryInfo{StackDir: filepath.Join(p.dir, "stacks", name)}, true
}
func (p *unitFailProvider) RecoverStackSecrets(string, []string) map[string]string { return nil }
func (p *unitFailProvider) RecreateStackDefinitionFromUnit(string, string, map[string]string) error {
return nil
}
func (p *unitFailProvider) StartStackServices(string, []string) error { return nil }
func (p *unitFailProvider) GetStackClassifiedBinds(string) ([]appbackup.ClassifiedBind, bool) {
return nil, false
}
var _ appbackup.StackDataProvider = (*unitFailProvider)(nil)
type unitEvent struct {
app string
err string
usage *UnitSpace
}
func newUnitNotifyManager(t *testing.T, stacks []string, fail map[string]bool) (*Manager, *[]unitEvent) {
t.Helper()
dir := t.TempDir()
m := &Manager{
logger: log.New(io.Discard, "", 0),
systemDataPath: dir,
stackProvider: &unitFailProvider{stacks: stacks, fail: fail, dir: dir},
}
var got []unitEvent
m.SetUnitNotify(func(name string, err error, usage *UnitSpace) {
got = append(got, unitEvent{app: name, err: err.Error(), usage: usage})
})
return m, &got
}
// --- Scenario C — a local unit capture failure reaches the operator ------------------------------
func TestCaptureAll_FailureNotifiesOnceWithTheSpaceFigures(t *testing.T) {
m, got := newUnitNotifyManager(t, []string{"immich"}, map[string]bool{"immich": true})
m.captureAllRecoveryUnits()
if len(*got) != 1 {
t.Fatalf("got %d unit-failure events, want exactly 1 — a per-app Tier-1 capture failure "+
"reached no hub channel, which is the R-158 gap un-fixed", len(*got))
}
e := (*got)[0]
if e.app != "immich" {
t.Fatalf("event names app %q, want immich — an operator cannot act on an unnamed app", e.app)
}
if e.err == "" {
t.Fatal("the event carries no error — the operator is told a capture failed but not why")
}
// The space figures are the point: the overwhelmingly likely cause is a full filesystem, and
// these answer "why" without an operator logging in.
if e.usage == nil {
t.Fatal("the event carries no space figures for a readable target filesystem — this is the " +
"pair of numbers that makes the alert actionable, and the same pair the customer fill " +
"warning reports (which is why the two ship together)")
}
if e.usage.Path == "" || e.usage.TotalGB <= 0 {
t.Fatalf("space figures are not populated: %+v", e.usage)
}
}
// --- Scenario D — one failing app does not silence the others ------------------------------------
func TestCaptureAll_OneFailureDoesNotAbortOrDuplicate(t *testing.T) {
m, got := newUnitNotifyManager(t,
[]string{"homebox", "immich", "nextcloud"},
map[string]bool{"immich": true})
m.captureAllRecoveryUnits()
if len(*got) != 1 {
t.Fatalf("got %d events, want exactly 1 — either the loop ABORTED on the middle app "+
"(and its siblings were never captured), or one failure produced several events: %+v",
len(*got), *got)
}
if (*got)[0].app != "immich" {
t.Fatalf("event names %q, want immich", (*got)[0].app)
}
// The siblings must have been ATTEMPTED after the failure — a positive observable, not the
// absence of an event. The provider records nothing, so assert via the failure set instead:
// flip the LAST app to failing and require both events.
m2, got2 := newUnitNotifyManager(t,
[]string{"homebox", "immich", "nextcloud"},
map[string]bool{"immich": true, "nextcloud": true})
m2.captureAllRecoveryUnits()
if len(*got2) != 2 {
t.Fatalf("got %d events, want 2 — the app AFTER the first failure was never reached, so the "+
"loop is aborting rather than continuing: %+v", len(*got2), *got2)
}
if (*got2)[0].app != "immich" || (*got2)[1].app != "nextcloud" {
t.Fatalf("events %+v, want immich then nextcloud in loop order", *got2)
}
}
// A successful capture must be SILENT. An alert that fires on success is an alert an operator learns
// to ignore.
func TestCaptureAll_SuccessIsSilent(t *testing.T) {
m, got := newUnitNotifyManager(t, []string{"homebox"}, nil)
m.captureAllRecoveryUnits()
if len(*got) != 0 {
t.Fatalf("a successful capture fired %d event(s): %+v", len(*got), *got)
}
}
// The seam must be nil-safe: an unwired notify is the pre-v0.191.0 behaviour (a WARN line), never a
// panic that takes the whole nightly backup down with it.
func TestCaptureAll_UnwiredNotifyDoesNotPanic(t *testing.T) {
dir := t.TempDir()
m := &Manager{
logger: log.New(io.Discard, "", 0),
systemDataPath: dir,
stackProvider: &unitFailProvider{stacks: []string{"immich"}, fail: map[string]bool{"immich": true}, dir: dir},
}
m.captureAllRecoveryUnits() // no SetUnitNotify — must not panic
}
// §8.4 in the failure direction: an unreadable target filesystem is reported as UNKNOWN, never as
// zeros. "0 GB free" and "we could not look" are opposite diagnoses, and rendering the second as the
// first is the presence-is-not-success trap pointing the other way.
func TestUnitSpace_NilRendersAsUnavailableNotZero(t *testing.T) {
var u *UnitSpace
s := u.String()
if !strings.Contains(s, "unavailable") {
t.Fatalf("nil UnitSpace renders as %q — it must say the reading is unavailable", s)
}
if strings.Contains(s, "0.0") {
t.Fatalf("nil UnitSpace renders zeros (%q) — an operator would read \"the disk is full\" "+
"from a filesystem nobody could read", s)
}
}
@@ -37,6 +37,7 @@ func (f *fakeRecoveryProvider) GetStackComposePath(string) (string, bool) {
func (f *fakeRecoveryProvider) ListDeployedStacks() []StackSummary { return nil } func (f *fakeRecoveryProvider) ListDeployedStacks() []StackSummary { return nil }
func (f *fakeRecoveryProvider) GetStackHDDMounts(string) []string { return nil } func (f *fakeRecoveryProvider) GetStackHDDMounts(string) []string { return nil }
func (f *fakeRecoveryProvider) GetStackHDDPath(string) string { return f.hdd } func (f *fakeRecoveryProvider) GetStackHDDPath(string) string { return f.hdd }
func (f *fakeRecoveryProvider) GetImportRoot() string { return "" } // R-75: no import binds in this fixture
func (f *fakeRecoveryProvider) GetDockerVolumes(string) []string { return nil } func (f *fakeRecoveryProvider) GetDockerVolumes(string) []string { return nil }
func (f *fakeRecoveryProvider) StopStack(string) error { func (f *fakeRecoveryProvider) StopStack(string) error {
f.stopped = true f.stopped = true
@@ -69,12 +70,21 @@ func (f *fakeRecoveryProvider) StartStackServices(_ string, services []string) e
return f.startSvcErr return f.startSvcErr
} }
// TestCaptureRecoveryUnitIsSecretFree proves the captured unit (a) contains compose+config+manifest, // TestCaptureRecoveryUnitCarriesPortableSecretsOnly proves the captured unit (a) contains
// (b) enumerates the existing dumps, and (c) is SECRET-FREE: a secret value present in the SOURCE // compose+config+manifest, (b) enumerates the existing dumps, and (c) implements the D5 secret split:
// app.yaml does NOT appear anywhere in the unit, because the capture writes the stripped NonSecretEnv // the PORTABLE class is written into the unit's app.yaml, and the WITHHELD class appears NOWHERE in
// (not the raw app.yaml). The manifest records the secret NAMES + data_key flag for recovery-from-guest. // the unit.
func TestCaptureRecoveryUnitIsSecretFree(t *testing.T) { //
const secretVal = "SUPERSECRETVALUE-do-not-leak" // This test replaces TestCaptureRecoveryUnitIsSecretFree, whose global "no secret value appears in the
// unit" invariant D5 deliberately overturns for the portable class. The wrong-outcome half — the
// withheld value must still leak nowhere — is kept verbatim, because that is the half that is still a
// security boundary.
func TestCaptureRecoveryUnitCarriesPortableSecretsOnly(t *testing.T) {
const (
dataKeyVal = "DATAKEY-must-travel-or-the-data-is-unreadable"
dbPwVal = "DBPASSWORD-must-travel-or-the-app-cannot-authenticate"
withheldVal = "ADMINLOGIN-must-never-reach-the-drive"
)
tmp := t.TempDir() tmp := t.TempDir()
stackDir := filepath.Join(tmp, "stack") stackDir := filepath.Join(tmp, "stack")
drive := filepath.Join(tmp, "drive") // in-guest namespace root (basename need not be felhom-data) drive := filepath.Join(tmp, "drive") // in-guest namespace root (basename need not be felhom-data)
@@ -82,25 +92,30 @@ func TestCaptureRecoveryUnitIsSecretFree(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
// Source stack files — the raw app.yaml DELIBERATELY holds a secret to prove it's not copied.
mustWrite(t, filepath.Join(stackDir, "docker-compose.yml"), mustWrite(t, filepath.Join(stackDir, "docker-compose.yml"),
"services:\n app:\n image: example/app:1.2.3\n") "services:\n app:\n image: example/app:1.2.3\n")
mustWrite(t, filepath.Join(stackDir, ".felhom.yml"), "display_name: Example\n") mustWrite(t, filepath.Join(stackDir, ".felhom.yml"), "display_name: Example\n")
// The SOURCE app.yaml holds the withheld admin login too, so the leak check below is not vacuous:
// it fails if anything ever copies the raw app.yaml into the unit instead of the generated one.
mustWrite(t, filepath.Join(stackDir, "app.yaml"), mustWrite(t, filepath.Join(stackDir, "app.yaml"),
"deployed: true\nenv:\n DB_PASSWORD: "+secretVal+"\n SUBDOMAIN: example\n") "deployed: true\nenv:\n DB_PASSWORD: "+dbPwVal+"\n ADMIN_PASSWORD: "+withheldVal+
"\n SUBDOMAIN: example\n")
// Pre-existing dumps (written by the dump flow before capture). // Pre-existing dumps (written by the dump flow before capture).
mustWrite(t, filepath.Join(AppDBDumpPath(drive, "example"), "example-postgres.sql"), "dump") mustWrite(t, filepath.Join(AppDBDumpPath(drive, "example"), "example-postgres.sql"), "dump")
mustWrite(t, filepath.Join(AppVolumeDumpPath(drive, "example"), "example_data.tar"), "tar") mustWrite(t, filepath.Join(AppVolumeDumpPath(drive, "example"), "example_data.tar"), "tar")
// RecoveryInfo as the adapter would build it: secret values already stripped from NonSecretEnv. // RecoveryInfo as the adapter builds it: NonSecretEnv holds no secret, PortableSecrets holds the
// decrypted portable class, and ADMIN_PASSWORD is named in SecretEnvVars but NOT portable.
info := RecoveryInfo{ info := RecoveryInfo{
StackDir: stackDir, StackDir: stackDir,
DisplayName: "Example", DisplayName: "Example",
ImagePins: []string{"example/app:1.2.3"}, ImagePins: []string{"example/app:1.2.3"},
NonSecretEnv: map[string]string{"SUBDOMAIN": "example", "HDD_PATH": drive}, NonSecretEnv: map[string]string{"SUBDOMAIN": "example", "HDD_PATH": drive},
SecretEnvVars: []string{"DB_PASSWORD", "SECRET_KEY"}, SecretEnvVars: []string{"DB_PASSWORD", "SECRET_KEY", "ADMIN_PASSWORD"},
DataKeyEnvVars: []string{"SECRET_KEY"}, DataKeyEnvVars: []string{"SECRET_KEY"},
PortableSecretEnvVars: []string{"DB_PASSWORD", "SECRET_KEY"},
PortableSecrets: map[string]string{"DB_PASSWORD": dbPwVal, "SECRET_KEY": dataKeyVal},
} }
m := &Manager{ m := &Manager{
logger: log.New(io.Discard, "", 0), logger: log.New(io.Discard, "", 0),
@@ -135,8 +150,8 @@ func TestCaptureRecoveryUnitIsSecretFree(t *testing.T) {
if len(man.ImagePins) != 1 || man.ImagePins[0] != "example/app:1.2.3" { if len(man.ImagePins) != 1 || man.ImagePins[0] != "example/app:1.2.3" {
t.Errorf("image pins: %v", man.ImagePins) t.Errorf("image pins: %v", man.ImagePins)
} }
if len(man.SecretEnvVars) != 2 { if len(man.SecretEnvVars) != 3 {
t.Errorf("secret env-var names: %v (want 2)", man.SecretEnvVars) t.Errorf("secret env-var names: %v (want 3)", man.SecretEnvVars)
} }
if len(man.DataKeyEnvVars) != 1 || man.DataKeyEnvVars[0] != "SECRET_KEY" { if len(man.DataKeyEnvVars) != 1 || man.DataKeyEnvVars[0] != "SECRET_KEY" {
t.Errorf("data-key env-vars: %v", man.DataKeyEnvVars) t.Errorf("data-key env-vars: %v", man.DataKeyEnvVars)
@@ -144,21 +159,48 @@ func TestCaptureRecoveryUnitIsSecretFree(t *testing.T) {
if len(man.DBDumps) != 1 || len(man.VolumeDumps) != 1 { if len(man.DBDumps) != 1 || len(man.VolumeDumps) != 1 {
t.Errorf("dumps enumerated: db=%v vol=%v", man.DBDumps, man.VolumeDumps) t.Errorf("dumps enumerated: db=%v vol=%v", man.DBDumps, man.VolumeDumps)
} }
// D5: schema 2 + the carried names, so the restore can tell secrets from plain config.
// app.yaml in the unit must carry the non-secret env but NOT the secret value. if man.SchemaVersion != 2 {
appy := mustRead(t, filepath.Join(composeDir, "app.yaml")) t.Errorf("schema version = %d, want 2 (D5 units carry secrets)", man.SchemaVersion)
if !strings.Contains(appy, "SUBDOMAIN") { }
t.Errorf("stripped app.yaml missing non-secret env: %s", appy) if len(man.PortableSecretEnvVars) != 2 {
t.Errorf("portable secret names: %v (want DB_PASSWORD + SECRET_KEY)", man.PortableSecretEnvVars)
}
// The manifest is 0644 — it must record NAMES, never a value.
if s := string(mfData); strings.Contains(s, dbPwVal) || strings.Contains(s, dataKeyVal) {
t.Error("SECRET LEAK: a secret VALUE reached manifest.json (names only)")
} }
// SECRET-FREE invariant: the secret value must not appear ANYWHERE in the unit. // app.yaml in the unit must carry the non-secret env AND the portable secrets.
appyPath := filepath.Join(composeDir, "app.yaml")
appy := mustRead(t, appyPath)
if !strings.Contains(appy, "SUBDOMAIN") {
t.Errorf("unit app.yaml missing non-secret env: %s", appy)
}
// THE CONSEQUENCE of D5 at capture time: without these two values on the drive, a guest-less
// restore cannot read the data sitting beside them.
if !strings.Contains(appy, dataKeyVal) {
t.Error("data-encrypting key did NOT travel — a guest-less restore would be impossible")
}
if !strings.Contains(appy, dbPwVal) {
t.Error("DB password did NOT travel — the restored app could not authenticate to its own data")
}
// Secret-bearing ⇒ owner-only.
if fi, err := os.Stat(appyPath); err != nil {
t.Fatal(err)
} else if perm := fi.Mode().Perm(); perm != 0600 {
t.Errorf("unit app.yaml mode = %04o, want 0600 (it carries secrets)", perm)
}
// THE WRONG-OUTCOME CHECK: the withheld class must appear NOWHERE in the unit. This is the half of
// the old secret-free invariant that D5 does not relax.
unitRoot := RecoveryUnitPath(drive, "example") unitRoot := RecoveryUnitPath(drive, "example")
_ = filepath.WalkDir(unitRoot, func(path string, d fs.DirEntry, err error) error { _ = filepath.WalkDir(unitRoot, func(path string, d fs.DirEntry, err error) error {
if err != nil || d.IsDir() { if err != nil || d.IsDir() {
return nil return nil
} }
if strings.Contains(mustRead(t, path), secretVal) { if strings.Contains(mustRead(t, path), withheldVal) {
t.Errorf("SECRET LEAK: %q found in %s", secretVal, path) t.Errorf("SECRET LEAK: withheld admin login %q found in %s", withheldVal, path)
} }
return nil return nil
}) })
@@ -0,0 +1,46 @@
package backup
import (
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-252 — THE PAGE'S QUESTION AND THE RESOLVER'S ANSWER ARE THE SAME QUESTION.
//
// The restore page renders its "re-attach your drive" notice from HasRestoreDestination(); every
// restore refuses from offboxRestoreScratchDir(). If those two ever disagree the page either warns a
// box that would restore fine, or stays silent on one that cannot — and the second is R-252 exactly
// as the fifth walk met it. So this asserts the CONSEQUENCE (does the resolver refuse?) against the
// predicate the page renders from, on one fixture.
//
// RED-PROOF: make HasRestoreDestination return true unconditionally and the first block fails — the
// predicate claims a destination while the resolver refuses in the very next assertion.
func TestHasRestoreDestination_AgreesWithTheResolver(t *testing.T) {
m, sett := bareManager(t)
// No registered storage path: the resolver refuses, and the predicate must say so.
if m.HasRestoreDestination() {
t.Error("HasRestoreDestination() is true with no registered storage path — the restore page " +
"would stay silent while every restore refuses")
}
if _, _, err := m.offboxRestoreScratchDir("calibre-web"); err == nil {
t.Fatal("the resolver found a destination with no registered storage path — fixture is wrong, " +
"so the agreement below would prove nothing")
} else if !strings.Contains(err.Error(), "Tárhely") {
t.Errorf("the refusal names no route: %v", err)
}
// Re-attach a drive, exactly as the customer does from Tárhely → Meghajtók.
if err := sett.AddStoragePath(settings.StoragePath{Path: "/mnt/felhom-drives/adatok", Label: "Adatok", Schedulable: true}); err != nil {
t.Fatalf("register storage path: %v", err)
}
if !m.HasRestoreDestination() {
t.Error("HasRestoreDestination() is still false after the drive was re-attached — the notice " +
"would stay on screen after the customer fixed the thing it asked them to fix")
}
if _, _, err := m.offboxRestoreScratchDir("calibre-web"); err != nil {
t.Errorf("the resolver still refuses after a drive was registered: %v", err)
}
}
+90 -28
View File
@@ -11,27 +11,53 @@ import (
) )
// reconcileRestoreSecrets merges the recovery unit's non-secret env with the secrets recovered from // reconcileRestoreSecrets merges the recovery unit's non-secret env with the secrets recovered from
// the guest's own app.yaml, and applies the FAIL-CLOSED data-key gate. It is the safety-critical heart // the unit itself (D5) and from the guest's own app.yaml, and applies the FAIL-CLOSED data-key gate.
// of Phase 2b and is deliberately a pure function (no I/O) so it can be exhaustively unit-tested. // It is the safety-critical heart of Phase 2b and is deliberately a pure function (no I/O) so it can
// be exhaustively unit-tested — the D5 source arrives as an ARGUMENT, not as a read.
// //
// Policy (per the Phase 2 design — see REPORT/CHANGELOG): // Policy:
// - Regenerate NOTHING. Every secret comes from the guest (live rootfs, or PBS whole-guest restore). // - Regenerate NOTHING here. Secrets come from the unit (portable class) or the guest (the rest).
// - A missing DATA-ENCRYPTING key (`dataKeyNames`) is FATAL: regenerating it would render the // - A missing DATA-ENCRYPTING key (`dataKeyNames`) is FATAL: regenerating it would render the
// restored data unreadable, so we refuse and tell the operator to do a PBS whole-guest restore. // restored data unreadable, so we refuse and tell the operator to do a PBS whole-guest restore.
// - A missing resettable secret (DB password, admin password) is NON-fatal: it's returned in // D5 means the key is normally IN the unit — but "normally" is not a reason to soften the gate.
// `missing` so the caller can warn; the app may simply need a credential reset, no data is lost. // - A missing resettable secret is NON-fatal: returned in `missing` so the caller can warn or
func reconcileRestoreSecrets(nonSecretEnv, recoveredSecrets map[string]string, secretNames, dataKeyNames []string) (fullEnv map[string]string, missing []string, err error) { // regenerate it (O4). No data is lost.
//
// PRECEDENCE — the UNIT WINS over the guest when both hold a value for the same name.
//
// This is not arbitrary and it is not "newest wins". The unit's secrets are captured in the SAME run
// as the dumps beside them (runVolumeDumps → captureAllRecoveryUnits, backup.go), so the unit's value
// is the one that MATCHES THE DATA ABOUT TO BE RESTORED, whereas the guest's value is merely the most
// recent. Where they disagree the guest's has been rotated since the capture, and preferring it is
// precisely the data-loss bug:
// - a rotated data-encrypting key does not decrypt data encrypted with the old one;
// - a rotated DB password does not match the scram/mysql hash inside the restored data directory
// (POSTGRES_PASSWORD is ignored once PGDATA is non-empty), so the app cannot reach its own rows.
//
// The restore persists fullEnv back to the guest's app.yaml (RecreateStackDefinitionFromUnit), so
// unit-wins also leaves the guest consistent with the data now on disk.
func reconcileRestoreSecrets(nonSecretEnv, unitSecrets, guestSecrets map[string]string, secretNames, dataKeyNames []string) (fullEnv map[string]string, missing []string, err error) {
fullEnv = make(map[string]string, len(nonSecretEnv)+len(secretNames)) fullEnv = make(map[string]string, len(nonSecretEnv)+len(secretNames))
for k, v := range nonSecretEnv { for k, v := range nonSecretEnv {
fullEnv[k] = v fullEnv[k] = v
} }
// resolve applies the precedence: unit first, guest only as a fallback.
resolve := func(n string) (string, bool) {
if v, ok := unitSecrets[n]; ok && v != "" {
return v, true
}
if v, ok := guestSecrets[n]; ok && v != "" {
return v, true
}
return "", false
}
have := func(n string) bool { have := func(n string) bool {
v, ok := recoveredSecrets[n] _, ok := resolve(n)
return ok && v != "" return ok
} }
for _, n := range secretNames { for _, n := range secretNames {
if have(n) { if v, ok := resolve(n); ok {
fullEnv[n] = recoveredSecrets[n] fullEnv[n] = v
} else { } else {
missing = append(missing, n) missing = append(missing, n)
} }
@@ -45,24 +71,42 @@ func reconcileRestoreSecrets(nonSecretEnv, recoveredSecrets map[string]string, s
} }
if len(missingDataKeys) > 0 { if len(missingDataKeys) > 0 {
return nil, missing, fmt.Errorf( return nil, missing, fmt.Errorf(
"refusing to restore: data-encrypting key(s) %v could not be recovered from the guest's app.yaml — "+ "refusing to restore: data-encrypting key(s) %v are in NEITHER the recovery unit nor the guest's app.yaml — "+
"a PBS whole-guest restore is required first (regenerating the key would render stored data unreadable)", "a PBS whole-guest restore is required first (regenerating the key would render stored data unreadable)",
missingDataKeys) missingDataKeys)
} }
return fullEnv, missing, nil return fullEnv, missing, nil
} }
// readStrippedEnv parses the non-secret env from a recovery unit's secret-stripped app.yaml. // readUnitEnv parses a recovery unit's app.yaml and SPLITS it into the plain config env and the
func readStrippedEnv(path string) map[string]string { // secrets the unit carries (D5), using the manifest's portable-secret names as the discriminator.
//
// The split is driven by the MANIFEST, not by guessing from key names: the manifest and the app.yaml
// are captured together and checksummed together, so they cannot disagree about which entries are
// secrets. A schema-1 unit has no portable names, so everything lands in nonSecret — exactly the
// pre-D5 behaviour, which is what makes an old unit still restorable.
func readUnitEnv(path string, portableNames []string) (nonSecret, unitSecrets map[string]string) {
nonSecret, unitSecrets = map[string]string{}, map[string]string{}
data, err := os.ReadFile(path) data, err := os.ReadFile(path)
if err != nil { if err != nil {
return map[string]string{} return nonSecret, unitSecrets
} }
var s strippedAppYaml var s strippedAppYaml
if yaml.Unmarshal(data, &s) != nil || s.Env == nil { if yaml.Unmarshal(data, &s) != nil || s.Env == nil {
return map[string]string{} return nonSecret, unitSecrets
} }
return s.Env isPortable := make(map[string]bool, len(portableNames))
for _, n := range portableNames {
isPortable[n] = true
}
for k, v := range s.Env {
if isPortable[k] {
unitSecrets[k] = v
continue
}
nonSecret[k] = v
}
return nonSecret, unitSecrets
} }
// hasReplayableDump reports whether dumpDir holds a .sql dump that the replay could actually use. // hasReplayableDump reports whether dumpDir holds a .sql dump that the replay could actually use.
@@ -85,13 +129,17 @@ func hasReplayableDump(dumpDir string) bool {
return false return false
} }
// RestoreFromRecoveryUnit recreates an app from its on-drive recovery unit + the guest's own secrets. // RestoreFromRecoveryUnit recreates an app from its on-drive recovery unit.
// //
// It reads the unit manifest, recovers the secret values from the guest's live app.yaml, applies the // It reads the unit manifest, takes the portable secrets from the UNIT and the rest from the guest's
// fail-closed data-key gate, restores the named-volume data from the unit's tars, then restores the // live app.yaml (unit wins — see reconcileRestoreSecrets), applies the fail-closed data-key gate,
// app's definition from the unit and redeploys it with the reconstructed env (re-pulling the pinned // restores the named-volume data from the unit's tars, then restores the app's definition from the unit
// image). No secret is ever regenerated, and no secret is read from the unit. If no unit exists it // and redeploys it with the reconstructed env (re-pulling the pinned image). If no unit exists it falls
// falls back to the legacy volume-only RestoreApp. // back to the legacy volume-only RestoreApp.
//
// D5: this no longer needs the guest. A restore with the guest's app.yaml absent succeeds, which is
// pinned by TestRestoreFromRecoveryUnitWithGuestAbsent — the withheld class is regenerated (O4) and
// only a data key missing from BOTH sources still refuses.
func (m *Manager) RestoreFromRecoveryUnit(stackName string) error { func (m *Manager) RestoreFromRecoveryUnit(stackName string) error {
if m.stackProvider == nil { if m.stackProvider == nil {
return fmt.Errorf("stack provider not configured") return fmt.Errorf("stack provider not configured")
@@ -126,11 +174,15 @@ func (m *Manager) RestoreFromRecoveryUnit(stackName string) error {
} }
composeDir := RecoveryUnitComposePath(nsRoot, stackName) composeDir := RecoveryUnitComposePath(nsRoot, stackName)
nonSecretEnv := readStrippedEnv(filepath.Join(composeDir, "app.yaml")) nonSecretEnv, unitSecrets := readUnitEnv(filepath.Join(composeDir, "app.yaml"), manifest.PortableSecretEnvVars)
// Recover secrets from the GUEST (never the unit), then apply the fail-closed gate. // D5: the unit carries the portable class, so this is the leg that no longer needs the guest. The
recovered := m.stackProvider.RecoverStackSecrets(stackName, manifest.SecretEnvVars) // guest is still consulted for the WITHHELD class (internet-reachable admin logins) and as the
fullEnv, missing, err := reconcileRestoreSecrets(nonSecretEnv, recovered, manifest.SecretEnvVars, manifest.DataKeyEnvVars) // fallback for a schema-1 unit — it returns an empty map when the guest is gone, which is the whole
// point: a Tier-1/2 restore must survive that. Precedence is unit-over-guest (see
// reconcileRestoreSecrets), then the fail-closed gate.
guestSecrets := m.stackProvider.RecoverStackSecrets(stackName, manifest.SecretEnvVars)
fullEnv, missing, err := reconcileRestoreSecrets(nonSecretEnv, unitSecrets, guestSecrets, manifest.SecretEnvVars, manifest.DataKeyEnvVars)
if err != nil { if err != nil {
m.logger.Printf("[ERROR] [backup] Restore REFUSED for %s: %v", stackName, err) m.logger.Printf("[ERROR] [backup] Restore REFUSED for %s: %v", stackName, err)
return err return err
@@ -141,6 +193,16 @@ func (m *Manager) RestoreFromRecoveryUnit(stackName string) error {
// encrypted in the guest app.yaml and round-trips on the next backup/restore. Data-keys are // encrypted in the guest app.yaml and round-trips on the next backup/restore. Data-keys are
// never generated: the fail-closed gate above already refused if one was missing, and the // never generated: the fail-closed gate above already refused if one was missing, and the
// generator itself refuses data-key fields (defense-in-depth). Values are never logged. // generator itself refuses data-key fields (defense-in-depth). Values are never logged.
//
// D5 shrinks this path to the rare case: the portable class now comes from the unit, so a
// generator run means the secret was empty at capture AND absent from the guest.
//
// It does NOT claim the reset is harmless. R-127: for a DB password it is not — a restored data
// directory keeps the OLD role hash (POSTGRES_PASSWORD is ignored once PGDATA is non-empty), so a
// regenerated value leaves the app unable to authenticate against its own restored rows while the
// dump replay, which uses the container's local trust socket, still reports success. The old wording
// here asserted "stored data is unaffected" for every non-data-key secret; that is false for the 18
// DB/root-password fields and is now scoped to what is actually true.
if len(missing) > 0 { if len(missing) > 0 {
dataKeySet := make(map[string]bool, len(manifest.DataKeyEnvVars)) dataKeySet := make(map[string]bool, len(manifest.DataKeyEnvVars))
for _, dk := range manifest.DataKeyEnvVars { for _, dk := range manifest.DataKeyEnvVars {
@@ -158,7 +220,7 @@ func (m *Manager) RestoreFromRecoveryUnit(stackName string) error {
unresolved = append(unresolved, name) unresolved = append(unresolved, name)
} }
if len(generated) > 0 { if len(generated) > 0 {
m.logger.Printf("[WARN] [backup] Restore %s: generated replacement for %v — the credential was reset (old value unrecoverable); stored data is unaffected (no data-key involved)", m.logger.Printf("[WARN] [backup] Restore %s: generated replacement for %v — the credential was reset (old value unrecoverable); no data-encrypting key was involved, but a regenerated DATABASE password will not match the restored data directory's stored hash (R-127) — check the app can reach its data",
stackName, generated) stackName, generated)
} }
if len(unresolved) > 0 { if len(unresolved) > 0 {
+260 -11
View File
@@ -3,10 +3,123 @@ package backup
import ( import (
"io" "io"
"log" "log"
"os"
"path/filepath" "path/filepath"
"strings"
"testing" "testing"
) )
// captureFixtureUnit writes a real stack tree, runs the REAL CaptureRecoveryUnit over it, and returns
// the drive (namespace root) holding the resulting unit.
//
// Fixtures come from a CAPTURED unit rather than hand-written YAML deliberately: the capture side and
// the restore side must meet at real bytes on a real filesystem, so a change to the on-disk shape
// (header text, key ordering, the non-secret/portable split) cannot pass by having a test agree with
// itself. Everything from buildUnitAppYaml through readUnitEnv is production code here.
func captureFixtureUnit(t *testing.T, portable map[string]string) (drive string) {
t.Helper()
tmp := t.TempDir()
stackDir := filepath.Join(tmp, "stack")
drive = filepath.Join(tmp, "drive")
if err := os.MkdirAll(stackDir, 0755); err != nil {
t.Fatal(err)
}
mustWrite(t, filepath.Join(stackDir, "docker-compose.yml"),
"services:\n app:\n image: example/app:1\n db:\n image: postgres:16\n")
mustWrite(t, filepath.Join(stackDir, ".felhom.yml"), "display_name: App\n")
mustWrite(t, filepath.Join(stackDir, "app.yaml"), "deployed: true\nenv:\n SUBDOMAIN: trips\n")
var names []string
for _, n := range []string{"DB_PASSWORD", "SECRET_KEY"} {
if _, ok := portable[n]; ok {
names = append(names, n)
}
}
info := RecoveryInfo{
StackDir: stackDir,
DisplayName: "App",
ImagePins: []string{"example/app:1"},
NonSecretEnv: map[string]string{"SUBDOMAIN": "trips"},
SecretEnvVars: []string{"DB_PASSWORD", "SECRET_KEY"},
DataKeyEnvVars: []string{"SECRET_KEY"},
PortableSecretEnvVars: names,
PortableSecrets: portable,
}
m := &Manager{
logger: log.New(io.Discard, "", 0),
systemDataPath: filepath.Join(tmp, "system"),
stackProvider: &fakeRecoveryProvider{info: info, hdd: drive},
version: "vtest",
}
if err := m.CaptureRecoveryUnit("app"); err != nil {
t.Fatalf("capture fixture: %v", err)
}
return drive
}
// TestRestoreFromRecoveryUnitWithGuestAbsent is D5's entire claim, as a test rather than a
// description: a Tier-1/2 restore SUCCEEDS when the guest's app.yaml is unavailable.
//
// SEAM (R-125): injection is at Manager.stackProvider only — i.e. the docker/compose operations and the
// guest's app.yaml decrypt. RecoverStackSecrets returning nil IS the guest being gone: it is exactly
// what the real adapter returns when the stack or its app.yaml cannot be read (main.go
// GetStack/LoadAppConfigDecrypted nil paths). Everything under test is production code: the unit was
// written by the real CaptureRecoveryUnit, read back by the real readUnitEnv, and reconciled by the
// real reconcileRestoreSecrets.
func TestRestoreFromRecoveryUnitWithGuestAbsent(t *testing.T) {
const (
dataKey = "deadbeefdeadbeef"
dbPw = "pw-from-the-drive"
)
drive := captureFixtureUnit(t, map[string]string{"DB_PASSWORD": dbPw, "SECRET_KEY": dataKey})
// The guest is GONE: no secrets recoverable from it at all.
fake := &fakeRecoveryProvider{hdd: drive, running: true, secrets: nil}
m := &Manager{logger: log.New(io.Discard, "", 0),
systemDataPath: filepath.Join(drive, "..", "sys"), stackProvider: fake}
if err := m.RestoreFromRecoveryUnit("app"); err != nil {
t.Fatalf("restore must succeed from the drive alone, got: %v", err)
}
if fake.gotEnv == nil {
t.Fatal("recreate was not called — the restore did not reach the redeploy")
}
// The consequence: the app is redeployed with the key that decrypts the data beside it.
if fake.gotEnv["SECRET_KEY"] != dataKey {
t.Errorf("data-encrypting key not recovered from the unit: %q", fake.gotEnv["SECRET_KEY"])
}
if fake.gotEnv["DB_PASSWORD"] != dbPw {
t.Errorf("DB password not recovered from the unit: %q", fake.gotEnv["DB_PASSWORD"])
}
if fake.gotEnv["SUBDOMAIN"] != "trips" {
t.Errorf("plain config lost: %v", fake.gotEnv)
}
}
// TestRestoreFromRecoveryUnitGuestAbsentStillFailsClosed proves D5 did not soften the gate: with the
// data key in NEITHER the unit nor the guest, the restore still REFUSES and mutates nothing.
func TestRestoreFromRecoveryUnitGuestAbsentStillFailsClosed(t *testing.T) {
// Unit carries only the DB password — the data key is absent from both sources.
drive := captureFixtureUnit(t, map[string]string{"DB_PASSWORD": "pw"})
fake := &fakeRecoveryProvider{hdd: drive, running: true, secrets: nil}
m := &Manager{logger: log.New(io.Discard, "", 0),
systemDataPath: filepath.Join(drive, "..", "sys"), stackProvider: fake}
err := m.RestoreFromRecoveryUnit("app")
if err == nil {
t.Fatal("expected fail-closed refusal when the data key is in neither source")
}
if !strings.Contains(err.Error(), "SECRET_KEY") {
t.Errorf("refusal should name the missing data key, got: %v", err)
}
if fake.gotEnv != nil {
t.Errorf("recreate must NOT be called on refusal, got %v", fake.gotEnv)
}
if fake.stopped {
t.Error("the live app must not be stopped when the restore refuses")
}
}
// TestRestoreFromRecoveryUnitOrchestration exercises the full in-process flow: read manifest → // TestRestoreFromRecoveryUnitOrchestration exercises the full in-process flow: read manifest →
// recover secrets → apply gate → recreate with the reconciled env. It proves (a) on success the // recover secrets → apply gate → recreate with the reconciled env. It proves (a) on success the
// recreate is called with non-secret env + recovered secrets merged, and (b) on a missing data-key the // recreate is called with non-secret env + recovered secrets merged, and (b) on a missing data-key the
@@ -15,7 +128,7 @@ func TestRestoreFromRecoveryUnitOrchestration(t *testing.T) {
newUnit := func(t *testing.T) (drive string) { newUnit := func(t *testing.T) (drive string) {
tmp := t.TempDir() tmp := t.TempDir()
drive = filepath.Join(tmp, "drive") drive = filepath.Join(tmp, "drive")
// stripped (secret-free) app.yaml in the unit // A schema-1 unit: no portable secrets, so the guest is the only source (pre-D5 behaviour).
mustWrite(t, filepath.Join(RecoveryUnitComposePath(drive, "app"), "app.yaml"), mustWrite(t, filepath.Join(RecoveryUnitComposePath(drive, "app"), "app.yaml"),
"deployed: true\nenv:\n SUBDOMAIN: trips\n") "deployed: true\nenv:\n SUBDOMAIN: trips\n")
man := &RecoveryManifest{SchemaVersion: 1, AppName: "app", ControllerVer: "v", man := &RecoveryManifest{SchemaVersion: 1, AppName: "app", ControllerVer: "v",
@@ -49,6 +162,21 @@ func TestRestoreFromRecoveryUnitOrchestration(t *testing.T) {
} }
}) })
t.Run("schema-1 unit still restores from the guest — no regression", func(t *testing.T) {
// An old unit carries nothing; the guest must still be able to supply everything.
drive := newUnit(t)
fake := &fakeRecoveryProvider{hdd: drive, running: true,
secrets: map[string]string{"DB_PASSWORD": "pw", "SECRET_KEY": "deadbeef"}}
m := &Manager{logger: log.New(io.Discard, "", 0),
systemDataPath: filepath.Join(drive, "..", "sys"), stackProvider: fake}
if err := m.RestoreFromRecoveryUnit("app"); err != nil {
t.Fatalf("a pre-D5 unit must still restore: %v", err)
}
if fake.gotEnv["SECRET_KEY"] != "deadbeef" {
t.Errorf("guest fallback lost the data key: %v", fake.gotEnv)
}
})
t.Run("data-key unrecoverable — REFUSED, recreate not called", func(t *testing.T) { t.Run("data-key unrecoverable — REFUSED, recreate not called", func(t *testing.T) {
drive := newUnit(t) drive := newUnit(t)
fake := &fakeRecoveryProvider{ fake := &fakeRecoveryProvider{
@@ -67,13 +195,82 @@ func TestRestoreFromRecoveryUnitOrchestration(t *testing.T) {
}) })
} }
// TestReconcileRestoreSecretsPrecedence pins the D5 precedence rule in BOTH directions. An undefined
// precedence between two sources of a decryption key is a data-loss bug waiting for its first
// disagreement, so this is not a style question.
//
// The UNIT wins: its secrets were captured in the same run as the dumps beside them, so the unit's
// value is the one that matches the data about to be restored. The guest's value is merely the most
// recent — and a rotated key does not decrypt data encrypted with the old one.
func TestReconcileRestoreSecretsPrecedence(t *testing.T) {
nonSecret := map[string]string{"SUBDOMAIN": "trips"}
names := []string{"DB_PASSWORD", "SECRET_KEY"}
dataKeys := []string{"SECRET_KEY"}
t.Run("both sources disagree — the UNIT wins", func(t *testing.T) {
unit := map[string]string{"DB_PASSWORD": "unit-pw", "SECRET_KEY": "unit-key"}
guest := map[string]string{"DB_PASSWORD": "guest-pw", "SECRET_KEY": "guest-key"}
full, missing, err := reconcileRestoreSecrets(nonSecret, unit, guest, names, dataKeys)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if len(missing) != 0 {
t.Errorf("missing: %v", missing)
}
if full["SECRET_KEY"] != "unit-key" {
t.Errorf("data key: got %q, want the UNIT's value (it matches the restored data)", full["SECRET_KEY"])
}
if full["DB_PASSWORD"] != "unit-pw" {
t.Errorf("DB password: got %q, want the UNIT's value (it matches the restored data dir hash)", full["DB_PASSWORD"])
}
})
t.Run("unit silent — the GUEST fills in", func(t *testing.T) {
// The withheld class (admin logins) is never in the unit, so this direction must work too.
guest := map[string]string{"DB_PASSWORD": "guest-pw", "SECRET_KEY": "guest-key"}
full, _, err := reconcileRestoreSecrets(nonSecret, nil, guest, names, dataKeys)
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if full["SECRET_KEY"] != "guest-key" || full["DB_PASSWORD"] != "guest-pw" {
t.Errorf("guest fallback not applied: %v", full)
}
})
t.Run("unit present but EMPTY for a name — the guest fills in", func(t *testing.T) {
// An empty value is not a value; it must not shadow a good one from the guest.
unit := map[string]string{"SECRET_KEY": ""}
guest := map[string]string{"SECRET_KEY": "guest-key", "DB_PASSWORD": "guest-pw"}
full, _, err := reconcileRestoreSecrets(nonSecret, unit, guest, names, dataKeys)
if err != nil {
t.Fatalf("an empty unit value must fall through to the guest, got: %v", err)
}
if full["SECRET_KEY"] != "guest-key" {
t.Errorf("empty unit value shadowed the guest: %q", full["SECRET_KEY"])
}
})
t.Run("a portable secret never shadows plain config", func(t *testing.T) {
// GetStackRecoveryInfo keeps the two sets disjoint; if that ever breaks, the merge order in
// buildUnitAppYaml decides silently. Pin the intended outcome.
full, _, err := reconcileRestoreSecrets(map[string]string{"DB_PASSWORD": "should-not-win"},
map[string]string{"DB_PASSWORD": "unit-pw"}, nil, []string{"DB_PASSWORD"}, nil)
if err != nil {
t.Fatal(err)
}
if full["DB_PASSWORD"] != "unit-pw" {
t.Errorf("the secret source must win over a stray non-secret entry: %q", full["DB_PASSWORD"])
}
})
}
// TestReconcileRestoreSecrets covers the safety-critical fail-closed gate + secret reconciliation. // TestReconcileRestoreSecrets covers the safety-critical fail-closed gate + secret reconciliation.
func TestReconcileRestoreSecrets(t *testing.T) { func TestReconcileRestoreSecrets(t *testing.T) {
nonSecret := map[string]string{"SUBDOMAIN": "trips", "DOMAIN": "demo-felhom.eu"} nonSecret := map[string]string{"SUBDOMAIN": "trips", "DOMAIN": "demo-felhom.eu"}
t.Run("all recovered, no data_key — full env, no error", func(t *testing.T) { t.Run("all recovered, no data_key — full env, no error", func(t *testing.T) {
recovered := map[string]string{"DB_PASSWORD": "pw", "SECRET_KEY": "deadbeef"} guest := map[string]string{"DB_PASSWORD": "pw", "SECRET_KEY": "deadbeef"}
full, missing, err := reconcileRestoreSecrets(nonSecret, recovered, full, missing, err := reconcileRestoreSecrets(nonSecret, nil, guest,
[]string{"DB_PASSWORD", "SECRET_KEY"}, nil) []string{"DB_PASSWORD", "SECRET_KEY"}, nil)
if err != nil { if err != nil {
t.Fatalf("unexpected error: %v", err) t.Fatalf("unexpected error: %v", err)
@@ -87,9 +284,9 @@ func TestReconcileRestoreSecrets(t *testing.T) {
} }
}) })
t.Run("data_key missing — FAIL CLOSED (refuse)", func(t *testing.T) { t.Run("data_key missing from BOTH sources — FAIL CLOSED (refuse)", func(t *testing.T) {
recovered := map[string]string{"DB_PASSWORD": "pw"} // SECRET_KEY (a data_key) is gone guest := map[string]string{"DB_PASSWORD": "pw"} // SECRET_KEY (a data_key) is gone
full, _, err := reconcileRestoreSecrets(nonSecret, recovered, full, _, err := reconcileRestoreSecrets(nonSecret, nil, guest,
[]string{"DB_PASSWORD", "SECRET_KEY"}, []string{"SECRET_KEY"}) []string{"DB_PASSWORD", "SECRET_KEY"}, []string{"SECRET_KEY"})
if err == nil { if err == nil {
t.Fatal("expected fail-closed error for missing data-encrypting key, got nil") t.Fatal("expected fail-closed error for missing data-encrypting key, got nil")
@@ -99,17 +296,34 @@ func TestReconcileRestoreSecrets(t *testing.T) {
} }
}) })
t.Run("data_key empty value — FAIL CLOSED", func(t *testing.T) { t.Run("data_key empty in both — FAIL CLOSED", func(t *testing.T) {
recovered := map[string]string{"SECRET_KEY": ""} // present but empty == unrecoverable guest := map[string]string{"SECRET_KEY": ""} // present but empty == unrecoverable
_, _, err := reconcileRestoreSecrets(nonSecret, recovered, []string{"SECRET_KEY"}, []string{"SECRET_KEY"}) _, _, err := reconcileRestoreSecrets(nonSecret, map[string]string{"SECRET_KEY": ""}, guest,
[]string{"SECRET_KEY"}, []string{"SECRET_KEY"})
if err == nil { if err == nil {
t.Fatal("empty data-key value must fail closed") t.Fatal("empty data-key value must fail closed")
} }
}) })
t.Run("data_key recovered from the UNIT — no refusal", func(t *testing.T) {
// The D5 case: the guest is gone but the unit carries the key, so the gate must NOT fire.
unit := map[string]string{"SECRET_KEY": "deadbeef", "DB_PASSWORD": "pw"}
full, missing, err := reconcileRestoreSecrets(nonSecret, unit, nil,
[]string{"DB_PASSWORD", "SECRET_KEY"}, []string{"SECRET_KEY"})
if err != nil {
t.Fatalf("the unit's data key must satisfy the gate: %v", err)
}
if len(missing) != 0 {
t.Errorf("nothing should be missing: %v", missing)
}
if full["SECRET_KEY"] != "deadbeef" {
t.Errorf("data key wrong: %v", full)
}
})
t.Run("resettable secret missing — proceed with warning", func(t *testing.T) { t.Run("resettable secret missing — proceed with warning", func(t *testing.T) {
recovered := map[string]string{"SECRET_KEY": "deadbeef"} // data_key ok; DB_PASSWORD missing guest := map[string]string{"SECRET_KEY": "deadbeef"} // data_key ok; DB_PASSWORD missing
full, missing, err := reconcileRestoreSecrets(nonSecret, recovered, full, missing, err := reconcileRestoreSecrets(nonSecret, nil, guest,
[]string{"DB_PASSWORD", "SECRET_KEY"}, []string{"SECRET_KEY"}) []string{"DB_PASSWORD", "SECRET_KEY"}, []string{"SECRET_KEY"})
if err != nil { if err != nil {
t.Fatalf("a missing resettable secret must NOT fail closed: %v", err) t.Fatalf("a missing resettable secret must NOT fail closed: %v", err)
@@ -125,3 +339,38 @@ func TestReconcileRestoreSecrets(t *testing.T) {
} }
}) })
} }
// TestReadUnitEnvSplitsByManifest proves the split is driven by the manifest's portable names, and that
// a schema-1 unit (no names) degrades to "everything is plain config" rather than losing entries.
func TestReadUnitEnvSplitsByManifest(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "app.yaml")
mustWrite(t, path, "deployed: true\nenv:\n SUBDOMAIN: trips\n DB_PASSWORD: pw\n SECRET_KEY: key\n")
t.Run("named secrets land in unitSecrets, the rest in nonSecret", func(t *testing.T) {
nonSecret, unitSecrets := readUnitEnv(path, []string{"DB_PASSWORD", "SECRET_KEY"})
if nonSecret["SUBDOMAIN"] != "trips" || len(nonSecret) != 1 {
t.Errorf("nonSecret = %v, want only SUBDOMAIN", nonSecret)
}
if unitSecrets["DB_PASSWORD"] != "pw" || unitSecrets["SECRET_KEY"] != "key" || len(unitSecrets) != 2 {
t.Errorf("unitSecrets = %v, want the two named secrets", unitSecrets)
}
})
t.Run("schema-1 (no portable names) — everything is plain config, nothing lost", func(t *testing.T) {
nonSecret, unitSecrets := readUnitEnv(path, nil)
if len(unitSecrets) != 0 {
t.Errorf("a schema-1 unit carries no secrets, got %v", unitSecrets)
}
if len(nonSecret) != 3 {
t.Errorf("nonSecret should keep every entry, got %v", nonSecret)
}
})
t.Run("absent file — empty maps, no panic", func(t *testing.T) {
nonSecret, unitSecrets := readUnitEnv(filepath.Join(dir, "nope.yaml"), []string{"DB_PASSWORD"})
if len(nonSecret) != 0 || len(unitSecrets) != 0 {
t.Errorf("want empty maps, got %v / %v", nonSecret, unitSecrets)
}
})
}
+270
View File
@@ -0,0 +1,270 @@
package backup
import (
"fmt"
"sort"
"strconv"
"strings"
"sync"
"time"
)
// ── The backup run digest (R-182) ────────────────────────────────────────────────────────────────
//
// WHAT WAS WRONG, MEASURED. On 2026-08-03 nine per-app `recovery_unit_capture_failed` events reached
// the hub and TWO operator e-mails went out. The hub's operator cooldown key is
// `customerID + ":" + eventType + tier-suffix`, and that event carries `app` but no `tier`, so the
// key held **no app identifier**: the first refused app took the hour's slot and every other app's
// failure was discarded — *before* anything was written down, so it left no row on any channel and
// could not be found afterwards.
//
// WHY NOT JUST PUT THE APP IN THE KEY. That was the obvious fix and the operator ruled against it:
// on a full disk it produces one e-mail per app, which is the volume problem wearing the correctness
// problem's clothes. **One digest per run instead**, listing every failure — and separately, every
// failure recorded when it happens.
//
// THE RECORD AND THE NOTIFICATION ARE DIFFERENT THINGS, and that separation is the durable part:
//
// - the RECORD is the per-app `recovery_unit_capture_failed` event, emitted unconditionally, now
// routed record-only by the hub so it never competes for an e-mail slot;
// - the NOTIFICATION is this digest, emitted once per run and only when something failed.
//
// WHY A DIGEST IS SAFE HERE — the one thing that could have made it dangerous. A digest introduces a
// silent-failure path if the ABSENCE of an e-mail could mean "the run never finished". It cannot:
// the hub's daily deadline check raises `expected_backup_missed` / `expected_dbdump_missed`
// (`hub/internal/monitor/deadline.go:396,417`) from the box's REPORT freshness and its stored
// events, entirely independently of any mail the controller chooses to send. Silence therefore still
// means "the run finished and found nothing wrong". If that check is ever weakened, this design
// loses its footing — which is why it is named here and not only in a report.
// runKind distinguishes the paths that can produce a digest. It is carried in the details and shown
// in the subject, because "the nightly run failed" and "the run I just triggered failed" are read
// differently at 07:00.
const (
runKindNightly = "nightly"
runKindManual = "manual"
// runKindRefresh is the PERIODIC status sweep, which captures units outside any backup run.
//
// IT DELIBERATELY CARRIES NO run_id. A run digest gets a unique run id so the hub's 1-hour
// cooldown can never collapse two real runs (the operator ruled on that explicitly). The sweep
// is the opposite case: it can fire every time the status page is polled, so it MUST fall under
// the ordinary cooldown or a full disk becomes a mail flood — the exact failure this whole
// change exists to avoid, arriving from the other direction.
//
// Without this path the sweep's failures would be recorded and never notified, because the
// per-app event is now record-only — a NEW silence introduced while fixing a silence. This is
// what stops that.
runKindRefresh = "refresh"
)
// runFailure is one app's failed or refused leg within a run.
type runFailure struct {
App string
Leg string
Reason string
}
// runSummary is the per-RUN collector. Same lifetime as `admissionSet` and for the same reason: an
// absent collector must mean "no run in flight", never "a stale answer from last night".
type runSummary struct {
mu sync.Mutex
kind string
runID string
attempted map[string]bool // apps this run actually tried to back up
failures []runFailure
}
// beginRunSummary opens the per-run digest scope and returns the closer, mirroring
// beginAdmissionRun. A second call while one is live REPLACES it and the closer restores the
// previous, so nesting cannot silently drop a caller's scope.
func (m *Manager) beginRunSummary(kind, runID string) func() {
m.summaryMu.Lock()
prev := m.summary
m.summary = &runSummary{kind: kind, runID: runID, attempted: map[string]bool{}}
m.summaryMu.Unlock()
return func() {
m.summaryMu.Lock()
m.summary = prev
m.summaryMu.Unlock()
}
}
// noteAttempted records that this run tried to back up an app.
//
// THE DENOMINATOR IS NOT DECORATION. "3 of 4 apps failed" is a catastrophe and "3 of 40" is a bad
// night, and a list of names cannot tell them apart — the operator's first decision, get up now or
// look after coffee, is made from exactly this ratio.
func (m *Manager) noteAttempted(app string) {
m.summaryMu.Lock()
defer m.summaryMu.Unlock()
if m.summary != nil {
m.summary.mu.Lock()
m.summary.attempted[app] = true
m.summary.mu.Unlock()
}
}
// noteFailure records one failed or refused leg. Deliberate skips must NOT come through here —
// §8.1: a drive that is unplugged or decommissioned has its own alert, and putting it in the digest
// turns a nightly e-mail into one the operator learns to ignore.
func (m *Manager) noteFailure(app, leg, reason string) {
m.summaryMu.Lock()
defer m.summaryMu.Unlock()
if m.summary == nil {
return // no run in flight — nothing to summarise
}
m.summary.mu.Lock()
m.summary.attempted[app] = true
m.summary.failures = append(m.summary.failures, runFailure{App: app, Leg: leg, Reason: reason})
m.summary.mu.Unlock()
}
// emitRunSummary sends the digest, if and only if something failed.
//
// A CLEAN RUN EMITS NOTHING — not an empty digest. A nightly "0 failures" mail is an unread mail
// within a week, and it would also destroy the property the whole design rests on: that silence
// means the run finished and found nothing wrong.
func (m *Manager) emitRunSummary() {
m.summaryMu.Lock()
s := m.summary
m.summaryMu.Unlock()
if s == nil {
return
}
s.mu.Lock()
failures := append([]runFailure(nil), s.failures...)
attempted := len(s.attempted)
kind, runID := s.kind, s.runID
s.mu.Unlock()
if len(failures) == 0 {
return
}
// Stable order so two identical runs render identically and a diff of two mails is meaningful.
sort.Slice(failures, func(i, j int) bool {
if failures[i].App != failures[j].App {
return failures[i].App < failures[j].App
}
return failures[i].Leg < failures[j].Leg
})
if m.runSummaryNotify == nil {
// Nil-safe, but say so: an unwired seam here means the digest exists and reaches nobody,
// which is the built-but-never-wired failure this project has shipped four times.
m.logger.Printf("[WARN] [backup] %d app(s) failed in this %s run but no run-summary notifier is wired — "+
"the failures are recorded per app and NOT summarised to the operator", len(failures), kind)
return
}
apps := make([]RunFailureDetail, 0, len(failures))
names := make([]string, 0, len(failures))
for _, f := range failures {
apps = append(apps, RunFailureDetail{App: f.App, Leg: f.Leg, Reason: f.Reason})
names = append(names, f.App)
}
msg := fmt.Sprintf("%d of %d apps failed to back up in this %s run: %s",
len(failures), attempted, kind, strings.Join(dedupeStable(names), ", "))
m.logger.Printf("[INFO] [backup] Run summary: %d of %d apps failed (%s run) — notifying the operator once",
len(failures), attempted, kind)
m.runSummaryNotify(RunSummary{
RunID: runID,
RunKind: kind,
Failed: len(failures),
Attempted: attempted,
Apps: apps,
Usage: m.summaryUsage(),
Message: msg,
})
}
// summaryUsage reads the target filesystem once for the digest. It uses the same seam the reserve
// does, so a test states occupancy as an input rather than manufacturing it. Nil when unreadable —
// which the hub renders as "unavailable", never as zeros.
func (m *Manager) summaryUsage() *UnitSpace {
if m.stackProvider == nil {
return nil
}
for _, st := range m.stackProvider.ListDeployedStacks() {
if u := m.readUnitSpace(st.Name); u != nil {
return u
}
}
return nil
}
func dedupeStable(in []string) []string {
seen := map[string]bool{}
out := in[:0:0]
for _, s := range in {
if !seen[s] {
seen[s] = true
out = append(out, s)
}
}
return out
}
// RunFailureDetail is one app's failure as the hub receives it.
type RunFailureDetail struct {
App string `json:"app"`
Leg string `json:"leg"`
Reason string `json:"reason"`
}
// RunSummary is the digest payload handed to the notifier seam.
type RunSummary struct {
RunID string
RunKind string
Failed int
Attempted int
Apps []RunFailureDetail
Usage *UnitSpace
Message string
}
// SetRunSummaryNotify wires the per-run digest. INIT-ONLY — call once at startup in main.go,
// alongside SetUnitNotify. Nil-safe, but an unwired seam is logged loudly rather than being silently
// the old behaviour.
func (m *Manager) SetRunSummaryNotify(fn func(RunSummary)) {
m.runSummaryNotify = fn
}
// admissionReason returns the human reason this run refused an app, taken from the verdict the
// reserve already recorded. Reused rather than re-derived: the verdict carries the bound term, the
// estimate and the disk figures, and re-deriving them here would be a second source of truth for a
// sentence the operator reads.
func (m *Manager) admissionReason(stackName string) string {
m.admissionMu.Lock()
defer m.admissionMu.Unlock()
if set := m.admission; set != nil {
if v, ok := set.v[stackName]; ok && v.err != nil {
return v.err.Error()
}
}
return "refused by the capture reserve"
}
// runKindFor reports whether this run was the scheduled one or one a person triggered.
//
// The distinction is the operator's ruling: someone pressing the button is actively trying to get a
// backup, so a manual run must report even if the nightly one already wrote this hour. It is carried
// into the subject line because "the nightly run failed" and "the run I just asked for failed" are
// acted on differently.
func (m *Manager) runKindFor() string {
if m.manualRun.Load() {
return runKindManual
}
return runKindNightly
}
// MarkManualRun tags the NEXT backup run as operator-triggered. Called by the API/debug handlers
// that expose a "run backup now" control; the scheduled path leaves it alone.
func (m *Manager) MarkManualRun() { m.manualRun.Store(true) }
// newRunID returns a per-run identifier. It only has to be distinct between two runs on one box
// within the hub's 1-hour cooldown window, which a nanosecond clock reading satisfies; it is never
// persisted, compared across boxes, or used as a security value.
func newRunID() string {
return "run-" + strconv.FormatInt(time.Now().UnixNano(), 36)
}
@@ -0,0 +1,270 @@
package backup
import (
"go/ast"
"go/parser"
"go/token"
"strings"
"testing"
)
// R-182 — one digest per run, listing every failure, and nothing in it that is not a failure.
//
// The assertions here are the CONTENT of the digest, not that a function was called: the defect this
// replaces was one where the machinery ran correctly and the operator was told about one app out of
// nine.
// digestOf runs the two legs that can run without Docker inside one digest scope and returns the
// summary that would have been sent (nil when none). It mirrors what runDBDumpsInternal does; the
// DB leg's wiring is pinned structurally by TestRunSummary_IsWiredIntoTheProductionPath, because
// DiscoverDatabases shells out to `docker`.
func (h *admissionHarness) digestOf(kind string) *RunSummary {
var got *RunSummary
h.m.SetRunSummaryNotify(func(rs RunSummary) { got = &rs })
doneAdm := h.m.beginAdmissionRun()
doneSum := h.m.beginRunSummary(kind, "run-test")
h.m.runVolumeDumps()
h.m.captureAllRecoveryUnits()
h.m.emitRunSummary()
doneSum()
doneAdm()
return got
}
// ── Scenario A — several failures, ONE digest, all of them in it ─────────────────────────────────
func TestRunSummary_ListsEveryFailedApp(t *testing.T) {
apps := []string{"opengist", "privatebin", "immich", "homebox", "nextcloud"}
h := newAdmissionHarness(t, apps...)
for _, a := range apps {
h.setSpace(a, 99, 0.1, 70) // every app refused by the reserve
}
rs := h.digestOf(runKindNightly)
if rs == nil {
t.Fatal("no digest was emitted although five apps failed — this is the 2026-08-03 defect: " +
"nine failures arrived and two e-mails went out")
}
if rs.Failed != len(apps) {
t.Fatalf("digest reports %d failures, want %d — the others were silently discarded, which is "+
"exactly what the per-app path did", rs.Failed, len(apps))
}
seen := map[string]bool{}
for _, a := range rs.Apps {
seen[a.App] = true
if a.Leg == "" {
t.Errorf("%s has no leg name — the operator cannot tell a failed database dump from a "+
"failed volume dump, and they are different problems", a.App)
}
if !strings.Contains(a.Reason, "reserve") {
t.Errorf("%s reason %q does not say why", a.App, a.Reason)
}
}
for _, a := range apps {
if !seen[a] {
t.Errorf("%s is missing from the digest", a)
}
}
// The denominator: "5 of 5" and "5 of 40" are different nights.
if rs.Attempted < len(apps) {
t.Fatalf("attempted=%d, want >= %d — without the denominator a catastrophe and a bad night "+
"read identically", rs.Attempted, len(apps))
}
if rs.Usage == nil {
t.Fatal("the digest carries no filesystem figures — 'one broken app' and 'a full disk' must " +
"be distinguishable before the reasons are read")
}
if !strings.Contains(rs.Message, "of") || !strings.Contains(rs.Message, "nightly") {
t.Fatalf("summary message is not readable as a sentence: %q", rs.Message)
}
}
// ── Scenario B — a clean run is silent ───────────────────────────────────────────────────────────
func TestRunSummary_CleanRunEmitsNothing(t *testing.T) {
h := newAdmissionHarness(t, "opengist", "privatebin")
h.setSpace("opengist", 20, 55, 70)
h.setSpace("privatebin", 20, 55, 70)
if rs := h.digestOf(runKindNightly); rs != nil {
t.Fatalf("a clean run emitted a digest (%+v) — an empty nightly mail is an unread mail within "+
"a week, and it would destroy the property this design rests on: that silence means the "+
"run finished and found nothing wrong", rs)
}
}
// ── Scenario F — deliberate skips are NOT failures ───────────────────────────────────────────────
// A drive that is unplugged or decommissioned has its own alert. Putting it in the digest produces a
// nightly e-mail on a box with an unplugged drive, which trains the operator to ignore digests — and
// an ignored digest is the same outcome as no digest.
func TestRunSummary_DeliberateSkipsAreNotFailures(t *testing.T) {
h := newAdmissionHarness(t, "gone", "retired", "healthy")
for _, a := range []string{"gone", "retired", "healthy"} {
h.setSpace(a, 20, 55, 70) // ample room — nothing may be refused for headroom
}
h.markDisconnected("gone")
h.markDecommissioned("retired")
rs := h.digestOf(runKindNightly)
if rs != nil {
t.Fatalf("a run whose only 'issues' were a disconnected drive and a decommissioned one "+
"emitted a digest: %+v — those have their own alerts", rs)
}
}
// ── The collector's lifetime — the admission.go rule, restated ───────────────────────────────────
// An absent collector must mean "no run in flight", never a stale answer from last night.
func TestRunSummary_ResetsBetweenRuns(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
h.setSpace("opengist", 99, 0.1, 70)
if rs := h.digestOf(runKindNightly); rs == nil || rs.Failed != 1 {
t.Fatalf("run 1: want 1 failure, got %+v", rs)
}
h.setSpace("opengist", 20, 55, 70) // space freed between runs
if rs := h.digestOf(runKindNightly); rs != nil {
t.Fatalf("run 2 reported %+v — the previous run's failures were carried over, so freeing "+
"space could never take effect", rs)
}
}
// A failure noted with NO run in flight must not panic and must not accumulate anywhere.
func TestRunSummary_NoteOutsideARunIsInert(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
h.m.noteFailure("opengist", "volume dump", "boom") // no scope open
h.m.noteAttempted("opengist")
h.m.emitRunSummary() // must be a no-op, not a nil dereference
}
// ── The refresh sweep carries NO run id, deliberately ────────────────────────────────────────────
// The run digest gets a unique id so the hub's 1-hour cooldown can never collapse two real runs. The
// periodic sweep is the opposite case — it can fire on every status poll — so it must fall UNDER the
// cooldown. Getting this backwards turns a full disk into a mail flood, which is the same failure as
// the one being fixed, arriving from the other side.
func TestRunSummary_RefreshSweepHasNoRunID(t *testing.T) {
h := newAdmissionHarness(t, "opengist")
h.setSpace("opengist", 99, 0.1, 70)
var got *RunSummary
h.m.SetRunSummaryNotify(func(rs RunSummary) { got = &rs })
func() {
defer h.m.beginAdmissionRun()()
defer h.m.beginRunSummary(runKindRefresh, "")()
defer h.m.emitRunSummary()
h.m.captureAllRecoveryUnits()
}()
if got == nil {
t.Fatal("the periodic sweep emitted no digest — with the per-app event now record-only, a " +
"capture failure found between runs would be recorded and NEVER notified, which is a " +
"new silence introduced while closing one")
}
if got.RunID != "" {
t.Fatalf("the refresh sweep carries run_id=%q — it must be EMPTY so the hub's ordinary "+
"1-hour cooldown caps it, or a polled status page becomes a mail flood", got.RunID)
}
if got.RunKind != runKindRefresh {
t.Fatalf("run kind = %q, want %q", got.RunKind, runKindRefresh)
}
}
// ── The seam is WIRED — walked as an AST, not grepped ────────────────────────────────────────────
// Four mechanisms in this project have been built and left disconnected. `strings.Contains` cannot
// tell a live call from a commented-out one, so this parses.
func TestRunSummary_IsWiredIntoTheProductionPath(t *testing.T) {
calls := callsByFunc(t, "backup.go")
if !hasStr(calls["runDBDumpsInternal"], "beginRunSummary") {
t.Fatal("runDBDumpsInternal does not open a run-summary scope — every failure would be " +
"recorded per app and none of them summarised, which is the pre-R-182 behaviour")
}
if !hasStr(calls["runDBDumpsInternal"], "emitRunSummary") {
t.Fatal("runDBDumpsInternal never emits the summary — the collector fills and is discarded")
}
// Both dump legs and the capture leg must feed it, or a whole class of failure is invisible.
for fn, where := range map[string]string{
"runDBDumpsInternal": "the database leg",
"runVolumeDumps": "the volume leg",
} {
if !hasStr(calls[fn], "noteFailure") {
t.Fatalf("%s (%s) never calls noteFailure — its failures cannot reach the digest", fn, where)
}
}
capCalls := callsByFunc(t, "recovery_unit.go")
if !hasStr(capCalls["captureAllRecoveryUnits"], "noteFailure") {
t.Fatal("the capture leg never calls noteFailure")
}
// AND THE SEAM ITSELF must be wired in main.go. Without this assertion the collector fills, the
// digest is built, and `runSummaryNotify` is nil — so nothing is ever sent and every test above
// still passes. That is the built-but-never-wired shape exactly, and the first version of this
// test missed it: commenting the wiring out left the suite green.
mainCalls := callsByFunc(t, "../../cmd/controller/main.go")
wired := false
for _, calls := range mainCalls {
if hasStr(calls, "SetRunSummaryNotify") {
wired = true
}
}
if !wired {
t.Fatal("main.go never calls SetRunSummaryNotify — the digest is assembled and handed to a " +
"nil seam, so no operator mail is ever sent. Note this is an AST walk: a commented-out " +
"call still CONTAINS the string, which is why strings.Contains cannot be used here")
}
// The manual paths must tag the run, or Scenario E cannot hold.
for _, f := range []string{"../web/handler_debug.go", "../api/router.go"} {
src := parseFile(t, f)
found := false
ast.Inspect(src, func(n ast.Node) bool {
if ce, ok := n.(*ast.CallExpr); ok {
if sel, ok := ce.Fun.(*ast.SelectorExpr); ok && sel.Sel.Name == "MarkManualRun" {
found = true
}
}
return true
})
if !found {
t.Fatalf("%s never calls MarkManualRun — an operator-triggered run would be labelled "+
"nightly and could be collapsed into it", f)
}
}
}
func parseFile(t *testing.T, path string) *ast.File {
t.Helper()
f, err := parser.ParseFile(token.NewFileSet(), path, nil, 0) // comments dropped
if err != nil {
t.Fatal(err)
}
return f
}
// callsByFunc maps each top-level func to the names it calls, comments excluded.
func callsByFunc(t *testing.T, path string) map[string][]string {
t.Helper()
file := parseFile(t, path)
out := map[string][]string{}
var cur string
ast.Inspect(file, func(n ast.Node) bool {
switch v := n.(type) {
case *ast.FuncDecl:
cur = v.Name.Name
case *ast.CallExpr:
name := ""
switch fn := v.Fun.(type) {
case *ast.Ident:
name = fn.Name
case *ast.SelectorExpr:
name = fn.Sel.Name
}
if name != "" && cur != "" {
out[cur] = append(out[cur], name)
}
}
return true
})
return out
}
+81 -49
View File
@@ -187,10 +187,12 @@ func (m *Manager) tier2FitsSystemDrive(sys string, unitSizeBytes int64) bool {
} }
// Tier-2 v2 layout (Task 3b, architecture §8): backups/secondary/<stack>/ holds // Tier-2 v2 layout (Task 3b, architecture §8): backups/secondary/<stack>/ holds
//
// .felhom-tier2-layout (marker file, content "2" — written LAST, after all legs + reconcile) // .felhom-tier2-layout (marker file, content "2" — written LAST, after all legs + reconcile)
// recovery-unit/ (the unit leg, layout-identical to v1) // recovery-unit/ (the unit leg, layout-identical to v1)
// hdd/<relpath>/ (per-bind HDD legs, relpath-mirroring) // hdd/<relpath>/ (per-bind HDD legs, relpath-mirroring)
// userdata/<relpath>/ (per-bind USERDATA legs) // userdata/<relpath>/ (per-bind USERDATA legs)
//
// Relpath-mirroring represents N>1 dirs + nested binds natively (the v1 flat-appdata N>1 refusal is // Relpath-mirroring represents N>1 dirs + nested binds natively (the v1 flat-appdata N>1 refusal is
// lifted structurally) and makes restore position-derivable (dest relpath → live path under the app's // lifted structurally) and makes restore position-derivable (dest relpath → live path under the app's
// current HDD_PATH). The whole tree is DERIVED from live data — migration is delete-and-rebuild. // current HDD_PATH). The whole tree is DERIVED from live data — migration is delete-and-rebuild.
@@ -528,67 +530,97 @@ func (m *Manager) Tier2Info(stackName string) Tier2Info {
// --- status persistence (drives the "2. mentés" UI card) --- // --- status persistence (drives the "2. mentés" UI card) ---
// withTier2Prefs carries the customer-preference fields (UserDisabled/PreferredTarget) from any // tier2Update applies a run outcome onto a COPY OF THE EXISTING ROW, then persists it.
// existing config into a freshly-built status struct, so a runner status write never clobbers them. //
func (m *Manager) withTier2Prefs(stackName string, cfg *settings.CrossDriveBackup) *settings.CrossDriveBackup { // R-101 Part 2 — SAFE BY CONSTRUCTION, and this replaced a real hazard rather than tidying one. The
if m.settings != nil { // three record* helpers each used to build a WHOLE `CrossDriveBackup` literal, with `withTier2Prefs`
// re-applying exactly two fields (UserDisabled, PreferredTarget). Every other field not named in the
// literal was silently zeroed on every status write. That is fine while the struct is stable and
// catastrophic the moment a field is added: R-101 adds `LastSuccess`, and under the old shape
// `recordTier2Failure` would have CLEARED it — the mirror-image of the defect being fixed, firing on
// the first failure instead of lying dormant.
//
// Starting from the existing row inverts the default: a new field carries over unless a caller
// deliberately overwrites it. The compile-safe form the R-100 review asked for; nothing is preserved
// by a list that can fall out of date.
//
// Callers must therefore CLEAR explicitly what a run invalidates (a stale LastError on success, a
// stale size on failure) — the old behaviour those clears reproduce is preserved exactly.
func (m *Manager) tier2Update(stackName string, mutate func(*settings.CrossDriveBackup)) {
if m.settings == nil {
return
}
var cfg settings.CrossDriveBackup
if existing := m.settings.GetCrossDriveConfig(stackName); existing != nil { if existing := m.settings.GetCrossDriveConfig(stackName); existing != nil {
cfg.UserDisabled = existing.UserDisabled cfg = *existing // value copy — EVERY field carries over by default
cfg.PreferredTarget = existing.PreferredTarget
} }
// One-time migration of a pre-anchor row. Under the old code `LastStatus=="ok"` with a LastRun
// means that run DID succeed, so adopting it as the initial anchor is truthful — and it is what
// keeps the deploy quiet: without it every existing row would flip to "never succeeded" at once
// (all 7 rows on the fleet were pre-anchor). A row whose last known state was an ERROR seeds
// nothing, because nothing in the old data evidences a success.
if !cfg.SuccessTracked {
if cfg.LastStatus == "ok" && cfg.LastRun != "" {
cfg.LastSuccess = cfg.LastRun
}
cfg.SuccessTracked = true
}
mutate(&cfg)
if err := m.settings.SetCrossDriveConfig(stackName, &cfg); err != nil {
m.logger.Printf("[WARN] [backup] Tier 2 status persist for %s failed: %v", stackName, err)
} }
return cfg
} }
func (m *Manager) recordTier2Success(stackName string, target *Tier2Target, sizeBytes int64, warning string, dur time.Duration) { func (m *Manager) recordTier2Success(stackName string, target *Tier2Target, sizeBytes int64, warning string, dur time.Duration) {
if m.settings == nil { now := time.Now().Format(time.RFC3339)
return m.tier2Update(stackName, func(c *settings.CrossDriveBackup) {
} c.Enabled = true
if err := m.settings.SetCrossDriveConfig(stackName, m.withTier2Prefs(stackName, &settings.CrossDriveBackup{ c.Method = "rsync"
Enabled: true, c.DestinationPath = target.NamespaceRoot
Method: "rsync", c.Schedule = "daily"
DestinationPath: target.NamespaceRoot, c.LastRun = now
Schedule: "daily", // R-101: the anchor. Only this branch advances it; no failure branch clears it.
LastRun: time.Now().Format(time.RFC3339), c.LastSuccess = now
LastStatus: "ok", c.LastStatus = "ok"
LastWarning: strings.TrimSpace(warning), c.LastWarning = strings.TrimSpace(warning)
LastDuration: dur.Round(time.Second).String(), c.LastDuration = dur.Round(time.Second).String()
LastSizeHuman: humanizeBytes(sizeBytes), c.LastSizeHuman = humanizeBytes(sizeBytes)
})); err != nil { c.LastError = "" // a success invalidates the previous error
m.logger.Printf("[WARN] [backup] Tier 2 status persist (ok) for %s failed: %v", stackName, err) })
}
} }
func (m *Manager) recordTier2Failure(stackName string, target *Tier2Target, cause error) { func (m *Manager) recordTier2Failure(stackName string, target *Tier2Target, cause error) {
if m.settings == nil { m.tier2Update(stackName, func(c *settings.CrossDriveBackup) {
return c.Enabled = true
} c.Method = "rsync"
if err := m.settings.SetCrossDriveConfig(stackName, m.withTier2Prefs(stackName, &settings.CrossDriveBackup{ c.DestinationPath = target.NamespaceRoot
Enabled: true, c.Schedule = "daily"
Method: "rsync", c.LastRun = time.Now().Format(time.RFC3339) // the ATTEMPT clock — advances on failure, by design
DestinationPath: target.NamespaceRoot, c.LastStatus = "error"
Schedule: "daily", c.LastError = cause.Error()
LastRun: time.Now().Format(time.RFC3339), // LastSuccess is deliberately UNTOUCHED: a failure neither advances nor clears the anchor.
LastStatus: "error", // Clearing it would make one bad night read as "no copy has ever succeeded".
LastError: cause.Error(), c.LastWarning = "" // a warning from the last successful run does not describe this one
})); err != nil { c.LastDuration = "" // preserving the old literal's clears exactly
m.logger.Printf("[WARN] [backup] Tier 2 status persist (error) for %s failed: %v", stackName, err) c.LastSizeHuman = "" // ditto — a stale size would describe a copy this run did not make
} })
} }
func (m *Manager) recordTier2NoTarget(stackName, reason string) { func (m *Manager) recordTier2NoTarget(stackName, reason string) {
if m.settings == nil { m.tier2Update(stackName, func(c *settings.CrossDriveBackup) {
return c.Enabled = false
} c.Method = "rsync"
if err := m.settings.SetCrossDriveConfig(stackName, m.withTier2Prefs(stackName, &settings.CrossDriveBackup{ c.Schedule = "daily"
Enabled: false, c.DestinationPath = ""
Method: "rsync", c.LastStatus = "no_target"
Schedule: "daily", c.LastError = reason
LastStatus: "no_target", c.LastRun = ""
LastError: reason, // LastSuccess survives: "there is no destination drive right now" is not evidence that the
})); err != nil { // last successful copy never happened. The UI gates on LastRun here, so nothing is rendered.
m.logger.Printf("[WARN] [backup] Tier 2 status persist (no_target) for %s failed: %v", stackName, err) c.LastWarning = ""
} c.LastDuration = ""
c.LastSizeHuman = ""
})
} }
func tier2NoTargetReason(err error) string { func tier2NoTargetReason(err error) string {
@@ -0,0 +1,174 @@
package backup
import (
"errors"
"io"
"log"
"path/filepath"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/settings"
)
// R-101 Part 2 — THE COPY-SITE HAZARD, exercised through the REAL record* helpers.
//
// The three record* helpers used to build a WHOLE CrossDriveBackup literal, with a helper re-applying
// exactly two fields. Anything not named in the literal was zeroed on every status write. Adding
// LastSuccess to that shape would have had `recordTier2Failure` CLEAR the anchor — the mirror image of
// the defect being fixed, and firing on the FIRST failure rather than lying dormant.
//
// The R-100 lesson applies: these call the production functions. A test that modelled the copy in a
// closure would stay green through any mutation of the real code.
func anchorMgr(t *testing.T) *Manager {
t.Helper()
sett, err := settings.Load(filepath.Join(t.TempDir(), "settings.json"), log.New(io.Discard, "", 0))
if err != nil {
t.Fatalf("settings.Load: %v", err)
}
return &Manager{settings: sett, logger: log.New(io.Discard, "", 0)}
}
func tgt() *Tier2Target { return &Tier2Target{NamespaceRoot: "/mnt/hdd_1/felhom-data/calibre-web"} }
// SCENARIO F — a FAILURE must not wipe the anchor. This is the Part 2 proof and it runs the real
// recordTier2Success → recordTier2Failure sequence.
//
// RED-PROOF: make recordTier2Failure build a whole literal again (or add `c.LastSuccess = ""` to its
// mutation) → this fails with "a FAILED run wiped the success anchor".
func TestTier2_FailureDoesNotWipeTheAnchor(t *testing.T) {
m := anchorMgr(t)
// Seed a row with a DISTINCT, older anchor (a real success on the 25th) already tracked, so the
// anchor and the attempt clock are distinguishable — a success recorded in this same test would
// land in the same second as the failures and prove nothing about which field moved.
const anchor = "2026-07-25T01:30:00Z"
if err := m.settings.SetCrossDriveConfig("calibre-web", &settings.CrossDriveBackup{
Enabled: true, Method: "rsync", Schedule: "daily",
LastRun: anchor, LastStatus: "ok", LastSuccess: anchor, SuccessTracked: true,
}); err != nil {
t.Fatal(err)
}
// Five consecutive failing nights, through the real helper.
for i := 0; i < 5; i++ {
m.recordTier2Failure("calibre-web", tgt(), errors.New("rsync: connection reset"))
got := m.settings.GetCrossDriveConfig("calibre-web")
if got.LastSuccess == "" {
t.Fatalf("a FAILED run wiped the success anchor (round %d) — one bad night would read as 'no copy has ever succeeded'", i+1)
}
if got.LastSuccess != anchor {
t.Fatalf("a FAILED run MOVED the anchor to %q (was %q) — that is the R-101 defect in mirror image", got.LastSuccess, anchor)
}
if got.LastStatus != "error" {
t.Errorf("the failure was not recorded (status=%q)", got.LastStatus)
}
// the ATTEMPT clock DOES move — that is what made the old rendering wrong
if got.LastRun == anchor {
t.Errorf("LastRun did not advance on the attempt — it still reads the old success time %q", got.LastRun)
}
}
}
// A successful run advances the anchor, or a tier would look permanently stale after one good night.
func TestTier2_SuccessAdvancesTheAnchor(t *testing.T) {
m := anchorMgr(t)
if err := m.settings.SetCrossDriveConfig("calibre-web", &settings.CrossDriveBackup{
Enabled: true, Method: "rsync", Schedule: "daily",
LastRun: "2026-07-25T01:30:00Z", LastStatus: "error",
LastSuccess: "2026-07-25T01:30:00Z", SuccessTracked: true,
}); err != nil {
t.Fatal(err)
}
m.recordTier2Success("calibre-web", tgt(), 1<<30, "", 0)
got := m.settings.GetCrossDriveConfig("calibre-web")
if got.LastSuccess == "2026-07-25T01:30:00Z" {
t.Error("a successful run did not advance the anchor")
}
if got.LastSuccess != got.LastRun {
t.Errorf("the anchor and the attempt clock disagree after a success (%q vs %q)", got.LastSuccess, got.LastRun)
}
if got.LastError != "" {
t.Errorf("a success left the previous error in place (%q)", got.LastError)
}
}
// A customer preference must still survive a status write — the behaviour the old helper existed to
// provide. The new copy-the-row form should give this for free, and this pins that it does.
//
// RED-PROOF: make tier2Update start from a zero-value struct instead of copying the existing row →
// this fails with "a status write wiped the customer's Tier-2 preference".
func TestTier2_StatusWritePreservesCustomerPreference(t *testing.T) {
m := anchorMgr(t)
if err := m.settings.SetTier2Preference("calibre-web", true, "/mnt/hdd_2"); err != nil {
t.Fatal(err)
}
m.recordTier2Failure("calibre-web", tgt(), errors.New("boom"))
got := m.settings.GetCrossDriveConfig("calibre-web")
if !got.UserDisabled || got.PreferredTarget != "/mnt/hdd_2" {
t.Errorf("a status write wiped the customer's Tier-2 preference (UserDisabled=%v PreferredTarget=%q)",
got.UserDisabled, got.PreferredTarget)
}
}
// A no-target write must not destroy the anchor either: "there is no destination drive right now" is
// not evidence that the last successful copy never happened.
func TestTier2_NoTargetKeepsTheAnchor(t *testing.T) {
m := anchorMgr(t)
m.recordTier2Success("calibre-web", tgt(), 1<<30, "", 0)
anchor := m.settings.GetCrossDriveConfig("calibre-web").LastSuccess
m.recordTier2NoTarget("calibre-web", "nincs elérhető második meghajtó")
got := m.settings.GetCrossDriveConfig("calibre-web")
if got.LastSuccess != anchor {
t.Errorf("a no_target write lost the anchor (%q, was %q)", got.LastSuccess, anchor)
}
if got.LastRun != "" {
t.Errorf("no_target should clear the attempt clock as before, got %q", got.LastRun)
}
}
// SCENARIO E (producer half) — a LEGACY row is migrated truthfully on first touch: a row whose last
// known state was a SUCCESS adopts that time as its anchor, so the deploy does not flip every existing
// customer to "never succeeded".
//
// RED-PROOF: delete the `if !cfg.SuccessTracked` seeding block in tier2Update → this fails with
// "a legacy OK row was not migrated".
func TestTier2_LegacyOkRowSeedsItsAnchor(t *testing.T) {
m := anchorMgr(t)
// A pre-R-101 row: status ok, a last_run, no anchor, not tracked.
if err := m.settings.SetCrossDriveConfig("calibre-web", &settings.CrossDriveBackup{
Enabled: true, Method: "rsync", Schedule: "daily",
LastRun: "2026-07-28T01:30:00Z", LastStatus: "ok",
}); err != nil {
t.Fatal(err)
}
// The next run fails — the first new-code touch of this row.
m.recordTier2Failure("calibre-web", tgt(), errors.New("boom"))
got := m.settings.GetCrossDriveConfig("calibre-web")
if !got.SuccessTracked {
t.Fatal("the row was not marked as tracked")
}
if got.LastSuccess != "2026-07-28T01:30:00Z" {
t.Errorf("a legacy OK row was not migrated — its known-good run should have become the anchor (got %q)", got.LastSuccess)
}
}
// ...but a legacy row whose last known state was an ERROR seeds NOTHING: the old data contains no
// evidence of a success, and inventing one would be the original defect.
func TestTier2_LegacyErrorRowSeedsNothing(t *testing.T) {
m := anchorMgr(t)
if err := m.settings.SetCrossDriveConfig("calibre-web", &settings.CrossDriveBackup{
Enabled: true, Method: "rsync", Schedule: "daily",
LastRun: "2026-07-28T01:30:00Z", LastStatus: "error",
}); err != nil {
t.Fatal(err)
}
m.recordTier2Failure("calibre-web", tgt(), errors.New("boom"))
if got := m.settings.GetCrossDriveConfig("calibre-web"); got.LastSuccess != "" {
t.Errorf("a legacy ERROR row invented an anchor (%q) — that is the defect, not the fix", got.LastSuccess)
}
}
+1 -1
View File
@@ -43,7 +43,7 @@ func (m *Manager) tier2CaptureSet(stack, nsRoot string) (legs []tier2Leg, warns
// Classified path. // Classified path.
if m.stackProvider != nil { if m.stackProvider != nil {
if binds, has := m.stackProvider.GetStackClassifiedBinds(stack); has { if binds, has := m.stackProvider.GetStackClassifiedBinds(stack); has {
cs := appbackup.ComputeCaptureSet(binds, has, appbackup.TierSecondary, nsRoot) cs := appbackup.ComputeCaptureSet(binds, has, appbackup.TierSecondary, nsRoot, m.stackProvider.GetImportRoot())
var gaps []string var gaps []string
for _, sk := range cs.Skipped { for _, sk := range cs.Skipped {
if sk.Class == appbackup.ClassMandatory { if sk.Class == appbackup.ClassMandatory {
@@ -0,0 +1,143 @@
package backup
import (
"errors"
"os"
"path/filepath"
"testing"
)
// C9-F1 — the Tier-2 restore reads `hdd/` and `userdata/` only. `recovery-unit/` — the app's DB dumps
// and named-volume tarballs — is written by EVERY Tier-2 run and read by NOTHING on this path.
//
// For 43 of the 53 catalog apps (BookStack, Docmost, Vaultwarden, Gitea, …) that is the app's ENTIRE
// dataset, so the restore was a guaranteed no-op that still took an outage and reported
// „Nincs hiányzó fájl — minden fájl megvan a helyén."
//
// These tests pin the asymmetry itself, so a future change that alters what the restore reads must
// either keep the coverage answer honest or fail here.
// unitOnlyCopy rewrites the fixture's copy into the BookStack shape: a recovery unit and nothing the
// restore can read. Mirrors the live demo-felhom layout observed in Campaign 9
// (`legs=[NONE] unit=156M`).
func unitOnlyCopy(t *testing.T, destDrive string) string {
t.Helper()
destBase := filepath.Join(destDrive, "backups", "secondary", "app")
if err := os.RemoveAll(filepath.Join(destBase, "hdd")); err != nil {
t.Fatal(err)
}
mustWrite(t, filepath.Join(destBase, "recovery-unit", "volume-dumps", "app_db_data.tar"), "TARBYTES")
mustWrite(t, filepath.Join(destBase, "recovery-unit", "db-dumps", "app.sql"), "SQLDUMP")
return destBase
}
// SCENARIO D — a restore that cannot cover an app refuses BEFORE the outage.
//
// RED-PROOF (observed): remove the `!cov.CanRestore()` guard from RestoreTier2Files →
//
// tier2_coverage_test.go:63: RestoreTier2Files returned <nil> — a copy with nothing restorable was treated as success
// tier2_coverage_test.go:69: THE APP WAS STOPPED for a restore that could not restore anything: [app]
func TestRestoreTier2Files_NoRestorableSubtree_RefusesBeforeStopping(t *testing.T) {
m, fake, _, destDrive := newT2RManager(t)
unitOnlyCopy(t, destDrive)
m.restoreFilesCopier = func(string, string) (int, error) {
t.Fatal("the copier ran for an app with no restorable subtree")
return 0, nil
}
n, err := m.RestoreTier2Files("app")
if !errors.Is(err, ErrTier2NoRestorableData) {
t.Errorf("RestoreTier2Files returned %v — a copy with nothing restorable was treated as success", err)
}
if n != 0 {
t.Errorf("filesRestored = %d, want 0", n)
}
// The whole point: no outage was taken.
if len(fake.stopped) != 0 {
t.Errorf("THE APP WAS STOPPED for a restore that could not restore anything: %v", fake.stopped)
}
if len(fake.started) != 0 {
t.Errorf("the app was restarted, so it must have been stopped: %v", fake.started)
}
}
// The coverage query itself — what the handler pre-flights on, so it can refuse without starting an
// operation at all.
func TestTier2RestoreCoverage_ReportsTheAsymmetry(t *testing.T) {
m, _, _, destDrive := newT2RManager(t)
// Class A (paperless/immich shape): an hdd leg the restore reads, plus a unit it does not.
mustWrite(t, filepath.Join(destDrive, "backups", "secondary", "app", "recovery-unit", "manifest.json"), "{}")
cov, err := m.Tier2RestoreCoverage("app")
if err != nil {
t.Fatalf("coverage: %v", err)
}
if !cov.CanRestore() {
t.Error("an app WITH an hdd leg reported as not restorable — this would refuse the one path that works")
}
if !cov.HasUnit {
t.Error("the recovery unit was not detected — the disclosure would be omitted")
}
// Class B (bookstack/docmost shape): unit only.
unitOnlyCopy(t, destDrive)
cov, err = m.Tier2RestoreCoverage("app")
if err != nil {
t.Fatalf("coverage: %v", err)
}
if cov.CanRestore() {
t.Error("a unit-only copy reported as restorable — this is exactly C9-F1")
}
if !cov.HasUnit {
t.Error("the unit that holds the app's whole dataset was not detected")
}
if len(cov.Legs) != 0 {
t.Errorf("legs = %v, want none", cov.Legs)
}
}
// SCENARIO E — an app the restore CAN cover is completely unchanged. This is the regression guard on
// Campaign 9's headline result (A1/A3, paperless-ngx): byte-identical restore, stop→copy→start, and
// the additive-only promises intact. Breaking this to fix BookStack would be a straight regression on
// the only restore path proven to work on live hardware.
//
// RED-PROOF (observed): make the coverage guard unconditional (`if true`) →
//
// tier2_coverage_test.go:118: a COVERED app was refused: ennek az alkalmazásnak az adatai nem ebből a másolatból állíthatók vissza
func TestRestoreTier2Files_CoveredAppIsUnchanged(t *testing.T) {
m, fake, liveDrive, destDrive := newT2RManager(t)
// A unit is present too — a covered app has one as well; it must not change the outcome.
mustWrite(t, filepath.Join(destDrive, "backups", "secondary", "app", "recovery-unit", "manifest.json"), "{}")
var copied [][2]string
m.restoreFilesCopier = func(src, dst string) (int, error) {
copied = append(copied, [2]string{src, dst})
fake.order = append(fake.order, "copy")
return 3, nil
}
n, err := m.RestoreTier2Files("app")
if err != nil {
t.Fatalf("a COVERED app was refused: %v", err)
}
if n != 3 {
t.Errorf("filesRestored = %d, want 3", n)
}
if len(fake.stopped) != 1 || len(fake.started) != 1 {
t.Errorf("stop/start did not happen exactly once: %v / %v", fake.stopped, fake.started)
}
if got := fake.order; len(got) < 3 || got[0] != "stop" || got[len(got)-1] != "start" {
t.Errorf("order = %v, want stop → copy → start", got)
}
if len(copied) == 0 {
t.Fatal("nothing was copied for a covered app")
}
wantSrc := filepath.Join(destDrive, "backups", "secondary", "app", "hdd")
if copied[0][0] != wantSrc {
t.Errorf("src = %q, want %q", copied[0][0], wantSrc)
}
if copied[0][1] != liveDrive {
t.Errorf("dst = %q, want the live namespace root %q", copied[0][1], liveDrive)
}
}
+87 -18
View File
@@ -33,8 +33,81 @@ var (
// marker). Refuse rather than read a flat layout we no longer understand — safe, because tier-2 // marker). Refuse rather than read a flat layout we no longer understand — safe, because tier-2
// restore is missing-file recovery and the live data still exists in that scenario. // restore is missing-file recovery and the live data still exists in that scenario.
errTier2OldLayout = errors.New("A 2. mentés régi formátumú — futtass előbb egy új másodlagos mentést.") errTier2OldLayout = errors.New("A 2. mentés régi formátumú — futtass előbb egy új másodlagos mentést.")
// ErrTier2NoRestorableData (C9-F1) — this app HAS a Tier-2 copy, but that copy contains no subtree
// this restore can read: its data lives entirely in Docker named volumes, which are captured into
// recovery-unit/ (db-dumps + volume-dumps) and NEVER read by this path. 43 of the 53 catalog apps
// are in this class. Exported so the handler can refuse BEFORE stopping the app and name the action
// that does work, instead of taking an outage and reporting "no missing files".
ErrTier2NoRestorableData = errors.New("ennek az alkalmazásnak az adatai nem ebből a másolatból állíthatók vissza")
) )
// Tier2Coverage says what a Tier-2 restore can and cannot return for one app — the asymmetry C9-F1
// is about. Computed from the RECORDED copy on disk, never guessed from the catalog, so an app whose
// template changed is judged by what its actual copy holds.
//
// The distinction that matters: Legs are the subtrees RestoreTier2Files reads (hdd/, userdata/);
// HasUnit means the copy ALSO holds a full recovery unit — the app's database dumps and named-volume
// tarballs — which this restore path never opens. An app can have HasUnit && no Legs (43 of 53), in
// which case the restore is a guaranteed no-op no matter how much data was lost.
type Tier2Coverage struct {
Legs []string // subtrees this restore reads and that exist in the copy: "hdd", "userdata"
HasUnit bool // recovery-unit/ present — captured, but NOT restorable by this path
}
// CanRestore reports whether the restore has any subtree to read at all.
func (c Tier2Coverage) CanRestore() bool { return len(c.Legs) > 0 }
// tier2CoverageAt inspects a resolved copy directory. Pure filesystem stat — no side effects.
func tier2CoverageAt(destBase string) Tier2Coverage {
var c Tier2Coverage
for _, leg := range []string{"hdd", "userdata"} {
if fi, err := os.Stat(filepath.Join(destBase, leg)); err == nil && fi.IsDir() {
c.Legs = append(c.Legs, leg)
}
}
if fi, err := os.Stat(filepath.Join(destBase, "recovery-unit")); err == nil && fi.IsDir() {
c.HasUnit = true
}
return c
}
// Tier2RestoreCoverage resolves the app's RECORDED Tier-2 copy and reports what a restore could
// return from it. Errors are the same refusals RestoreTier2Files itself would raise, so the caller
// can surface them before starting anything — this is what lets the handler refuse without an outage.
func (m *Manager) Tier2RestoreCoverage(stackName string) (Tier2Coverage, error) {
destBase, err := m.tier2RecordedCopyDir(stackName)
if err != nil {
return Tier2Coverage{}, err
}
return tier2CoverageAt(destBase), nil
}
// tier2RecordedCopyDir resolves the RECORDED Tier-2 copy dir for a stack, applying every
// source-side refusal in one place so the pre-flight check and the restore itself cannot drift.
func (m *Manager) tier2RecordedCopyDir(stackName string) (string, error) {
var destBase string
if m.settings != nil {
if cfg := m.settings.GetCrossDriveConfig(stackName); cfg != nil && cfg.LastRun != "" && cfg.DestinationPath != "" {
if m.settings.IsDisconnected(cfg.DestinationPath) {
return "", errTier2DriveGone
}
destBase = filepath.Join(cfg.DestinationPath, "backups", "secondary", stackName)
}
}
if destBase == "" {
return "", errNoTier2Copy
}
if _, statErr := os.Stat(destBase); statErr != nil {
return "", errNoTier2Copy // recorded but the copy dir is gone — same honest refusal
}
// §7-G2 marker gate: a pre-v2 (flat) copy has no marker → refuse rather than read a layout we no
// longer understand (live data still exists for missing-file recovery).
if _, mErr := os.Stat(filepath.Join(destBase, tier2LayoutMarker)); mErr != nil {
return "", errTier2OldLayout
}
return destBase, nil
}
// RestoreTier2Files restores the app's MISSING user files in place from its recorded Tier-2 copy // RestoreTier2Files restores the app's MISSING user files in place from its recorded Tier-2 copy
// (additive-only; see the package comment above). Returns how many regular files were copied back. // (additive-only; see the package comment above). Returns how many regular files were copied back.
// //
@@ -70,25 +143,21 @@ func (m *Manager) RestoreTier2Files(stackName string) (filesRestored int, err er
liveNsRoot := m.namespaceRoot(drive) liveNsRoot := m.namespaceRoot(drive)
// Source side: the RECORDED Tier-2 copy must exist, its drive connected, and it must be v2. // Source side: the RECORDED Tier-2 copy must exist, its drive connected, and it must be v2.
var destBase string destBase, err := m.tier2RecordedCopyDir(stackName)
if m.settings != nil { if err != nil {
if cfg := m.settings.GetCrossDriveConfig(stackName); cfg != nil && cfg.LastRun != "" && cfg.DestinationPath != "" { return 0, err
if m.settings.IsDisconnected(cfg.DestinationPath) {
return 0, errTier2DriveGone
} }
destBase = filepath.Join(cfg.DestinationPath, "backups", "secondary", stackName)
} // C9-F1: refuse BEFORE the app is stopped if this copy holds nothing this path can read. Without
} // this the app was stopped, zero files were copied, it was restarted, and the customer was told
if destBase == "" { // "Nincs hiányzó fájl — minden fájl megvan a helyén." — an outage plus a claim about data the
return 0, errNoTier2Copy // restore never looked at. Placed with the other source-side refusals, all of which precede the
} // stop, so the promise "all refusals happen BEFORE the app is stopped" stays true.
if _, statErr := os.Stat(destBase); statErr != nil { cov := tier2CoverageAt(destBase)
return 0, errNoTier2Copy // recorded but the copy dir is gone — same honest refusal if !cov.CanRestore() {
} m.logger.Printf("[WARN] [backup] Tier-2 file restore refused for %s: the recorded copy has no restorable subtree (unit_present=%v) — the app was NOT stopped",
// §7-G2 marker gate: a pre-v2 (flat) copy has no marker → refuse rather than read a layout we no stackName, cov.HasUnit)
// longer understand (live data still exists for missing-file recovery). return 0, ErrTier2NoRestorableData
if _, mErr := os.Stat(filepath.Join(destBase, tier2LayoutMarker)); mErr != nil {
return 0, errTier2OldLayout
} }
copier := m.restoreFilesCopier copier := m.restoreFilesCopier
@@ -27,6 +27,7 @@ func (f *t2rFakeProvider) GetStackComposePath(string) (string, bool) { return ""
func (f *t2rFakeProvider) ListDeployedStacks() []StackSummary { return nil } func (f *t2rFakeProvider) ListDeployedStacks() []StackSummary { return nil }
func (f *t2rFakeProvider) GetStackHDDMounts(string) []string { return f.mounts } func (f *t2rFakeProvider) GetStackHDDMounts(string) []string { return f.mounts }
func (f *t2rFakeProvider) GetStackHDDPath(string) string { return f.hdd } func (f *t2rFakeProvider) GetStackHDDPath(string) string { return f.hdd }
func (f *t2rFakeProvider) GetImportRoot() string { return "" } // R-75: no import binds in this fixture
func (f *t2rFakeProvider) GetDockerVolumes(string) []string { return nil } func (f *t2rFakeProvider) GetDockerVolumes(string) []string { return nil }
func (f *t2rFakeProvider) StopStack(name string) error { func (f *t2rFakeProvider) StopStack(name string) error {
f.stopped = append(f.stopped, name) f.stopped = append(f.stopped, name)
@@ -25,6 +25,7 @@ func (p *t2v2Provider) GetStackComposePath(string) (string, bool) { return "", f
func (p *t2v2Provider) ListDeployedStacks() []StackSummary { return nil } func (p *t2v2Provider) ListDeployedStacks() []StackSummary { return nil }
func (p *t2v2Provider) GetStackHDDMounts(n string) []string { return p.mounts[n] } func (p *t2v2Provider) GetStackHDDMounts(n string) []string { return p.mounts[n] }
func (p *t2v2Provider) GetStackHDDPath(string) string { return p.hdd } func (p *t2v2Provider) GetStackHDDPath(string) string { return p.hdd }
func (p *t2v2Provider) GetImportRoot() string { return "" } // R-75: no import binds in this fixture
func (p *t2v2Provider) GetDockerVolumes(string) []string { return nil } func (p *t2v2Provider) GetDockerVolumes(string) []string { return nil }
func (p *t2v2Provider) StopStack(string) error { return nil } func (p *t2v2Provider) StopStack(string) error { return nil }
func (p *t2v2Provider) StartStack(string) error { return nil } func (p *t2v2Provider) StartStack(string) error { return nil }
@@ -25,6 +25,7 @@ func (f *volDumpFakeProvider) GetStackComposePath(string) (string, bool) { retur
func (f *volDumpFakeProvider) ListDeployedStacks() []StackSummary { return f.stacks } func (f *volDumpFakeProvider) ListDeployedStacks() []StackSummary { return f.stacks }
func (f *volDumpFakeProvider) GetStackHDDMounts(string) []string { return nil } func (f *volDumpFakeProvider) GetStackHDDMounts(string) []string { return nil }
func (f *volDumpFakeProvider) GetStackHDDPath(name string) string { return f.hdd[name] } func (f *volDumpFakeProvider) GetStackHDDPath(name string) string { return f.hdd[name] }
func (f *volDumpFakeProvider) GetImportRoot() string { return "" } // R-75: no import binds in this fixture
func (f *volDumpFakeProvider) GetDockerVolumes(name string) []string { return f.volumes[name] } func (f *volDumpFakeProvider) GetDockerVolumes(name string) []string { return f.volumes[name] }
func (f *volDumpFakeProvider) StopStack(name string) error { func (f *volDumpFakeProvider) StopStack(name string) error {
f.stopped = append(f.stopped, name) f.stopped = append(f.stopped, name)
@@ -0,0 +1,83 @@
// Package backupwindow holds the pure time arithmetic for the customer-configurable backup window
// (v0.168.0). ONE setting — the window start W — drives every nightly leg at FIXED offsets so the
// legs can never be misordered, and never stores a derived time: the DB dump runs at W, the tier-2
// mirror at W+60m, the off-box copy at W+105m; the whole-guest (PBS/vzdump) cycle is gated to
// [W+2h, W+6h). Offsets are constants here, never persisted and never surfaced in the UI.
package backupwindow
import "fmt"
// DefaultWindow is the last-resort window when neither settings nor controller.yaml supplies one.
// It equals the historical hardcoded DB-dump time, so an un-configured box behaves exactly as before.
const DefaultWindow = "02:30"
// Fixed leg offsets from the window start W (minutes). NEVER stored, NEVER exposed in the UI —
// changing spacing/ordering is a code change here, not customer data.
const (
tier2OffsetMin = 60 // tier-2 mirror at W+60m
offboxOffsetMin = 105 // off-box copy at W+105m
gateStartMin = 120 // whole-guest gate opens at W+2h
gateEndMin = 360 // whole-guest gate closes (exclusive) at W+6h
)
// ParseHHMM parses "HH:MM" (24h) into minutes-since-midnight. It rejects anything but a valid
// hour:minute — the same contract as the scheduler's parseDailyTime, kept here so this package is
// dependency-free and reusable by the quiesce gate.
func ParseHHMM(s string) (int, error) {
var h, m int
n, err := fmt.Sscanf(s, "%d:%d", &h, &m)
if err != nil || n != 2 {
return 0, fmt.Errorf("expected HH:MM format, got %q", s)
}
if h < 0 || h > 23 || m < 0 || m > 59 {
return 0, fmt.Errorf("invalid time %q: hour must be 0-23, minute 0-59", s)
}
return h*60 + m, nil
}
// FmtHHMM renders minutes-since-midnight back to "HH:MM", wrapping across midnight (modulo 24h) so
// derived legs past 23:59 read correctly (e.g. 23:30 + 60m → 00:30).
func FmtHHMM(minutes int) string {
minutes = ((minutes % 1440) + 1440) % 1440
return fmt.Sprintf("%02d:%02d", minutes/60, minutes%60)
}
// Valid reports whether s is a well-formed HH:MM window value (nil error = valid).
func Valid(s string) error {
_, err := ParseHHMM(s)
return err
}
// LegTimes returns the three derived nightly-leg times (db=W, tier2=W+60m, offbox=W+105m),
// wrap-safe across midnight. On an invalid start it returns three empty strings — callers pass a
// value already resolved through EffectiveWindow, which never yields an invalid string.
func LegTimes(start string) (db, tier2, offbox string) {
m, err := ParseHHMM(start)
if err != nil {
return "", "", ""
}
return FmtHHMM(m), FmtHHMM(m + tier2OffsetMin), FmtHHMM(m + offboxOffsetMin)
}
// GateWindow returns the whole-guest backup gate bounds [W+2h, W+6h) as HH:MM strings (for the UI
// "kb. <from><to> között" line and the gate-denial log). Empty strings on an invalid start.
func GateWindow(start string) (from, to string) {
m, err := ParseHHMM(start)
if err != nil {
return "", ""
}
return FmtHHMM(m + gateStartMin), FmtHHMM(m + gateEndMin)
}
// EffectiveWindow resolves the active window by precedence: a valid settings value wins over a valid
// controller.yaml value, which wins over DefaultWindow. An empty or corrupted value simply falls
// through — so a bad settings string degrades to the yaml default rather than breaking scheduling.
func EffectiveWindow(settingsVal, yamlVal string) string {
if Valid(settingsVal) == nil {
return settingsVal
}
if Valid(yamlVal) == nil {
return yamlVal
}
return DefaultWindow
}
@@ -0,0 +1,69 @@
package backupwindow
import "testing"
// Group A — LegTimes derives the three nightly legs at fixed offsets, wrap-safe across midnight.
// Red-proof: drop the modulo in FmtHHMM → the 23:30 case yields "24:30"/"25:15" and fails.
func TestLegTimes(t *testing.T) {
cases := []struct{ start, db, tier2, offbox string }{
{"02:30", "02:30", "03:30", "04:15"}, // the default window
{"23:30", "23:30", "00:30", "01:15"}, // wraps past midnight
{"22:00", "22:00", "23:00", "23:45"},
{"00:00", "00:00", "01:00", "01:45"},
{"2:30", "02:30", "03:30", "04:15"}, // normalizes a missing leading zero
}
for _, c := range cases {
db, tier2, offbox := LegTimes(c.start)
if db != c.db || tier2 != c.tier2 || offbox != c.offbox {
t.Errorf("LegTimes(%q) = (%q,%q,%q), want (%q,%q,%q)", c.start, db, tier2, offbox, c.db, c.tier2, c.offbox)
}
}
}
// Group A — invalid input is rejected (LegTimes → empty; Valid → error). Callers pass a value
// already resolved through EffectiveWindow, so an empty result is never rendered.
func TestLegTimes_InvalidRejected(t *testing.T) {
for _, bad := range []string{"25:77", "2200", "", "ab:cd", "24:00", "12:60", "-1:00"} {
db, tier2, offbox := LegTimes(bad)
if db != "" || tier2 != "" || offbox != "" {
t.Errorf("LegTimes(%q) = (%q,%q,%q), want all empty (rejected)", bad, db, tier2, offbox)
}
if Valid(bad) == nil {
t.Errorf("Valid(%q) = nil, want an error", bad)
}
}
if Valid("02:30") != nil {
t.Error("Valid(02:30) returned an error for a well-formed time")
}
}
// Group A — the whole-guest gate span is [W+2h, W+6h), wrap-safe.
func TestGateWindow(t *testing.T) {
if from, to := GateWindow("02:30"); from != "04:30" || to != "08:30" {
t.Errorf("GateWindow(02:30) = (%q,%q), want (04:30,08:30)", from, to)
}
if from, to := GateWindow("23:00"); from != "01:00" || to != "05:00" {
t.Errorf("GateWindow(23:00) = (%q,%q), want (01:00,05:00) — must wrap", from, to)
}
if from, to := GateWindow("bad"); from != "" || to != "" {
t.Errorf("GateWindow(bad) = (%q,%q), want empties", from, to)
}
}
// Group B — precedence: a valid settings value wins over a valid yaml value, which wins over the
// "02:30" default; an empty/corrupt settings value falls through the chain.
func TestEffectiveWindow(t *testing.T) {
cases := []struct{ settingsVal, yamlVal, want string }{
{"22:00", "02:30", "22:00"}, // settings wins over yaml
{"", "03:00", "03:00"}, // yaml when settings empty
{"", "", "02:30"}, // default when both empty
{"garbage", "03:00", "03:00"}, // corrupt settings → fall through to yaml
{"garbage", "nope", "02:30"}, // both invalid → default
{"22:00", "", "22:00"}, // settings valid, yaml empty
}
for _, c := range cases {
if got := EffectiveWindow(c.settingsVal, c.yamlVal); got != c.want {
t.Errorf("EffectiveWindow(%q,%q) = %q, want %q", c.settingsVal, c.yamlVal, got, c.want)
}
}
}
+146 -12
View File
@@ -11,10 +11,15 @@
// - **Bounded, never a loop.** At most `attempts` tries, `retryDelay` apart, then it stops and the // - **Bounded, never a loop.** At most `attempts` tries, `retryDelay` apart, then it stops and the
// alarm owns the problem. A restart loop would paper over a genuinely broken app forever and // alarm owns the problem. A restart loop would paper over a genuinely broken app forever and
// hammer docker while doing it. // hammer docker while doing it.
// - **A user's Stop survives a reboot.** The UI's Stop is `docker compose down`, which REMOVES the // - **A user's Stop survives a reboot.** This is still the whole safety argument; only the way it
// containers; a boot interruption leaves them behind as Exited. So "has containers on disk that // is established changed. Until v0.189.0 it was inferred — the UI's Stop is `docker compose
// are down" is the boot-orphan signature, and a stack with ZERO containers is deliberately never // down`, which REMOVES containers, so "zero containers" was read as "the customer stopped it"
// touched. This distinction is the whole safety argument — see TestReconcile_UserStoppedAppIsNeverStarted. // and left alone. Since v0.189.0 (R-166) the customer's intent is RECORDED in app.yaml and read
// directly, because the inference could not distinguish a deliberate Stop from a power cut or an
// interrupted backup, and silently stranded both. An app.yaml with no recorded intent — every
// app on every box predating the field — keeps the old inference exactly. See isBootOrphan,
// TestReconcile_UserStoppedAppIsNeverStarted and
// TestReconcile_LegacyNoDesiredState_BehavesExactlyAsBefore.
// //
// It runs inside the notifier's boot grace (cmd/controller/main.go `deadAppBootGrace`), so a // It runs inside the notifier's boot grace (cmd/controller/main.go `deadAppBootGrace`), so a
// successful recovery never fires an alert and a failed one alerts honestly once the grace expires. // successful recovery never fires an alert and a failed one alerts honestly once the grace expires.
@@ -36,6 +41,45 @@ type StackProvider interface {
RefreshStatus() error RefreshStatus() error
} }
// StartGate answers the one question this package must ask before starting anything: **may this app
// be started right now?** Declared consumer-side, in the style of StackProvider, so `bootrecon`
// still imports `stacks` alone and knows nothing about settings, the agent, quiesce or the web layer.
//
// It is ONE seam rather than three because the three reasons a boot orphan must NOT be started share
// a shape — something else is deliberately holding this app — and differ only in the reason string:
//
// the drive-absent gate stopped it → its drive is not live (R-171, below)
// a quiesce is holding it for a backup → the quiesce loop restarts its own stacks
// an app-data operation stopped it → the app-stop guard's own Recover owns it
//
// A widened boot window (R-157 mechanism A) is what makes the last two reachable at all: the old
// T+5 s single sweep never overlapped them.
//
// R-171 — WHY THIS EXISTS, and it is a regression this package caused. Until v0.189.0 the sweep
// required a stack to still HAVE containers, and an app the drive-absent gate had stopped has zero,
// so such apps were skipped by accident. v0.189.0 replaced that term with the customer's recorded
// intent — correctly — and the drive gate does NOT change `desired_state` (it is not the customer),
// so a gate-stopped app now reads as `running` + zero containers, i.e. a boot orphan. Observed live
// on 2026-08-02: the sweep found and started an app whose drive was unmounted, burned both attempts,
// and handed it to the dead-app alarm — a false alarm about an app the drive gate is deliberately
// holding (audits/DIAG-bootrecon-drive-absent-2026-08-02.md).
//
// The rule itself is not new and is not invented here: the API's own start path already refuses this
// (`startGatedByMissingDrive`, internal/api/router.go) with a Hungarian message to the customer. The
// sweep simply bypassed it by calling Manager.StartStack directly. This seam gives the sweep the
// same question to ask.
//
// CONTRACT — the answer is fail-safe by design (§8.4): an implementation that CANNOT DETERMINE
// whether the drive is live must return false, not true. Not starting is recoverable — the drive
// gate's `Return` branch restarts the app when the drive comes back, and the dead-app alarm reports
// it meanwhile. Starting on an absent drive is not recoverable by anything automatic: compose
// creates the bind sources wherever the mountpoint currently points, which is the guest rootfs.
type StartGate interface {
// MayStart reports whether the named stack may be started. The reason is for the log line and is
// only read when may is false.
MayStart(stackName string) (may bool, reason string)
}
const ( const (
// DefaultAttempts is the total number of start attempts per boot (not per app per retry-forever). // DefaultAttempts is the total number of start attempts per boot (not per app per retry-forever).
DefaultAttempts = 2 DefaultAttempts = 2
@@ -53,6 +97,13 @@ type Reconciler struct {
// sleep is the inter-attempt wait; injectable so tests never spend 30 real seconds. // sleep is the inter-attempt wait; injectable so tests never spend 30 real seconds.
sleep func(context.Context, time.Duration) sleep func(context.Context, time.Duration)
// startGate (R-171) refuses to start an app something else is deliberately holding. nil = NOT
// WIRED, which means "this caller has no such concept" and is permissive — the test fixtures'
// case. It is NOT the same as "cannot determine", which the gate itself answers with false (see
// StartGate's contract). Production MUST wire it; TestMainWiresBootDriveGate walks main.go's AST
// for the call, because an unwired seam here is silently the pre-v0.190.0 behaviour.
startGate StartGate
} }
// Result is the outcome, returned for logging/testing (the hub learns about failures only through // Result is the outcome, returned for logging/testing (the hub learns about failures only through
@@ -62,6 +113,23 @@ type Result struct {
Recovered []string // running again by the end Recovered []string // running again by the end
StillDown []string // still down after the last attempt — the alarm's problem now StillDown []string // still down after the last attempt — the alarm's problem now
Attempts int // attempts actually made (0 when there was nothing to do) Attempts int // attempts actually made (0 when there was nothing to do)
// HeldByDrive (R-171) are apps that ARE boot orphans by intent but which something else is
// deliberately holding (an absent drive, a quiesce, an app-data operation), so they were not
// started. Reported separately from StillDown because they are not a fault this sweep failed to
// fix — the holder owns their recovery. Collapsing the two would put a deliberately-held app in
// the same bucket as a broken one, which is the false alarm R-171 removes.
HeldByDrive []string
}
// SetDriveGate wires the R-171 start refusal. INIT-ONLY — call once, before Run.
func (r *Reconciler) SetDriveGate(g StartGate) { r.startGate = g }
// mayStart asks the gate, or allows when none is wired (see the startGate field comment).
func (r *Reconciler) mayStart(stackName string) (bool, string) {
if r.startGate == nil {
return true, ""
}
return r.startGate.MayStart(stackName)
} }
// New builds a Reconciler with the shipped defaults. // New builds a Reconciler with the shipped defaults.
@@ -87,17 +155,66 @@ func sleepCtx(ctx context.Context, d time.Duration) {
// isBootOrphan reports whether a stack is an app the boot left behind. // isBootOrphan reports whether a stack is an app the boot left behind.
// //
// The gate, term by term: // The gate, term by term:
// - Deployed — an app the customer asked to have running. // - Deployed — an app that is installed. NOTE: `Deployed` means INSTALLED, not "wanted running";
// the two were conflated until v0.189.0 and that conflation is what the desired-state term below
// repairs.
// - not Protected — traefik/cloudflared/felhom-controller have their own supervision; this must // - not Protected — traefik/cloudflared/felhom-controller have their own supervision; this must
// never race the base-stack self-heal. // never race the base-stack self-heal.
// - not Deploying — mid-deploy is not a fault. // - not Deploying — mid-deploy is not a fault.
// - has containers — the D-case guard: a UI Stop removes them, and a deliberate stop must survive // - desired state — see below. REPLACES the old container-count term.
// a reboot.
// - IsDownState — stopped/exited/degraded (R-51 included: a boot that half-started a stack is the // - IsDownState — stopped/exited/degraded (R-51 included: a boot that half-started a stack is the
// same interrupted-boot shape). // same interrupted-boot shape).
//
// ── WHY INTENT REPLACED THE CONTAINER COUNT (R-166, closing R-157 mechanism B) ────────────────────
//
// This gate used to end in `len(s.Containers) > 0`, and its comment called that "the D-case guard":
// a UI Stop is `compose down`, which REMOVES containers, so zero containers was read as "the
// customer stopped this" and left alone. The safety goal was right and still holds. The SIGNAL was
// wrong, because zero containers has at least three causes and the count cannot tell them apart:
//
// a deliberate Stop → must stay down
// a power cut mid-compose, or an interrupted deploy → must come back
// a backup that stopped the app and died before restarting it → must come back
//
// Two of those three were silently unrecoverable: the app simply stayed gone until a human noticed.
// The count was never capable of separating them, so the fix is not a better inference — it is to
// stop inferring and read what the customer actually asked for, which app.yaml now records.
//
// ── WHAT ABSENT STILL MEANS, AND WHY THE OLD BEHAVIOUR IS KEPT ────────────────────────────────────
//
// DesiredStateUnknown falls back to the ORIGINAL container-count rule, byte-for-byte. This is the
// single most important line in the change. Every app.yaml on every existing box predates the field,
// so absent is what the whole fleet reads on upgrade; treating absent as "running" would start, on
// the first boot after the upgrade, every app its owner had deliberately stopped. The fallback is
// what makes this feature inert for an app nobody has pressed a button on since — see
// TestReconcile_LegacyNoDesiredState_BehavesExactlyAsBefore and its red-proof.
//
// The full decision table (§8.1):
//
// desired containers state → result
// stopped any any → never an orphan (the customer said so)
// running 0 — → ORPHAN ← the R-157 case, invisible before v0.189.0
// running >0 IsDownState → ORPHAN (unchanged)
// running >0 up → not an orphan
// absent 0 — → not an orphan (exactly the pre-v0.189.0 behaviour)
// absent >0 IsDownState → ORPHAN (exactly the pre-v0.189.0 behaviour)
func isBootOrphan(s stacks.Stack) bool { func isBootOrphan(s stacks.Stack) bool {
return s.Deployed && !s.Protected && !s.Deploying && if !s.Deployed || s.Protected || s.Deploying {
len(s.Containers) > 0 && stacks.IsDownState(s.State) return false
}
switch stacks.DesiredStateOf(s) {
case stacks.DesiredStateStopped:
// The customer pressed Stop. No observation may overturn that — not a missing container, not
// a down state, not a reboot. Nothing else in this package starts an app.
return false
case stacks.DesiredStateRunning:
// Wanted running. ANY way of not being up is a fault to repair, including having no
// containers at all — which is the case the old count term structurally could not see.
return len(s.Containers) == 0 || stacks.IsDownState(s.State)
default:
// DesiredStateUnknown — legacy. Keep the pre-R-166 rule exactly.
return len(s.Containers) > 0 && stacks.IsDownState(s.State)
}
} }
// Run performs the sweep once and returns what happened. It is safe to call with no boot orphans // Run performs the sweep once and returns what happened. It is safe to call with no boot orphans
@@ -108,16 +225,33 @@ func (r *Reconciler) Run(ctx context.Context) Result {
pending := map[string]bool{} pending := map[string]bool{}
for _, s := range r.stacks.GetStacks() { for _, s := range r.stacks.GetStacks() {
if isBootOrphan(s) { if !isBootOrphan(s) {
continue
}
// R-171: intent says this app should be running and it is not — but if something else is
// deliberately holding it (absent drive, quiesce, an app-data operation), starting it is the
// wrong repair. Refuse, loudly, and let the holder own it.
if live, reason := r.mayStart(s.Name); !live {
res.HeldByDrive = append(res.HeldByDrive, s.Name)
r.logger.Printf("[INFO] [bootrecon] %q is a boot orphan by intent but is HELD (%s) — NOT starting it; whatever is holding it owns its recovery",
s.Name, reason)
continue
}
pending[s.Name] = true pending[s.Name] = true
res.Candidates = append(res.Candidates, s.Name) res.Candidates = append(res.Candidates, s.Name)
} }
}
sortStrings(res.Candidates) sortStrings(res.Candidates)
sortStrings(res.HeldByDrive)
if len(pending) == 0 { if len(pending) == 0 {
// The healthy path must be observable — "no alarms" and "never ran" have to be // The healthy path must be observable — "no alarms" and "never ran" have to be
// distinguishable in a log (the v0.91.2 lesson). // distinguishable in a log (the v0.91.2 lesson). "Nothing to start" and "everything I found
// is held by an absent drive" must be distinguishable too, or the held case reads as healthy.
if len(res.HeldByDrive) > 0 {
r.logger.Printf("[INFO] [bootrecon] Boot reconciliation: nothing to start — %d app(s) held (absent drive / quiesce / app-data operation): %v",
len(res.HeldByDrive), res.HeldByDrive)
return res
}
r.logger.Printf("[INFO] [bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start)") r.logger.Printf("[INFO] [bootrecon] Boot reconciliation: no boot-orphaned apps (nothing to start)")
return res return res
} }
@@ -0,0 +1,236 @@
package bootrecon
import (
"context"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
)
// R-166 / decision D-b: the boot reconciler reads the CUSTOMER'S RECORDED INTENT instead of
// inferring it from a container count. These tests are the §8.1 decision table, one row each, plus
// the two red-proofs that make the safety properties falsifiable.
//
// The assertions are EFFECTS — which apps the sweep actually started — not "isBootOrphan returned
// true". A predicate can be right while the sweep does nothing with it.
// withDesired returns a copy of s carrying a recorded desired state.
func withDesired(s stacks.Stack, desired string) stacks.Stack {
s.AppConfig = &stacks.AppConfig{Deployed: true, DesiredState: desired}
return s
}
// vanished is the R-157 shape this whole change exists to see: the app is deployed and wanted
// running, and its containers are simply GONE — a power cut mid-compose, or an interrupted deploy.
// Byte-identical on the Docker side to a user stop, which is exactly why the old container-count
// rule could not tell them apart.
func vanished(name string) stacks.Stack {
return stacks.Stack{Name: name, Deployed: true, State: stacks.StateStopped, Containers: nil}
}
// runSweep runs one full reconciliation and returns which apps were started, and how often.
func runSweep(t *testing.T, list []stacks.Stack) (*fakeStacks, Result) {
t.Helper()
f := &fakeStacks{list: list, onStart: comesUp}
r, _ := newTestReconciler(f)
res := r.Run(context.Background())
return f, res
}
// --- Scenario A — the customer's Stop survives everything ---------------------------------------
func TestReconcile_DesiredStopped_IsNeverStartedAndNeverACandidate(t *testing.T) {
// Recorded stopped, and down in every way the box can be down: no containers at all, and (second
// app) containers present but exited. Neither may be touched, and neither may even be LISTED —
// a candidate that is never started still tells the operator an app is broken when it is not.
f, res := runSweep(t, []stacks.Stack{
withDesired(vanished("nextcloud"), stacks.DesiredStateStopped),
withDesired(bootOrphan("immich"), stacks.DesiredStateStopped),
})
if len(f.starts) != 0 {
t.Fatalf("an app the customer deliberately stopped was started: %v", f.starts)
}
if len(res.Candidates) != 0 {
t.Fatalf("desired=stopped app listed as a boot orphan: %v", res.Candidates)
}
if res.Attempts != 0 {
t.Fatalf("attempts=%d, want 0 — the sweep should have had nothing to do", res.Attempts)
}
}
// --- Scenario B — the power-cut app comes back (THE R-157 CASE) ---------------------------------
func TestReconcile_DesiredRunning_ZeroContainers_IsRecovered(t *testing.T) {
// THE POINT OF THE RELEASE. Before v0.189.0 this app was invisible to the reconciler: zero
// containers failed the `len(s.Containers) > 0` term, so it was skipped as "the customer stopped
// it" and stayed down until a human noticed.
//
// RED-PROOF: restore that term in isBootOrphan's DesiredStateRunning branch — i.e. make it
// return len(s.Containers) > 0 && stacks.IsDownState(s.State)
// and this test fails with `zero starts`. Demonstrated in REPORT.md §5.
f, res := runSweep(t, []stacks.Stack{withDesired(vanished("immich"), stacks.DesiredStateRunning)})
if f.starts["immich"] == 0 {
t.Fatalf("an app recorded desired=running with zero containers was NOT started — this is the R-157 defect")
}
if len(res.Recovered) != 1 || res.Recovered[0] != "immich" {
t.Fatalf("recovered=%v, want [immich]", res.Recovered)
}
if len(res.StillDown) != 0 {
t.Fatalf("still down after a successful start: %v", res.StillDown)
}
}
func TestReconcile_DesiredRunning_ContainersDown_IsRecovered(t *testing.T) {
// The pre-existing F5 shape, unchanged by R-166 — proven still covered so the rewrite cannot
// have traded one case for the other.
f, _ := runSweep(t, []stacks.Stack{withDesired(bootOrphan("calibre-web"), stacks.DesiredStateRunning)})
if f.starts["calibre-web"] == 0 {
t.Fatal("an app recorded desired=running with exited containers was not started")
}
}
func TestReconcile_DesiredRunning_AlreadyUp_IsLeftAlone(t *testing.T) {
up := withDesired(stacks.Stack{
Name: "vaultwarden", Deployed: true, State: stacks.StateRunning,
Containers: []stacks.ContainerInfo{{Name: "vw", State: stacks.StateRunning}},
}, stacks.DesiredStateRunning)
f, res := runSweep(t, []stacks.Stack{up})
if len(f.starts) != 0 {
t.Fatalf("a running app was restarted: %v", f.starts)
}
if len(res.Candidates) != 0 {
t.Fatalf("a running app was listed as a boot orphan: %v", res.Candidates)
}
}
// --- Scenario C — a legacy app.yaml behaves EXACTLY as it does today -----------------------------
func TestReconcile_LegacyNoDesiredState_BehavesExactlyAsBefore(t *testing.T) {
// THE MOST DANGEROUS MISTAKE AVAILABLE IN THIS CHANGE. Every app.yaml on every existing box was
// written before desired_state existed, so `absent` is what the whole fleet reads on upgrade.
// Treating absent as "running" would start, on the first boot after the upgrade, every app its
// owner had deliberately stopped — silently, fleet-wide.
//
// Both legacy rows of §8.1 asserted together, because the safety property is the PAIR: absent +
// zero containers must be skipped, and absent + down containers must still be recovered. A
// change that broke only one of them would look correct from the other.
//
// RED-PROOF: make the `default:` branch of isBootOrphan return
// len(s.Containers) == 0 || stacks.IsDownState(s.State)
// (i.e. treat absent as running) and this test fails on the "started" assertion.
// Demonstrated in REPORT.md §5.
legacyStopped := vanished("nextcloud") // no AppConfig at all — the true legacy shape
legacyOrphan := bootOrphan("calibre-web") // no AppConfig, containers present and exited
legacyOrphan.AppConfig = nil
legacyStopped.AppConfig = nil
f, res := runSweep(t, []stacks.Stack{legacyStopped, legacyOrphan})
if n := f.starts["nextcloud"]; n != 0 {
t.Fatalf("a LEGACY app with no recorded intent and zero containers was started %d time(s) — "+
"this is the upgrade regression that restarts apps customers deliberately stopped", n)
}
if f.starts["calibre-web"] == 0 {
t.Fatal("a LEGACY boot orphan (containers present, exited) was not recovered — the pre-R-166 behaviour regressed")
}
if len(res.Candidates) != 1 || res.Candidates[0] != "calibre-web" {
t.Fatalf("candidates=%v, want exactly [calibre-web]", res.Candidates)
}
}
func TestReconcile_LegacyAppConfigPresentButFieldAbsent_IsAlsoLegacy(t *testing.T) {
// An app.yaml that EXISTS but predates the field: AppConfig is non-nil, DesiredState is "".
// This is the realistic fleet shape (nil AppConfig only happens with no app.yaml at all), and it
// must take the same legacy path — a nil-vs-empty distinction slipping in here would silently
// split the fleet in two.
s := vanished("immich")
s.AppConfig = &stacks.AppConfig{Deployed: true} // DesiredState is the zero value
f, _ := runSweep(t, []stacks.Stack{s})
if len(f.starts) != 0 {
t.Fatalf("an app.yaml with no desired_state key was treated as running: %v", f.starts)
}
}
// --- The §8.1 table, every row, in one place ----------------------------------------------------
func TestIsBootOrphan_DecisionTable(t *testing.T) {
cases := []struct {
name string
desired string
containers int
state stacks.ContainerState
want bool
}{
{"stopped/no containers", stacks.DesiredStateStopped, 0, stacks.StateStopped, false},
{"stopped/down containers", stacks.DesiredStateStopped, 2, stacks.StateExited, false},
{"stopped/running", stacks.DesiredStateStopped, 2, stacks.StateRunning, false},
{"running/no containers", stacks.DesiredStateRunning, 0, stacks.StateStopped, true},
{"running/down containers", stacks.DesiredStateRunning, 2, stacks.StateExited, true},
{"running/degraded", stacks.DesiredStateRunning, 2, stacks.StateDegraded, true},
{"running/up", stacks.DesiredStateRunning, 2, stacks.StateRunning, false},
{"absent/no containers", stacks.DesiredStateUnknown, 0, stacks.StateStopped, false},
{"absent/down containers", stacks.DesiredStateUnknown, 2, stacks.StateExited, true},
{"absent/up", stacks.DesiredStateUnknown, 2, stacks.StateRunning, false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
s := stacks.Stack{
Name: "app", Deployed: true, State: tc.state,
Containers: make([]stacks.ContainerInfo, tc.containers),
AppConfig: &stacks.AppConfig{Deployed: true, DesiredState: tc.desired},
}
if got := isBootOrphan(s); got != tc.want {
t.Fatalf("isBootOrphan(desired=%q containers=%d state=%s) = %v, want %v",
tc.desired, tc.containers, tc.state, got, tc.want)
}
})
}
}
func TestIsBootOrphan_ExistingGuardsSurviveTheRewrite(t *testing.T) {
// Protected and Deploying were guards before R-166 and must still be, at the strongest desired
// state available — the rewrite reordered the terms, and a reorder is exactly how a guard gets
// dropped without anyone noticing.
base := func() stacks.Stack {
return withDesired(vanished("traefik"), stacks.DesiredStateRunning)
}
protected := base()
protected.Protected = true
if isBootOrphan(protected) {
t.Fatal("a PROTECTED stack became a boot orphan — the base-stack self-heal owns those")
}
deploying := base()
deploying.Deploying = true
if isBootOrphan(deploying) {
t.Fatal("a DEPLOYING stack became a boot orphan — mid-deploy is not a fault")
}
notDeployed := base()
notDeployed.Deployed = false
if isBootOrphan(notDeployed) {
t.Fatal("a stack that is not deployed became a boot orphan")
}
}
// --- Scenario G — the two recoveries do not fight ------------------------------------------------
func TestReconcile_AppAlreadyRestartedByTheMarker_IsNotAlsoAnOrphan(t *testing.T) {
// §8.4's REPORTING requirement. The app-stop marker's Recover runs to completion before this
// sweep is launched, so by the time the reconciler looks, the app it restarted is UP. It must
// therefore not appear as a candidate at all — an app the marker already explained must not also
// be reported as an unexplained boot orphan, or one fault reads as two.
restoredByMarker := withDesired(stacks.Stack{
Name: "immich", Deployed: true, State: stacks.StateRunning,
Containers: []stacks.ContainerInfo{{Name: "immich-server", State: stacks.StateRunning}},
}, stacks.DesiredStateRunning)
f, res := runSweep(t, []stacks.Stack{restoredByMarker})
if len(res.Candidates) != 0 {
t.Fatalf("an app the marker had already restarted was ALSO reported as a boot orphan: %v", res.Candidates)
}
if n := f.starts["immich"]; n != 0 {
t.Fatalf("the app was started a second time (%d) — one fault, one start", n)
}
}
@@ -0,0 +1,217 @@
package bootrecon
import (
"context"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/stacks"
)
// R-171 — the sweep must not start an app whose data drive is absent.
//
// This is a REGRESSION TEST for a defect this package caused: v0.189.0 replaced the container-count
// term with recorded intent, and a drive-gate-stopped app reads as `running` + zero containers, i.e.
// a boot orphan. Confirmed live on 2026-08-02 (the sweep started it, burned both attempts, and
// handed it to the dead-app alarm).
// fakeDriveGate answers a scripted liveness verdict per app.
type fakeDriveGate struct {
dead map[string]string // app → reason it is not live
asked []string
unsure map[string]bool // app → the gate cannot determine (must be treated as NOT live)
}
func (g *fakeDriveGate) MayStart(name string) (bool, string) {
g.asked = append(g.asked, name)
if r, ok := g.dead[name]; ok {
return false, r
}
if g.unsure[name] {
return false, "cannot determine"
}
return true, ""
}
// --- Group G — Scenario A / Part 0's finding, as a test ------------------------------------------
func TestReconcile_DriveAbsentApp_IsNeverStarted(t *testing.T) {
// The exact live shape: the drive gate stopped it (zero containers), it is still recorded
// `running` because the gate is not the customer, and its drive is gone.
//
// RED-PROOF: delete the `if live, reason := r.driveLive(...)` block from Run and this test fails
// with a start count of 1 — which is precisely what was observed on the box before the fix.
// Demonstrated in REPORT.md §4.
app := withDesired(vanished("calibre-web"), stacks.DesiredStateRunning)
f := &fakeStacks{list: []stacks.Stack{app}, onStart: comesUp}
r, _ := newTestReconciler(f)
gate := &fakeDriveGate{dead: map[string]string{"calibre-web": "drive /mnt/felhom-drives/hdd_1 is not a live mountpoint"}}
r.SetDriveGate(gate)
res := r.Run(context.Background())
if n := f.starts["calibre-web"]; n != 0 {
t.Fatalf("an app whose data drive is ABSENT was started %d time(s) — compose would create its "+
"bind sources on the guest rootfs, which is the hazard the drive gate exists to prevent", n)
}
if len(res.Candidates) != 0 {
t.Fatalf("a drive-held app was listed as a start candidate: %v", res.Candidates)
}
if len(res.HeldByDrive) != 1 || res.HeldByDrive[0] != "calibre-web" {
t.Fatalf("HeldByDrive = %v, want [calibre-web] — a held app must be reported, not silently dropped", res.HeldByDrive)
}
if len(res.StillDown) != 0 {
t.Fatalf("a deliberately-held app was reported as StillDown %v — that is the dead-app alarm's "+
"bucket, and putting it there is the false alarm this fix removes", res.StillDown)
}
if res.Attempts != 0 {
t.Fatalf("attempts=%d, want 0 — nothing should have been attempted", res.Attempts)
}
}
func TestReconcile_UndeterminableDrive_IsNotStarted(t *testing.T) {
// §8.4's fail-safe direction. "Cannot determine" must behave exactly like "absent": not starting
// is recoverable (the drive gate's Return branch owns it); starting on an absent drive is not.
app := withDesired(vanished("immich"), stacks.DesiredStateRunning)
f := &fakeStacks{list: []stacks.Stack{app}, onStart: comesUp}
r, _ := newTestReconciler(f)
r.SetDriveGate(&fakeDriveGate{unsure: map[string]bool{"immich": true}})
res := r.Run(context.Background())
if len(f.starts) != 0 {
t.Fatalf("an app whose drive liveness could NOT be determined was started: %v — the fail-safe "+
"direction is to refuse", f.starts)
}
if len(res.HeldByDrive) != 1 {
t.Fatalf("HeldByDrive = %v, want the undeterminable app held", res.HeldByDrive)
}
}
func TestReconcile_LiveDriveApp_IsStillRecovered(t *testing.T) {
// The gate must not become a blanket refusal — an app on a LIVE drive is still the R-157 case
// and must still be recovered. Without this, a "fix" that returns false always would pass the
// test above and silently disable the whole feature.
app := withDesired(vanished("bookstack"), stacks.DesiredStateRunning)
f := &fakeStacks{list: []stacks.Stack{app}, onStart: comesUp}
r, _ := newTestReconciler(f)
gate := &fakeDriveGate{}
r.SetDriveGate(gate)
res := r.Run(context.Background())
if f.starts["bookstack"] == 0 {
t.Fatal("an app on a LIVE drive was not recovered — the drive gate must refuse absent drives, not all of them")
}
if len(res.HeldByDrive) != 0 {
t.Fatalf("an app on a live drive was reported held: %v", res.HeldByDrive)
}
if len(gate.asked) != 1 || gate.asked[0] != "bookstack" {
t.Fatalf("the gate was asked %v, want exactly [bookstack] — one question per candidate", gate.asked)
}
}
func TestReconcile_DriveGateIsOnlyAskedAboutOrphans(t *testing.T) {
// A running app and a customer-stopped app are not candidates, so the gate must never be asked
// about them. Asking is not merely wasteful: the production gate reads app.yaml off disk per
// call, and a stopped app's drive being absent is not a fault anyone should hear about.
running := withDesired(stacks.Stack{
Name: "docmost", Deployed: true, State: stacks.StateRunning,
Containers: []stacks.ContainerInfo{{Name: "docmost", State: stacks.StateRunning}},
}, stacks.DesiredStateRunning)
stopped := withDesired(vanished("nextcloud"), stacks.DesiredStateStopped)
orphan := withDesired(vanished("immich"), stacks.DesiredStateRunning)
f := &fakeStacks{list: []stacks.Stack{running, stopped, orphan}, onStart: comesUp}
r, _ := newTestReconciler(f)
gate := &fakeDriveGate{}
r.SetDriveGate(gate)
r.Run(context.Background())
if len(gate.asked) != 1 || gate.asked[0] != "immich" {
t.Fatalf("the drive gate was asked about %v, want exactly [immich] — only boot orphans", gate.asked)
}
}
func TestReconcile_NoDriveGateWired_IsPermissive(t *testing.T) {
// nil gate = "this caller has no drive concept" (the fixtures' case), NOT "cannot determine".
// Production wiring is pinned separately by TestMainWiresBootDriveGate — an unwired gate here
// would silently be the pre-v0.190.0 behaviour, which is why that AST test exists.
app := withDesired(vanished("immich"), stacks.DesiredStateRunning)
f := &fakeStacks{list: []stacks.Stack{app}, onStart: comesUp}
r, _ := newTestReconciler(f)
r.Run(context.Background())
if f.starts["immich"] == 0 {
t.Fatal("with no drive gate wired the sweep must behave as before — the nil case is permissive")
}
}
// --- Scenario F — the two boot gates agree -------------------------------------------------------
func TestBothBootGatesAgreeOnIntent(t *testing.T) {
// R-170 + R-166: isBootOrphan and shouldRecreateOnBoot answer the SAME question — did the
// customer want this running? — and until v0.190.0 they answered it with different signals.
//
// shouldRecreateOnBoot lives in internal/web and cannot be called from here without an import
// cycle, so this test pins THIS side of the agreement and its sibling
// TestShouldRecreateOnBoot_AgreesWithBootrecon (internal/web) pins the other, against the same
// fixture table. Both must be updated together if the table changes.
cases := []struct {
desired string
containers int
wantWanted bool // "the customer wanted this running"
}{
{stacks.DesiredStateStopped, 0, false},
{stacks.DesiredStateStopped, 2, false},
{stacks.DesiredStateRunning, 0, true},
{stacks.DesiredStateRunning, 2, true},
{stacks.DesiredStateUnknown, 0, false}, // legacy: zero containers ⇒ treated as stopped
{stacks.DesiredStateUnknown, 2, true}, // legacy: containers present ⇒ treated as wanted
}
for _, c := range cases {
s := stacks.Stack{
Name: "app", Deployed: true, State: stacks.StateExited,
Containers: make([]stacks.ContainerInfo, c.containers),
AppConfig: &stacks.AppConfig{Deployed: true, DesiredState: c.desired},
}
if got := isBootOrphan(s); got != c.wantWanted {
t.Fatalf("isBootOrphan(desired=%q containers=%d) = %v, want %v — the two boot gates must "+
"answer the intent question identically", c.desired, c.containers, got, c.wantWanted)
}
}
}
// --- Group F / §8.2 — every holder the widened window can now overlap ----------------------------
func TestReconcile_HeldByAnyHolder_IsNeverStarted(t *testing.T) {
// §8.2's table, one case per row that the gate is responsible for. The reasons differ; the
// required behaviour is identical, which is why they share one seam.
//
// The first two rows only became reachable when R-157 mechanism A widened the boot window — the
// old T+5 s single sweep never overlapped a quiesce or a running app-data operation. Widening the
// window without these would have traded a fixed bug for two new ones.
for _, reason := range []string{
"a whole-guest backup (quiesce) is holding it — the quiesce loop restarts its own stacks",
"an app-data operation is holding it — the app-stop guard restarts it when the operation ends",
"drive /mnt/felhom-drives/hdd_1 is not a live mountpoint",
} {
app := withDesired(vanished("immich"), stacks.DesiredStateRunning)
f := &fakeStacks{list: []stacks.Stack{app}, onStart: comesUp}
r, _ := newTestReconciler(f)
r.SetDriveGate(&fakeDriveGate{dead: map[string]string{"immich": reason}})
res := r.Run(context.Background())
if len(f.starts) != 0 {
t.Fatalf("held by %q but started anyway: %v", reason, f.starts)
}
if len(res.HeldByDrive) != 1 {
t.Fatalf("held by %q but not reported as held: %+v", reason, res)
}
if len(res.StillDown) != 0 {
t.Fatalf("held by %q and reported as StillDown %v — that is the dead-app alarm's bucket",
reason, res.StillDown)
}
}
}
@@ -320,3 +320,95 @@ func writeFileAtomic(path string, b []byte) error {
} }
return os.Rename(tmp, path) return os.Rename(tmp, path)
} }
// --- Endpoint-drift detection (R-77, from DIAG-agent-channel-2026-07-26) ---------------------
//
// THE OUTAGE THIS EXISTS FOR: the R-50 island migration rewrote bootstrap.json's
// local_api.endpoint to the island address; controller.yaml kept the pre-island LAN address on the
// whole fleet; the agent no longer binds that address. Both controllers went dark for ~17.5 h, and
// the only alert said "agent unreachable" — indistinguishable from a dead agent or a network blip,
// so it read as infrastructure noise rather than a config fault. ensureLocalAPI could not catch it:
// it fills an ABSENT local_api block and returns early on a present one, stale or not.
//
// This is DETECTION AND NAMING ONLY. It deliberately does NOT reconcile the two files:
//
// the mirror-image failure is just as bad — on a guest whose controller.yaml is correct and whose
// bootstrap.json is stale, auto-reconcile would clobber a WORKING channel, fleet-wide, on the next
// restart. Which file is authoritative is a real, unresolved question and is tracked as R-78.
//
// Naming it is enough to have converted that outage into a specific, actionable alert on the first
// health cycle, which is the whole lesson of the incident.
// EndpointDrift is a detected divergence between the two local_api sources. It carries no secrets:
// the fingerprint is reported as an agreement BOOLEAN and the token is not compared or exposed at
// all (a token mismatch is a different failure — see the field comment).
type EndpointDrift struct {
ConfigPath string // controller.yaml
BootstrapPath string // bootstrap.json
ConfigEndpoint string // what the controller is actually dialling
BootstrapEndpoint string // what the provisioning side last wrote
// FingerprintAgrees is false when the pin ALSO moved. That is a materially different (and worse)
// situation than a moved address — fixing the endpoint alone would then fail closed on the pin —
// so it is surfaced, as a boolean, never as a value.
FingerprintAgrees bool
}
// DetectEndpointDrift compares controller.yaml's live local_api.endpoint against bootstrap.json's.
// Returns nil (silent, no alert) in every ambiguous or not-applicable case:
//
// - cfg is nil, or its endpoint is EMPTY — that is the fill-if-missing path ensureLocalAPI owns,
// not drift;
// - bootstrap.json is absent, unreadable or unparseable — a legacy / manually-configured /
// unprovisioned guest is not a drifted one;
// - the bootstrap local_api block is incomplete (any of endpoint/fingerprint/token empty) — the
// same completeness bar ensureLocalAPI applies before it will merge;
// - the endpoints agree.
//
// It reads two files and writes NOTHING. Emitting the ERROR here (rather than at the call site)
// keeps the diagnosis in one line of log even when the alert path is unavailable.
func DetectEndpointDrift(configPath string, cfg *config.Config, logger *log.Logger) *EndpointDrift {
if cfg == nil || cfg.LocalAPI.Endpoint == "" {
return nil
}
data, err := os.ReadFile(Path())
if err != nil {
return nil
}
var b Bootstrap
if err := json.Unmarshal(data, &b); err != nil {
return nil
}
if b.LocalAPI.Endpoint == "" || b.LocalAPI.Fingerprint == "" || b.LocalAPI.Token == "" {
return nil
}
if cfg.LocalAPI.Endpoint == b.LocalAPI.Endpoint {
return nil
}
d := &EndpointDrift{
ConfigPath: configPath,
BootstrapPath: Path(),
ConfigEndpoint: cfg.LocalAPI.Endpoint,
BootstrapEndpoint: b.LocalAPI.Endpoint,
FingerprintAgrees: cfg.LocalAPI.Fingerprint == b.LocalAPI.Fingerprint,
}
if logger != nil {
logger.Printf("[ERROR] bootstrap: local_api endpoint DRIFT — %s says %q but %s says %q; "+
"the controller is dialling the FORMER. Pin agrees: %v. Not auto-corrected (R-78 owns the "+
"authority ruling) — fix the intended file and restart the controller.",
d.ConfigPath, d.ConfigEndpoint, d.BootstrapPath, d.BootstrapEndpoint, d.FingerprintAgrees)
}
return d
}
// EnglishMessage is the operator-tier alert body (operator events are English by convention).
func (d *EndpointDrift) EnglishMessage() string {
return fmt.Sprintf("local_api endpoint drift: controller.yaml=%s bootstrap.json=%s (pin agrees: %v) "+
"— the controller is dialling controller.yaml's value; the agent may be listening on the other.",
d.ConfigEndpoint, d.BootstrapEndpoint, d.FingerprintAgrees)
}
// HungarianMessage is the customer-facing dashboard line, matching channelhealth's tone (short,
// no addresses — the operator gets those in the event and the log).
func (d *EndpointDrift) HungarianMessage() string {
return "A tárolókezelő ügynök címe elavult a beállításokban."
}
+223
View File
@@ -0,0 +1,223 @@
package bootstrap
import (
"bytes"
"encoding/json"
"io"
"log"
"os"
"path/filepath"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-controller/internal/config"
)
// R-77 endpoint-drift detection. The outage this guards is DIAG-agent-channel-2026-07-26: the island
// migration rewrote bootstrap.json, controller.yaml kept the pre-island LAN address, and the fleet's
// controllers dialled the wrong host for 17.5 h while alerting only "agent unreachable".
const driftYAML = `customer:
id: demo-hp
local_api:
endpoint: %ENDPOINT%
fingerprint: aaaa1111
token: tok-secret
`
// writeDriftFixture lays out a controller.yaml + bootstrap.json pair and points Path() at the latter.
func writeDriftFixture(t *testing.T, cfgEndpoint, bsEndpoint, bsFingerprint, bsToken string) (cfgPath string, cfg *config.Config) {
t.Helper()
dir := t.TempDir()
cfgPath = filepath.Join(dir, "controller.yaml")
if err := os.WriteFile(cfgPath, []byte(strings.ReplaceAll(driftYAML, "%ENDPOINT%", cfgEndpoint)), 0o600); err != nil {
t.Fatal(err)
}
bsPath := filepath.Join(dir, "bootstrap.json")
b := Bootstrap{}
b.LocalAPI = BootstrapLocalAPI{Endpoint: bsEndpoint, Fingerprint: bsFingerprint, Token: bsToken}
raw, _ := json.Marshal(b)
if err := os.WriteFile(bsPath, raw, 0o600); err != nil {
t.Fatal(err)
}
t.Setenv("FELHOM_BOOTSTRAP_PATH", bsPath)
var err error
cfg, err = config.LoadPermissive(cfgPath)
if err != nil {
t.Fatal(err)
}
return cfgPath, cfg
}
func sha(t *testing.T, p string) []byte {
t.Helper()
b, err := os.ReadFile(p)
if err != nil {
t.Fatal(err)
}
return b
}
// Scenario A — drift is DETECTED, NAMED, and nothing is written.
//
// The "nothing is written" half is the load-bearing assertion: auto-reconcile is R-78 and would
// clobber a working channel on any guest whose controller.yaml is the correct one. "An error was
// logged" alone would be a hollow test.
func TestScenarioA_DriftDetectedAndNamed_NoWrite(t *testing.T) {
cfgPath, cfg := writeDriftFixture(t,
"192.168.0.87:8443", // the live demo-hp value
"169.254.253.1:8443", // the island value the migration wrote
"aaaa1111", "tok-secret")
before := sha(t, cfgPath)
var buf bytes.Buffer
d := DetectEndpointDrift(cfgPath, cfg, log.New(&buf, "", 0))
if d == nil {
t.Fatal("drift must be DETECTED — this is the exact live shape from the 2026-07-25 outage")
}
if d.ConfigEndpoint != "192.168.0.87:8443" || d.BootstrapEndpoint != "169.254.253.1:8443" {
t.Errorf("wrong values captured: %+v", d)
}
if !d.FingerprintAgrees {
t.Error("fingerprints are identical in this fixture — must report agreement")
}
// The log line alone must diagnose it: BOTH values AND BOTH paths.
logged := buf.String()
for _, want := range []string{"192.168.0.87:8443", "169.254.253.1:8443", cfgPath, "bootstrap.json", "ERROR"} {
if !strings.Contains(logged, want) {
t.Errorf("the ERROR line must contain %q; got:\n%s", want, logged)
}
}
// Never log a secret.
if strings.Contains(logged, "tok-secret") {
t.Error("the token must NEVER be logged")
}
// THE assertion: controller.yaml is byte-identical.
if after := sha(t, cfgPath); !bytes.Equal(before, after) {
t.Errorf("controller.yaml was MODIFIED — detection must never write (that is R-78)\nbefore:\n%s\nafter:\n%s", before, after)
}
}
// Scenario B — agreement is silent. A spurious alert on every healthy boot would be worse than the
// bug: it trains the operator to ignore the banner.
func TestScenarioB_AgreementIsSilent(t *testing.T) {
cfgPath, cfg := writeDriftFixture(t, "169.254.253.1:8443", "169.254.253.1:8443", "aaaa1111", "tok-secret")
before := sha(t, cfgPath)
var buf bytes.Buffer
if d := DetectEndpointDrift(cfgPath, cfg, log.New(&buf, "", 0)); d != nil {
t.Errorf("matching endpoints must not report drift: %+v", d)
}
if strings.Contains(buf.String(), "ERROR") {
t.Errorf("no ERROR on agreement; got: %s", buf.String())
}
if after := sha(t, cfgPath); !bytes.Equal(before, after) {
t.Error("controller.yaml must not be touched")
}
}
// Scenario D — an absent / unparseable / incomplete bootstrap is NOT drift. An unprovisioned or
// legacy guest must stay silent, not alarm.
func TestScenarioD_IncompleteBootstrapIsFailSafe(t *testing.T) {
t.Run("bootstrap absent", func(t *testing.T) {
cfgPath, cfg := writeDriftFixture(t, "192.168.0.87:8443", "169.254.253.1:8443", "aaaa1111", "tok")
os.Remove(os.Getenv("FELHOM_BOOTSTRAP_PATH"))
if d := DetectEndpointDrift(cfgPath, cfg, log.New(io.Discard, "", 0)); d != nil {
t.Errorf("a missing bootstrap is not drift: %+v", d)
}
})
t.Run("bootstrap unparseable", func(t *testing.T) {
cfgPath, cfg := writeDriftFixture(t, "192.168.0.87:8443", "169.254.253.1:8443", "aaaa1111", "tok")
os.WriteFile(os.Getenv("FELHOM_BOOTSTRAP_PATH"), []byte("{not json"), 0o600)
if d := DetectEndpointDrift(cfgPath, cfg, log.New(io.Discard, "", 0)); d != nil {
t.Errorf("an unparseable bootstrap is not drift: %+v", d)
}
})
// Incomplete = any of endpoint/fingerprint/token empty — the same completeness bar ensureLocalAPI
// applies before it will merge.
for _, tc := range []struct{ name, ep, fp, tok string }{
{"no endpoint", "", "aaaa1111", "tok"},
{"no fingerprint", "169.254.253.1:8443", "", "tok"},
{"no token", "169.254.253.1:8443", "aaaa1111", ""},
} {
t.Run(tc.name, func(t *testing.T) {
cfgPath, cfg := writeDriftFixture(t, "192.168.0.87:8443", tc.ep, tc.fp, tc.tok)
if d := DetectEndpointDrift(cfgPath, cfg, log.New(io.Discard, "", 0)); d != nil {
t.Errorf("an incomplete bootstrap is not drift: %+v", d)
}
})
}
t.Run("controller.yaml has no endpoint — that is the fill-if-missing path, not drift", func(t *testing.T) {
cfgPath, cfg := writeDriftFixture(t, `""`, "169.254.253.1:8443", "aaaa1111", "tok")
if cfg.LocalAPI.Endpoint != "" {
t.Skipf("fixture did not produce an empty endpoint (got %q)", cfg.LocalAPI.Endpoint)
}
if d := DetectEndpointDrift(cfgPath, cfg, log.New(io.Discard, "", 0)); d != nil {
t.Errorf("an absent local_api endpoint is ensureLocalAPI's job, not drift: %+v", d)
}
})
t.Run("nil cfg", func(t *testing.T) {
if d := DetectEndpointDrift("/nonexistent", nil, log.New(io.Discard, "", 0)); d != nil {
t.Error("nil cfg must be silent")
}
})
}
// A moved PIN alongside a moved address is a materially different failure — fixing the endpoint
// alone would then fail closed on the pin. It must be surfaced, as a boolean, never as a value.
func TestDrift_FingerprintDisagreementIsSurfacedNotLeaked(t *testing.T) {
cfgPath, cfg := writeDriftFixture(t, "192.168.0.87:8443", "169.254.253.1:8443", "bbbb2222", "tok-secret")
var buf bytes.Buffer
d := DetectEndpointDrift(cfgPath, cfg, log.New(&buf, "", 0))
if d == nil {
t.Fatal("drift expected")
}
if d.FingerprintAgrees {
t.Error("fingerprints differ in this fixture — must report DISagreement")
}
for _, secret := range []string{"aaaa1111", "bbbb2222", "tok-secret"} {
if strings.Contains(buf.String(), secret) {
t.Errorf("a fingerprint/token value leaked into the log: %q", secret)
}
if strings.Contains(d.EnglishMessage(), secret) {
t.Errorf("a fingerprint/token value leaked into the operator message: %q", secret)
}
}
if !strings.Contains(d.EnglishMessage(), "false") {
t.Errorf("the operator message must carry the pin-agreement boolean: %q", d.EnglishMessage())
}
}
// Scenario C — the fill-if-missing path is untouched by any of this (it is the ONLY writer).
func TestScenarioC_AbsentBlockStillMerges(t *testing.T) {
dir := t.TempDir()
cfgPath := filepath.Join(dir, "controller.yaml")
if err := os.WriteFile(cfgPath, []byte("customer:\n id: demo-hp\n"), 0o600); err != nil {
t.Fatal(err)
}
bsPath := filepath.Join(dir, "bootstrap.json")
b := Bootstrap{}
b.LocalAPI = BootstrapLocalAPI{Endpoint: "169.254.253.1:8443", Fingerprint: "aaaa1111", Token: "tok"}
raw, _ := json.Marshal(b)
os.WriteFile(bsPath, raw, 0o600)
t.Setenv("FELHOM_BOOTSTRAP_PATH", bsPath)
cfg, err := config.LoadPermissive(cfgPath)
if err != nil {
t.Fatal(err)
}
if cfg.LocalAPI.Endpoint != "" {
t.Fatalf("precondition: fixture should have no local_api, got %q", cfg.LocalAPI.Endpoint)
}
got := ensureLocalAPI(cfgPath, cfg, log.New(io.Discard, "", 0))
if got == nil || got.LocalAPI.Endpoint != "169.254.253.1:8443" {
t.Fatalf("the fill-if-missing merge must still work; got %+v", got)
}
// And the drift check stays silent on the now-merged config.
if d := DetectEndpointDrift(cfgPath, got, log.New(io.Discard, "", 0)); d != nil {
t.Errorf("after a successful merge the two files agree — no drift: %+v", d)
}
}

Some files were not shown because too many files have changed in this diff Show More